pyrit.executor.attack.ObjectiveEvaluator

pyrit.executor.attack.ObjectiveEvaluator#

class ObjectiveEvaluator(*, scorer: Scorer, use_score_as_feedback: bool = True, successful_objective_threshold: float = 0.8)[source]#

Bases: object

Evaluates scores from a Scorer to determine objective achievement and provide feedback.

This component wraps a Scorer and provides logic for: - Determining if an objective is achieved based on score values and thresholds. - Extracting feedback from score rationales for use in subsequent prompts. - Supporting both true/false and numeric score types.

__init__(*, scorer: Scorer, use_score_as_feedback: bool = True, successful_objective_threshold: float = 0.8)[source]#

Initialize the objective evaluator.

Parameters:

  • scorer (Scorer) – The scorer to use for evaluating responses

  • use_score_as_feedback (bool) – Whether to use the score as feedback for future prompts

  • successful_objective_threshold (float) – The score threshold to consider an objective successful [0.0 to 1.0, inclusive].

Raises:

ValueError – If successful_objective_threshold is not between 0.0 and 1.0 (inclusive).

Methods

__init__(*, scorer[, use_score_as_feedback, ...])

Initialize the objective evaluator.

get_feedback(score)

Get feedback from a score for use in future prompts.

is_objective_achieved([score])

Determine if the objective is achieved based on score.

Attributes

scorer_type

Get the type of the scorer.
get_feedback(score: Score) → str | None[source]#

Get feedback from a score for use in future prompts.

Parameters:

score (Optional[Score]) – The score to get feedback from.

Returns:

Feedback string, or None if no suitable feedback exists.

Return type:

Optional[str]

is_objective_achieved(score: Score | None = None) → bool[source]#

Determine if the objective is achieved based on score.

Parameters:

score (Optional[Score]) – The score to check.

Returns:

True if the objective is achieved, False otherwise.

Return type:

bool

Raises:

ValueError – If score value cannot be converted to the expected type.

property scorer_type: str#

Get the type of the scorer.

Returns:

The type identifier of the scorer.

Return type:

str

Contents