pyrit.score.QuestionAnswerScorer

pyrit.score.QuestionAnswerScorer#

class QuestionAnswerScorer(*, correct_answer_matching_patterns: list[str] = ['{correct_answer_index}:', '{correct_answer}'], category: str = '')[source]#

Bases: Scorer

A class that represents a question answering scorer.

__init__(*, correct_answer_matching_patterns: list[str] = ['{correct_answer_index}:', '{correct_answer}'], category: str = '') → None[source]#

Scores PromptRequestResponse objects that contain correct_answer_index and/or correct_answer metadata

Parameters:

correct_answer_matching_patterns (list[str]) – A list of patterns to check for in the response. If any pattern is found in the response, the score will be True. These patterns should be format strings that will be formatted with the correct answer metadata.

Methods

__init__(*[, ...])

Scores PromptRequestResponse objects that contain correct_answer_index and/or correct_answer metadata

get_identifier()

Returns an identifier dictionary for the scorer.

scale_value_float(value, min_value, max_value)

Scales a value from 0 to 1 based on the given min and max values.

score_async(request_response, *[, task])

Score the request_reponse using the QuestionAnsweringEntry and return a single score object

score_image_async(image_path, *[, task])

Scores the given image using the chat target.

score_prompts_with_tasks_batch_async(*, ...)

score_responses_inferring_tasks_batch_async(*, ...)

Scores a batch of responses (ignores non-assistant messages).

score_text_async(text, *[, task])

Scores the given text based on the task using the chat target.

validate(request_response, *[, task])

Validates the request_response piece to score.

Attributes

CORRECT_ANSWER_MATCHING_PATTERNS

scorer_type

CORRECT_ANSWER_MATCHING_PATTERNS = ['{correct_answer_index}:', '{correct_answer}']#
async score_async(request_response: PromptRequestPiece, *, task: str | None = None) → list[Score][source]#

Score the request_reponse using the QuestionAnsweringEntry and return a single score object

Parameters:
Returns:

A single Score object representing the result

Return type:

Score

scorer_type: Literal['true_false', 'float_scale']#
validate(request_response: PromptRequestPiece, *, task: str | None = None)[source]#

Validates the request_response piece to score. Because some scorers may require specific PromptRequestPiece types or values.

Parameters:

  • request_response (PromptRequestPiece) – The request response to be validated.

  • task (str) – The task based on which the text should be scored (the original attacker model’s objective).

Contents