pyrit.score.SelfAskQuestionAnswerScorer

pyrit.score.SelfAskQuestionAnswerScorer#

class SelfAskQuestionAnswerScorer(*, chat_target: PromptChatTarget, true_false_question_path: Path | None = None)[source]#

Bases: SelfAskTrueFalseScorer

A class that represents a self-ask question answering scorer.

Usually, QuestionAnswerScorer should be used, but this can be useful for when choices are not sent to objective target or you need more flexibility in determining if the questions were answered correctly.

__init__(*, chat_target: PromptChatTarget, true_false_question_path: Path | None = None) → None[source]#

Initializes the SelfAskQuestionAnswerScorer object.

Parameters:

  • chat_target (PromptChatTarget) – The chat target to use for the scorer.

  • true_false_question_path (pathlib.Path) – The path to the true/false question file.

Methods

__init__(*, chat_target[, ...])

Initializes the SelfAskQuestionAnswerScorer object.

get_identifier()

Returns an identifier dictionary for the scorer.

scale_value_float(value, min_value, max_value)

Scales a value from 0 to 1 based on the given min and max values.

score_async(request_response, *[, task])

Score the request_reponse using the QuestionAnsweringEntry and return a single score object

score_image_async(image_path, *[, task])

Scores the given image using the chat target.

score_prompts_with_tasks_batch_async(*, ...)

score_responses_inferring_tasks_batch_async(*, ...)

Scores a batch of responses (ignores non-assistant messages).

score_text_async(text, *[, task])

Scores the given text based on the task using the chat target.

validate(request_response, *[, task])

Validates the request_response piece to score.

Attributes

scorer_type

async score_async(request_response: PromptRequestPiece, *, task: str | None = None) → list[Score][source]#

Score the request_reponse using the QuestionAnsweringEntry and return a single score object

Parameters:
Returns:

A single Score object representing the result

Return type:

Score

scorer_type: Literal['true_false', 'float_scale']#
validate(request_response: PromptRequestPiece, *, task: str | None = None)[source]#

Validates the request_response piece to score. Because some scorers may require specific PromptRequestPiece types or values.

Parameters:

  • request_response (PromptRequestPiece) – The request response to be validated.

  • task (str) – The task based on which the text should be scored (the original attacker model’s objective).

Contents