pyrit.score.QuestionAnswerScorer

pyrit.score.QuestionAnswerScorer#

class QuestionAnswerScorer(*, correct_answer_matching_patterns: list[str] = ['{correct_answer_index}:', '{correct_answer}'], category: list[str] | None = None, validator: ~pyrit.score.scorer_prompt_validator.ScorerPromptValidator | None = None, score_aggregator: ~typing.Callable[[~typing.Iterable[~pyrit.models.score.Score]], ~pyrit.score.score_aggregator_result.ScoreAggregatorResult] = <function _create_aggregator.<locals>.aggregator>)[source]#

Bases: TrueFalseScorer

A class that represents a question answering scorer.

__init__(*, correct_answer_matching_patterns: list[str] = ['{correct_answer_index}:', '{correct_answer}'], category: list[str] | None = None, validator: ~pyrit.score.scorer_prompt_validator.ScorerPromptValidator | None = None, score_aggregator: ~typing.Callable[[~typing.Iterable[~pyrit.models.score.Score]], ~pyrit.score.score_aggregator_result.ScoreAggregatorResult] = <function _create_aggregator.<locals>.aggregator>) → None[source]#

Initialize the QuestionAnswerScorer.

Parameters:

  • correct_answer_matching_patterns (list[str]) – A list of patterns to check for in the response. If any pattern is found in the response, the score will be True. These patterns should be format strings that will be formatted with the correct answer metadata. Defaults to CORRECT_ANSWER_MATCHING_PATTERNS.

  • category (Optional[list[str]]) – Optional list of categories for the score. Defaults to None.

  • validator (Optional[ScorerPromptValidator]) – Custom validator. Defaults to None.

  • score_aggregator (TrueFalseAggregatorFunc) – The aggregator function to use. Defaults to TrueFalseScoreAggregator.OR.

Methods

__init__(*[, ...])

Initialize the QuestionAnswerScorer.

get_identifier()

Returns an identifier dictionary for the scorer.

get_scorer_metrics(dataset_name[, metrics_type])

Returns evaluation statistics for the scorer using the dataset_name of the human labeled dataset that this scorer was run against.

scale_value_float(value, min_value, max_value)

Scales a value from 0 to 1 based on the given min and max values.

score_async(request_response, *[, ...])

Score the request_response, add the results to the database and return a list of Score objects.

score_image_async(image_path, *[, objective])

Scores the given image using the chat target.

score_image_batch_async(*, image_paths[, ...])

score_prompts_batch_async(*, request_responses)

Score multiple prompts in batches using the provided objectives.

score_response_async(*, response[, ...])

Score a response using an objective scorer and optional auxiliary scorers.

score_response_multiple_scorers_async(*, ...)

Score a response using multiple scorers in parallel.

score_text_async(text, *[, objective])

Scores the given text based on the task using the chat target.

validate_return_scores(scores)

Validates the scores returned by the scorer.

Attributes

CORRECT_ANSWER_MATCHING_PATTERNS

scorer_type

CORRECT_ANSWER_MATCHING_PATTERNS = ['{correct_answer_index}:', '{correct_answer}']#
Contents