pyrit.score.SelfAskQuestionAnswerScorer

pyrit.score.SelfAskQuestionAnswerScorer#

class SelfAskQuestionAnswerScorer(*, chat_target: ~pyrit.prompt_target.common.prompt_chat_target.PromptChatTarget, true_false_question_path: ~pathlib.Path | None = None, validator: ~pyrit.score.scorer_prompt_validator.ScorerPromptValidator | None = None, score_aggregator: ~typing.Callable[[~typing.Iterable[~pyrit.models.score.Score]], ~pyrit.score.score_aggregator_result.ScoreAggregatorResult] = <function _create_aggregator.<locals>.aggregator>)[source]#

Bases: SelfAskTrueFalseScorer

A class that represents a self-ask question answering scorer.

Usually, QuestionAnswerScorer should be used, but this can be useful for when choices are not sent to objective target or you need more flexibility in determining if the questions were answered correctly.

__init__(*, chat_target: ~pyrit.prompt_target.common.prompt_chat_target.PromptChatTarget, true_false_question_path: ~pathlib.Path | None = None, validator: ~pyrit.score.scorer_prompt_validator.ScorerPromptValidator | None = None, score_aggregator: ~typing.Callable[[~typing.Iterable[~pyrit.models.score.Score]], ~pyrit.score.score_aggregator_result.ScoreAggregatorResult] = <function _create_aggregator.<locals>.aggregator>) → None[source]#

Initializes the SelfAskQuestionAnswerScorer object.

Parameters:

  • chat_target (PromptChatTarget) – The chat target to use for the scorer.

  • true_false_question_path (Optional[pathlib.Path]) – The path to the true/false question file. Defaults to None, which uses the default question_answering.yaml file.

  • validator (Optional[ScorerPromptValidator]) – Custom validator. Defaults to None.

  • score_aggregator (TrueFalseAggregatorFunc) – The aggregator function to use. Defaults to TrueFalseScoreAggregator.OR.

Methods

__init__(*, chat_target[, ...])

Initializes the SelfAskQuestionAnswerScorer object.

get_identifier()

Returns an identifier dictionary for the scorer.

get_scorer_metrics(dataset_name[, metrics_type])

Returns evaluation statistics for the scorer using the dataset_name of the human labeled dataset that this scorer was run against.

scale_value_float(value, min_value, max_value)

Scales a value from 0 to 1 based on the given min and max values.

score_async(request_response, *[, ...])

Score the request_response, add the results to the database and return a list of Score objects.

score_image_async(image_path, *[, objective])

Scores the given image using the chat target.

score_image_batch_async(*, image_paths[, ...])

score_prompts_batch_async(*, request_responses)

Score multiple prompts in batches using the provided objectives.

score_response_async(*, response[, ...])

Score a response using an objective scorer and optional auxiliary scorers.

score_response_multiple_scorers_async(*, ...)

Score a response using multiple scorers in parallel.

score_text_async(text, *[, objective])

Scores the given text based on the task using the chat target.

validate_return_scores(scores)

Validates the scores returned by the scorer.

Attributes

scorer_type

Contents