pyrit.score.SelfAskQuestionAnswerScorer

pyrit.score.SelfAskQuestionAnswerScorer#

class SelfAskQuestionAnswerScorer(*, chat_target: ~pyrit.prompt_target.common.prompt_chat_target.PromptChatTarget, true_false_question_path: ~pathlib.Path | None = None, validator: ~pyrit.score.scorer_prompt_validator.ScorerPromptValidator | None = None, score_aggregator: ~typing.Callable[[~typing.Iterable[~pyrit.models.score.Score]], ~pyrit.score.score_aggregator_result.ScoreAggregatorResult] = <function _create_aggregator.<locals>.aggregator>)[source]#

Bases: SelfAskTrueFalseScorer

A class that represents a self-ask question answering scorer.

Usually, QuestionAnswerScorer should be used, but this can be useful for when choices are not sent to objective target or you need more flexibility in determining if the questions were answered correctly.

__init__(*, chat_target: ~pyrit.prompt_target.common.prompt_chat_target.PromptChatTarget, true_false_question_path: ~pathlib.Path | None = None, validator: ~pyrit.score.scorer_prompt_validator.ScorerPromptValidator | None = None, score_aggregator: ~typing.Callable[[~typing.Iterable[~pyrit.models.score.Score]], ~pyrit.score.score_aggregator_result.ScoreAggregatorResult] = <function _create_aggregator.<locals>.aggregator>) → None[source]#

Initialize the SelfAskQuestionAnswerScorer object.

Parameters:

  • chat_target (PromptChatTarget) – The chat target to use for the scorer.

  • true_false_question_path (Optional[pathlib.Path]) – The path to the true/false question file. Defaults to None, which uses the default question_answering.yaml file.

  • validator (Optional[ScorerPromptValidator]) – Custom validator. Defaults to None.

  • score_aggregator (TrueFalseAggregatorFunc) – The aggregator function to use. Defaults to TrueFalseScoreAggregator.OR.

Methods

__init__(*, chat_target[, ...])

Initialize the SelfAskQuestionAnswerScorer object.

get_identifier()

Get an identifier dictionary for the scorer.

get_scorer_metrics(dataset_name[, metrics_type])

Get evaluation statistics for the scorer using the dataset_name of the human labeled dataset.

scale_value_float(value, min_value, max_value)

Scales a value from 0 to 1 based on the given min and max values.

score_async(message, *[, objective, ...])

Score the message, add the results to the database, and return a list of Score objects.

score_image_async(image_path, *[, objective])

Score the given image using the chat target.

score_image_batch_async(*, image_paths[, ...])

Score a batch of images asynchronously.

score_prompts_batch_async(*, messages[, ...])

Score multiple prompts in batches using the provided objectives.

score_response_async(*, response[, ...])

Score a response using an objective scorer and optional auxiliary scorers.

score_response_multiple_scorers_async(*, ...)

Score a response using multiple scorers in parallel.

score_text_async(text, *[, objective])

Scores the given text based on the task using the chat target.

validate_return_scores(scores)

Validate the scores returned by the scorer.

Attributes

scorer_type

Contents