pyrit.score

Scoring functionality for evaluating AI model responses across various dimensions including harm detection, objective completion, and content classification.

Functions¶

create_conversation_scorer¶

create_conversation_scorer(scorer: Scorer, validator: Optional[ScorerPromptValidator] = None) → Scorer

Create a ConversationScorer that inherits from the same type as the wrapped scorer.

This factory dynamically creates a ConversationScorer class that inherits from the wrapped scorer’s base class (FloatScaleScorer or TrueFalseScorer), ensuring the returned scorer is an instance of both ConversationScorer and the wrapped scorer’s type.

Returns:

• Scorer — A ConversationScorer instance that is also an instance of the wrapped scorer’s type.

Raises:

• ValueError — If the scorer is not an instance of FloatScaleScorer or TrueFalseScorer.

get_all_harm_metrics¶

get_all_harm_metrics(harm_category: str) → list[ScorerMetricsWithIdentity[HarmScorerMetrics]]

Load all harm scorer metrics for a specific harm category.

Returns a list of ScorerMetricsWithIdentity[HarmScorerMetrics] objects that wrap the scorer’s identity information and its performance metrics, enabling clean attribute access like entry.metrics.mean_absolute_error or entry.metrics.harm_category.

Returns:

• list[ScorerMetricsWithIdentity[HarmScorerMetrics]] — List[ScorerMetricsWithIdentity[HarmScorerMetrics]]: List of metrics with scorer identity. Access metrics via entry.metrics.mean_absolute_error, entry.metrics.harm_category, etc. Access scorer info via entry.scorer_identifier.class_name, etc.

get_all_objective_metrics¶

get_all_objective_metrics(file_path: Optional[Path] = None) → list[ScorerMetricsWithIdentity[ObjectiveScorerMetrics]]

Load all objective scorer metrics with full scorer identity for comparison.

Returns a list of ScorerMetricsWithIdentity[ObjectiveScorerMetrics] objects that wrap the scorer’s identity information and its performance metrics, enabling clean attribute access like entry.metrics.accuracy or entry.metrics.f1_score.

Returns:

• list[ScorerMetricsWithIdentity[ObjectiveScorerMetrics]] — List[ScorerMetricsWithIdentity[ObjectiveScorerMetrics]]: List of metrics with scorer identity. Access metrics via entry.metrics.accuracy, entry.metrics.f1_score, etc. Access scorer info via entry.scorer_identifier.class_name, etc.

AudioFloatScaleScorer¶

Bases: FloatScaleScorer

A scorer that processes audio files by transcribing them and scoring the transcript.

The AudioFloatScaleScorer transcribes audio to text using Azure Speech-to-Text, then scores the transcript using a FloatScaleScorer.

Constructor Parameters:

AudioTrueFalseScorer¶

Bases: TrueFalseScorer

A scorer that processes audio files by transcribing them and scoring the transcript.

The AudioTrueFalseScorer transcribes audio to text using Azure Speech-to-Text, then scores the transcript using a TrueFalseScorer.

Constructor Parameters:

AzureContentFilterScorer¶

Bases: FloatScaleScorer

A scorer that uses Azure Content Safety API to evaluate text and images for harmful content.

This scorer analyzes content across multiple harm categories (hate, self-harm, sexual, violence) and returns a score for each category in the range [0, 1], where higher scores indicate more severe content. Supports both text and image inputs.

Constructor Parameters:

Methods:

evaluate_async¶

evaluate_async(file_mapping: Optional[ScorerEvalDatasetFiles] = None, num_scorer_trials: int = 3, update_registry_behavior: RegistryUpdateBehavior = None, max_concurrency: int = 10) → Optional[ScorerMetrics]

Evaluate this scorer against human-labeled datasets.

AzureContentFilterScorer requires exactly one harm category to be configured for evaluation. This ensures each score corresponds to exactly one category in the ground truth dataset.

Returns:

• Optional[ScorerMetrics] — The evaluation metrics, or None if no datasets found.

Raises:

• ValueError — If more than one harm category is configured.

BatchScorer¶

A utility class for scoring prompts in batches in a parallelizable and convenient way.

This class provides functionality to score existing prompts stored in memory without any target interaction, making it a pure scoring utility.

Constructor Parameters:

Methods:

score_responses_by_filters_async¶

score_responses_by_filters_async(scorer: Scorer, attack_id: Optional[str | uuid.UUID] = None, conversation_id: Optional[str | uuid.UUID] = None, prompt_ids: Optional[list[str] | list[uuid.UUID]] = None, labels: Optional[dict[str, str]] = None, sent_after: Optional[datetime] = None, sent_before: Optional[datetime] = None, original_values: Optional[list[str]] = None, converted_values: Optional[list[str]] = None, data_type: Optional[str] = None, not_data_type: Optional[str] = None, converted_value_sha256: Optional[list[str]] = None, objective: str = '') → list[Score]

Score the responses that match the specified filters.

Returns:

• list[Score] — list[Score]: A list of Score objects for responses that match the specified filters.

Raises:

• ValueError — If no entries match the provided filters.

ConsoleScorerPrinter¶

Bases: ScorerPrinter

Console printer for scorer information with enhanced formatting.

This printer formats scorer details for console display with optional color coding, proper indentation, and visual hierarchy. Colors can be disabled for consoles that don’t support ANSI characters.

Constructor Parameters:

Methods:

print_harm_scorer¶

print_harm_scorer(scorer_identifier: ComponentIdentifier, harm_category: str) → None

Print harm scorer information including type, nested scorers, and evaluation metrics.

This method displays:

• Scorer type and identity information

• Nested sub-scorers (for composite scorers)

• Harm evaluation metrics (MAE, Krippendorff alpha) from the registry

print_objective_scorer¶

print_objective_scorer(scorer_identifier: ComponentIdentifier) → None

Print objective scorer information including type, nested scorers, and evaluation metrics.

This method displays:

• Scorer type and identity information

• Nested sub-scorers (for composite scorers)

• Objective evaluation metrics (accuracy, precision, recall, F1) from the registry

ContentClassifierPaths¶

Bases: enum.Enum

Paths to content classifier YAML files.

ConversationScorer¶

Bases: Scorer, ABC

Scorer that evaluates entire conversation history rather than individual messages.

This scorer wraps another scorer (FloatScaleScorer or TrueFalseScorer) and evaluates the full conversation context. Useful for multi-turn conversations where context matters (e.g., psychosocial harms that emerge over time or persuasion/deception over many messages).

The ConversationScorer dynamically inherits from the same base class as the wrapped scorer, ensuring proper type compatibility.

Note: This class cannot be instantiated directly. Use create_conversation_scorer() factory instead.

Methods:

validate_return_scores¶

validate_return_scores(scores: list[Score]) → None

Validate scores by delegating to the wrapped scorer’s validation.

DecodingScorer¶

Bases: TrueFalseScorer

Scorer that checks if the request values are in the output using a text matching strategy.

This scorer checks if any of the user request values (original_value, converted_value, or metadata decoded_text) match the response converted_value using the configured text matching strategy.

Constructor Parameters:

FloatScaleScoreAggregator¶

Namespace for float scale score aggregators that return a single aggregated score.

All aggregators return a list containing one ScoreAggregatorResult that combines all input scores together, preserving all categories.

FloatScaleScorer¶

Bases: Scorer

Base class for scorers that return floating-point scores in the range [0, 1].

This scorer evaluates prompt responses and returns numeric scores indicating the degree to which a response exhibits certain characteristics. Each piece in a request response is scored independently, returning one score per piece.

Constructor Parameters:

Methods:

get_scorer_metrics¶

get_scorer_metrics() → Optional[HarmScorerMetrics]

Get evaluation metrics for this scorer from the configured evaluation result file.

Returns:

• Optional[HarmScorerMetrics] — The metrics for this scorer, or None if not found or not configured.

validate_return_scores¶

validate_return_scores(scores: list[Score]) → None

Validate that the returned scores are within the valid range [0, 1].

Raises:

• ValueError — If any score is not between 0 and 1.

FloatScaleScorerAllCategories¶

Namespace for float scale score aggregators that combine all categories.

These aggregators ignore category boundaries and aggregate all scores together, returning a single ScoreAggregatorResult with all categories combined.

FloatScaleScorerByCategory¶

Namespace for float scale score aggregators that group by category.

These aggregators return multiple ScoreAggregatorResult objects (one per category). Useful for scorers like AzureContentFilterScorer that return multiple scores per item.

FloatScaleThresholdScorer¶

Bases: TrueFalseScorer

A scorer that applies a threshold to a float scale score to make it a true/false score.

Constructor Parameters:

GandalfScorer¶

Bases: TrueFalseScorer

A scorer for evaluating responses in Gandalf challenges.

This scorer checks whether a password or secret has been successfully extracted from a Gandalf-style challenge by submitting the response to the Gandalf API and checking if the password is correct. Returns True if the password was successfully extracted.

Constructor Parameters:

HarmHumanLabeledEntry¶

Bases: HumanLabeledEntry

A class that represents a human-labeled dataset entry for a specific harm category. This class includes the Messages and a list of human scores, which are floats between 0.0 and 1.0 inclusive, representing the degree of harm severity where 0.0 is minimal and 1.0 is maximal. The harm category is a string that represents the type of harm (e.g., “hate_speech”, “misinformation”, etc.).

HarmScorerEvaluator¶

Bases: ScorerEvaluator

A class that evaluates a harm scorer against HumanLabeledDatasets of type HARM.

HarmScorerMetrics¶

Bases: ScorerMetrics

Metrics for evaluating a harm scorer against a HumanLabeledDataset.

Methods:

get_harm_definition¶

get_harm_definition() → Optional[HarmDefinition]

Load and return the HarmDefinition object for this metrics instance.

Loads the harm definition YAML file specified in harm_definition and returns it as a HarmDefinition object. The result is cached after the first load.

Returns:

• Optional[HarmDefinition] — The loaded harm definition object, or None if harm_definition is not set.

Raises:

• FileNotFoundError — If the harm definition file does not exist.

• ValueError — If the harm definition file is invalid.

HumanInTheLoopScorerGradio¶

Bases: TrueFalseScorer

Create scores from manual human input using Gradio and adds them to the database.

In the future this will not be a TrueFalseScorer. However, it is all that is supported currently.

.. deprecated:: This Gradio-based scorer is deprecated and will be removed in v0.13.0. Use the React-based GUI instead.

Constructor Parameters:

Methods:

retrieve_score¶

retrieve_score(request_prompt: MessagePiece, objective: Optional[str] = None) → list[Score]

Retrieve a score from the human evaluator through the RPC server.

Returns:

• list[Score] — list[Score]: A list containing a single Score object from the human evaluator.

HumanLabeledDataset¶

A class that represents a human-labeled dataset, including the entries and each of their corresponding human scores. This dataset is used to evaluate PyRIT scorer performance via the ScorerEvaluator class. HumanLabeledDatasets can be constructed from a CSV file.

Constructor Parameters:

Methods:

from_csv¶

from_csv(csv_path: Union[str, Path], metrics_type: MetricsType, dataset_name: Optional[str] = None, version: Optional[str] = None, harm_definition: Optional[str] = None, harm_definition_version: Optional[str] = None) → HumanLabeledDataset

Load a human-labeled dataset from a CSV file with standard column names.

Expected CSV format:

• ‘assistant_response’: The assistant’s response text

• ‘human_score’: Human-assigned label (can have multiple columns for multiple raters)

• ‘objective’: For OBJECTIVE datasets, the objective being evaluated

• ‘data_type’: Optional data type (defaults to ‘text’ if not present)

You can optionally include a # comment line at the top of the CSV file to specify the dataset version and harm definition path. The format is:

• For harm datasets: # dataset_version=x.y, harm_definition=path/to/definition.yaml, harm_definition_version=x.y

• For objective datasets: # dataset_version=x.y

Returns:

• HumanLabeledDataset — The human-labeled dataset object.

Raises:

• FileNotFoundError — If the CSV file does not exist.

• ValueError — If version is not provided and not found in the CSV file.

get_harm_definition¶

get_harm_definition() → Optional[HarmDefinition]

Load and return the HarmDefinition object for this dataset.

For HARM datasets, this loads the harm definition YAML file specified in harm_definition and returns it as a HarmDefinition object. The result is cached after the first load.

Returns:

• Optional[HarmDefinition] — The loaded harm definition object, or None if this is not a HARM dataset or harm_definition is not set.

Raises:

• FileNotFoundError — If the harm definition file does not exist.

• ValueError — If the harm definition file is invalid.

validate¶

validate() → None

Validate that the dataset is internally consistent.

Checks that all entries match the dataset’s metrics_type and, for HARM datasets, that all entries have the same harm_category, that harm_definition is specified, and that the harm definition file exists and is loadable.

Raises:

• ValueError — If entries don’t match metrics_type, harm categories are inconsistent, or harm_definition is missing for HARM datasets.

• FileNotFoundError — If the harm definition file does not exist.

HumanLabeledEntry¶

A class that represents an entry in a dataset of assistant responses that have been scored by humans. It is used to evaluate PyRIT scorer performance as measured by degree of alignment with human labels. This class includes the Messages and a list of human-assigned scores, which are floats between 0.0 and 1.0 inclusive (representing degree of severity) for harm datasets, and booleans for objective datasets.

InsecureCodeScorer¶

Bases: FloatScaleScorer

A scorer that uses an LLM to evaluate code snippets for potential security vulnerabilities. Configuration is loaded from a YAML file for dynamic prompts and instructions.

Constructor Parameters:

LikertScaleEvalFiles¶

Configuration for evaluating a Likert scale scorer on a set of dataset files.

LikertScalePaths¶

Bases: enum.Enum

Enum containing Likert scale configurations including YAML paths and evaluation file mappings.

Each enum value is a tuple of (yaml_path, evaluation_files) where:

• yaml_path: Path to the YAML file containing the Likert scale definition

• evaluation_files: Optional LikertScaleEvalFiles for scorer evaluation, or None if no dataset exists

MarkdownInjectionScorer¶

Bases: TrueFalseScorer

A scorer that detects markdown injection attempts in text responses.

This scorer checks for the presence of markdown syntax patterns that could be used for injection attacks, such as links, images, or other markdown constructs that might be exploited. Returns True if markdown injection is detected.

Constructor Parameters:

MetricsType¶

Bases: Enum

Enum representing the type of metrics when evaluating scorers on human-labeled datasets.

ObjectiveHumanLabeledEntry¶

Bases: HumanLabeledEntry

A class that represents a human-labeled dataset entry for a specific objective. This class includes the Messages and a list of human scores, which are booleans indicating whether the response/conversation meets the objective (e.g., 0 for not meeting the objective, 1 for meeting the objective). The objective is a string that represents the objective (e.g., "how to make a Molotov cocktail?).

ObjectiveScorerEvaluator¶

Bases: ScorerEvaluator

A class that evaluates an objective scorer against HumanLabeledDatasets of type OBJECTIVE.

ObjectiveScorerMetrics¶

Bases: ScorerMetrics

Metrics for evaluating an objective scorer against a HumanLabeledDataset.

PlagiarismMetric¶

Bases: Enum

Enum representing different plagiarism detection metrics.

PlagiarismScorer¶

Bases: FloatScaleScorer

A scorer that measures plagiarism by computing word-level similarity between the AI response and a reference text.

This scorer implements three similarity metrics:

1. Word-level longest common subsequence (LCS)

2. Word-level Levenshtein similarity

3. Word-level n-gram Jaccard similarity

Constructor Parameters:

PromptShieldScorer¶

Bases: TrueFalseScorer

Returns true if an attack or jailbreak has been detected by Prompt Shield.

Constructor Parameters:

QuestionAnswerScorer¶

Bases: TrueFalseScorer

A class that represents a question answering scorer.

Constructor Parameters:

RefusalScorerPaths¶

Bases: enum.Enum

Paths to refusal scorer system prompt YAML files.

Each enum value represents a different refusal detection strategy:

• DEFAULT: Standard refusal detection that works with or without an explicit objective. If an objective is provided, evaluates refusal against it; if not, evaluates against the implied objective. Safe completions (including partial information, redirections, asking questions, or excessive caveats) are NOT considered refusals.

• STRICT: Strict refusal detection that treats “safe completions” as refusals. Works best when an explicit objective is provided.

RegistryUpdateBehavior¶

Bases: Enum

Enum representing how the evaluation registry should be updated.

Scorer¶

Bases: Identifiable, abc.ABC

Abstract base class for scorers.

Constructor Parameters:

Methods:

evaluate_async¶

evaluate_async(file_mapping: Optional[ScorerEvalDatasetFiles] = None, num_scorer_trials: int = 3, update_registry_behavior: RegistryUpdateBehavior = None, max_concurrency: int = 10) → Optional[ScorerMetrics]

Evaluate this scorer against human-labeled datasets.

Uses file mapping to determine which datasets to evaluate and how to aggregate results.

Returns:

• Optional[ScorerMetrics] — The evaluation metrics, or None if no datasets found.

Raises:

• ValueError — If no file_mapping is provided and no evaluation_file_mapping is configured.

get_identifier¶

get_identifier() → ComponentIdentifier

Get the scorer’s identifier with eval_hash always attached.

Overrides the base Identifiable.get_identifier() so that to_dict() always emits the eval_hash key.

Returns:

• ComponentIdentifier — The identity with eval_hash set.

get_scorer_metrics¶

get_scorer_metrics() → Optional[ScorerMetrics]

Get evaluation metrics for this scorer from the configured evaluation result file.

Looks up metrics by this scorer’s identity hash in the JSONL result file. The result file may contain entries for multiple scorer configurations.

Subclasses must implement this to return the appropriate metrics type:

• TrueFalseScorer subclasses should return ObjectiveScorerMetrics

• FloatScaleScorer subclasses should return HarmScorerMetrics

Returns:

• Optional[ScorerMetrics] — The metrics for this scorer, or None if not found or not configured.

scale_value_float¶

scale_value_float(value: float, min_value: float, max_value: float) → float

Scales a value from 0 to 1 based on the given min and max values. E.g. 3 stars out of 5 stars would be .5.

Returns:

• float — The scaled value.

score_async¶

score_async(message: Message, objective: Optional[str] = None, role_filter: Optional[ChatMessageRole] = None, skip_on_error_result: bool = False, infer_objective_from_request: bool = False) → list[Score]

Score the message, add the results to the database, and return a list of Score objects.

Returns:

• list[Score] — list[Score]: A list of Score objects representing the results.

Raises:

• PyritException — If scoring raises a PyRIT exception (re-raised with enhanced context).

• RuntimeError — If scoring raises a non-PyRIT exception (wrapped with scorer context).

score_image_async¶

score_image_async(image_path: str, objective: Optional[str] = None) → list[Score]

Score the given image using the chat target.

Returns:

• list[Score] — list[Score]: A list of Score objects representing the results.

score_image_batch_async¶

score_image_batch_async(image_paths: Sequence[str], objectives: Optional[Sequence[str]] = None, batch_size: int = 10) → list[Score]

Score a batch of images asynchronously.

Returns:

• list[Score] — list[Score]: A list of Score objects representing the scoring results for all images.

Raises:

• ValueError — If the number of objectives does not match the number of image_paths.

score_prompts_batch_async¶

score_prompts_batch_async(messages: Sequence[Message], objectives: Optional[Sequence[str]] = None, batch_size: int = 10, role_filter: Optional[ChatMessageRole] = None, skip_on_error_result: bool = False, infer_objective_from_request: bool = False) → list[Score]

Score multiple prompts in batches using the provided objectives.

Returns:

• list[Score] — list[Score]: A flattened list of Score objects from all scored prompts.

Raises:

• ValueError — If objectives is empty or if the number of objectives doesn’t match the number of messages.

score_response_async¶

score_response_async(response: Message, objective_scorer: Optional[Scorer] = None, auxiliary_scorers: Optional[list[Scorer]] = None, role_filter: ChatMessageRole = 'assistant', objective: Optional[str] = None, skip_on_error_result: bool = True) → dict[str, list[Score]]

Score a response using an objective scorer and optional auxiliary scorers.

Returns:

• dict[str, list[Score]] — Dict[str, List[Score]]: Dictionary with keys auxiliary_scores and objective_scores containing lists of scores from each type of scorer.

Raises:

• ValueError — If response is not provided.

score_response_multiple_scorers_async¶

score_response_multiple_scorers_async(response: Message, scorers: list[Scorer], role_filter: ChatMessageRole = 'assistant', objective: Optional[str] = None, skip_on_error_result: bool = True) → list[Score]

Score a response using multiple scorers in parallel.

This method applies each scorer to the first scorable response piece (filtered by role and error), and returns all scores. This is typically used for auxiliary scoring where all results are needed.

Returns:

• list[Score] — List[Score]: All scores from all scorers

score_text_async¶

score_text_async(text: str, objective: Optional[str] = None) → list[Score]

Scores the given text based on the task using the chat target.

Returns:

• list[Score] — list[Score]: A list of Score objects representing the results.

validate_return_scores¶

validate_return_scores(scores: list[Score]) → None

Validate the scores returned by the scorer. Because some scorers may require specific Score types or values.

ScorerEvalDatasetFiles¶

Configuration for evaluating a scorer on a set of dataset files.

Maps input dataset files (via glob patterns) to an output result file. Multiple files matching the patterns will be concatenated before evaluation.

ScorerEvaluator¶

Bases: abc.ABC

A class that evaluates an LLM scorer against HumanLabeledDatasets, calculating appropriate metrics and saving them to a file.

Constructor Parameters:

Methods:

evaluate_dataset_async¶

evaluate_dataset_async(labeled_dataset: HumanLabeledDataset, num_scorer_trials: int = 1, max_concurrency: int = 10) → ScorerMetrics

Run the evaluation for the scorer/policy combination on the passed in HumanLabeledDataset.

This method performs pure computation without side effects (no file writing). It can be called directly with an in-memory HumanLabeledDataset for experiments that don’t use file-based datasets (e.g., iterative rubric tuning with custom splits).

Returns:

• ScorerMetrics — The metrics for the scorer. This will be either HarmScorerMetrics or ObjectiveScorerMetrics depending on the type of the HumanLabeledDataset (HARM or OBJECTIVE).

Raises:

• ValueError — If the labeled_dataset is invalid.

from_scorer¶

from_scorer(scorer: Scorer, metrics_type: Optional[MetricsType] = None) → ScorerEvaluator

Create a ScorerEvaluator based on the type of scoring.

Returns:

• ScorerEvaluator — An instance of HarmScorerEvaluator or ObjectiveScorerEvaluator.

run_evaluation_async¶

run_evaluation_async(dataset_files: ScorerEvalDatasetFiles, num_scorer_trials: int = 3, update_registry_behavior: RegistryUpdateBehavior = RegistryUpdateBehavior.SKIP_IF_EXISTS, max_concurrency: int = 10) → Optional[ScorerMetrics]

Evaluate scorer using dataset files configuration.

The update_registry_behavior parameter controls how existing registry entries are handled:

• SKIP_IF_EXISTS (default): Check registry for existing results matching scorer config, dataset version, and num_scorer_trials. If found, return cached metrics. If not found, run evaluation and write to registry.

• ALWAYS_UPDATE: Always run evaluation and overwrite any existing registry entry.

• NEVER_UPDATE: Always run evaluation but never write to registry (for debugging).

Returns:

• Optional[ScorerMetrics] — ScorerMetrics if evaluation completed, None if no files found.

Raises:

• ValueError — If harm_category is not specified for harm scorer evaluations.

ScorerMetrics¶

Base dataclass for storing scorer evaluation metrics.

This class provides methods for serializing metrics to JSON and loading them from JSON files.

Methods:

from_json¶

from_json(file_path: Union[str, Path]) → T

Load the metrics from a JSON file.

Returns:

• T — An instance of ScorerMetrics with the loaded data.

Raises:

• FileNotFoundError — If the specified file does not exist.

to_json¶

to_json() → str

Convert the metrics to a JSON string.

Returns:

• str — The JSON string representation of the metrics.

ScorerMetricsWithIdentity¶

Bases: Generic[M]

Wrapper that combines scorer metrics with the scorer’s identity information.

This class provides a clean interface for working with evaluation results, allowing access to both the scorer configuration and its performance metrics.

Generic over the metrics type M, so:

• ScorerMetricsWithIdentity[ObjectiveScorerMetrics] has metrics: ObjectiveScorerMetrics

• ScorerMetricsWithIdentity[HarmScorerMetrics] has metrics: HarmScorerMetrics

ScorerPrinter¶

Bases: ABC

Abstract base class for printing scorer information.

This interface defines the contract for printing scorer details including type information, nested sub-scorers, and evaluation metrics from the registry. Implementations can render output to console, logs, files, or other outputs.

Methods:

print_harm_scorer¶

print_harm_scorer(scorer_identifier: ComponentIdentifier, harm_category: str) → None

Print harm scorer information including type, nested scorers, and evaluation metrics.

This method displays:

• Scorer type and identity information

• Nested sub-scorers (for composite scorers)

• Harm evaluation metrics (MAE, Krippendorff alpha) from the registry

print_objective_scorer¶

print_objective_scorer(scorer_identifier: ComponentIdentifier) → None

Print objective scorer information including type, nested scorers, and evaluation metrics.

This method displays:

• Scorer type and identity information

• Nested sub-scorers (for composite scorers)

• Objective evaluation metrics (accuracy, precision, recall, F1) from the registry

ScorerPromptValidator¶

Validates message pieces and scorer configurations.

This class provides validation for scorer inputs, ensuring that message pieces meet required criteria such as data types, roles, and metadata requirements.

Constructor Parameters:

Methods:

is_message_piece_supported¶

is_message_piece_supported(message_piece: MessagePiece) → bool

Check if a message piece is supported by this validator.

Returns:

• bool — True if the message piece meets all validation criteria, False otherwise.

validate¶

validate(message: Message, objective: str | None) → None

Validate a message and objective against configured requirements.

Raises:

• ValueError — If validation fails due to unsupported pieces, exceeding max pieces, or missing objective.

SelfAskCategoryScorer¶

Bases: TrueFalseScorer

A class that represents a self-ask score for text classification and scoring. Given a classifier file, it scores according to these categories and returns the category the MessagePiece fits best.

There is also a false category that is used if the MessagePiece does not fit any of the categories.

Constructor Parameters:

SelfAskGeneralFloatScaleScorer¶

Bases: FloatScaleScorer

A general-purpose self-ask float-scale scorer that uses a chat target and a configurable system prompt and prompt format. The final score is normalized to [0, 1].

Constructor Parameters:

SelfAskGeneralTrueFalseScorer¶

Bases: TrueFalseScorer

A general-purpose self-ask True/False scorer that uses a chat target and a configurable system prompt and prompt format.

Constructor Parameters:

SelfAskLikertScorer¶

Bases: FloatScaleScorer

A class that represents a “self-ask” score for text scoring based on a Likert scale. A Likert scale consists of ranked, ordered categories and is often on a 5 or 7 point basis, but you can configure a scale with any set of non-negative integer score values and descriptions by providing a custom YAML file.

Constructor Parameters:

SelfAskQuestionAnswerScorer¶

Bases: SelfAskTrueFalseScorer

A class that represents a self-ask question answering scorer.

Usually, QuestionAnswerScorer should be used, but this can be useful for when choices are not sent to objective target or you need more flexibility in determining if the questions were answered correctly.

Constructor Parameters:

SelfAskRefusalScorer¶

Bases: TrueFalseScorer

A self-ask scorer that detects refusal in AI responses.

This scorer uses a language model to determine whether a response contains a refusal to answer or comply with the given prompt. It’s useful for evaluating whether AI systems are appropriately refusing harmful requests.

The scorer supports two modes via RefusalScorerPaths:

• DEFAULT: Works with or without an explicit objective. Safe completions are NOT considered refusals.

• STRICT: Treats safe completions as refusals. Works best with an explicit objective.

Constructor Parameters:

SelfAskScaleScorer¶

Bases: FloatScaleScorer

A class that represents a “self-ask” score for text scoring for a customizable numeric scale.

Constructor Parameters:

SelfAskTrueFalseScorer¶

Bases: TrueFalseScorer

A class that represents a self-ask true/false for scoring.

Given written descriptions of “true” and “false” (passed as a file or a TrueFalseQuestion), it returns the value that matches either description most closely.

If no descriptions are provided, it defaults to the TASK_ACHIEVED scorer.

Constructor Parameters:

SubStringScorer¶

Bases: TrueFalseScorer

Scorer that checks if a given substring is present in the text.

This scorer performs substring matching using a configurable text matching strategy. Supports both exact substring matching and approximate matching.

Constructor Parameters:

TrueFalseAggregatorFunc¶

TrueFalseCompositeScorer¶

Bases: TrueFalseScorer

Composite true/false scorer that aggregates results from other true/false scorers.

This scorer invokes a collection of constituent TrueFalseScorer instances and reduces their single-score outputs into one final true/false score using the supplied aggregation function (e.g., TrueFalseScoreAggregator.AND, TrueFalseScoreAggregator.OR, TrueFalseScoreAggregator.MAJORITY).

Constructor Parameters:

TrueFalseInverterScorer¶

Bases: TrueFalseScorer

A scorer that inverts a true false score.

Constructor Parameters:

TrueFalseQuestion¶

A class that represents a true/false question.

This is sent to an LLM and can be used as an alternative to a yaml file from TrueFalseQuestionPaths.

Constructor Parameters:

TrueFalseQuestionPaths¶

Bases: enum.Enum

Paths to true/false question YAML files.

TrueFalseScoreAggregator¶

Namespace for true/false score aggregators that return a single aggregated score.

All aggregators return a list containing one ScoreAggregatorResult that combines all input scores together, preserving all categories.

TrueFalseScorer¶

Bases: Scorer

Base class for scorers that return true/false binary scores.

This scorer evaluates prompt responses and returns a single boolean score indicating whether the response meets a specific criterion. Multiple pieces in a request response are aggregated using a TrueFalseAggregatorFunc function (default: TrueFalseScoreAggregator.OR).

Constructor Parameters:

Methods:

get_scorer_metrics¶

get_scorer_metrics() → Optional[ObjectiveScorerMetrics]

Get evaluation metrics for this scorer from the configured evaluation result file.

Returns:

• Optional[ObjectiveScorerMetrics] — The metrics for this scorer, or None if not found or not configured.

validate_return_scores¶

validate_return_scores(scores: list[Score]) → None

Validate the scores returned by the scorer.

Raises:

• ValueError — If the number of scores is not exactly one.

• ValueError — If the score value is not “true” or “false”.

VideoFloatScaleScorer¶

Bases: FloatScaleScorer, _BaseVideoScorer

A scorer that processes videos by extracting frames and scoring them using a float scale image scorer.

The VideoFloatScaleScorer breaks down a video into frames and uses a float scale scoring mechanism. Frame scores are aggregated using a FloatScaleAggregatorFunc.

By default, uses FloatScaleScorerByCategory.MAX which groups scores by category (useful for scorers like AzureContentFilterScorer that return multiple scores per frame). This returns one aggregated score per category (e.g., one for “Hate”, one for “Violence”, etc.).

For scorers that return a single score per frame, or to combine all categories together, use FloatScaleScoreAggregator.MAX, FloatScaleScorerAllCategories.MAX, etc.

Optionally, an audio_scorer can be provided to also score the video’s audio track. When provided, the audio is extracted, transcribed, and scored. The audio scores are included in the aggregation.

Constructor Parameters:

VideoTrueFalseScorer¶

Bases: TrueFalseScorer, _BaseVideoScorer

A scorer that processes videos by extracting frames and scoring them using a true/false image scorer.

Aggregation Logic (hard-coded): - Frame scores are aggregated using OR: if ANY frame meets the objective, the visual score is True. - When audio_scorer is provided, the final score uses AND: BOTH visual (frames) AND audio must be True for the overall video score to be True.

Constructor Parameters: