pyrit.score.VideoTrueFalseScorer

pyrit.score.VideoTrueFalseScorer#

class VideoTrueFalseScorer(*, image_capable_scorer: TrueFalseScorer, audio_scorer: TrueFalseScorer | None = None, num_sampled_frames: int | None = None, validator: ScorerPromptValidator | None = None, image_objective_template: str | None = 'The objective is: {objective}\n\nThis request includes an image from the resultant video.', audio_objective_template: str | None = None)[source]#

Bases: TrueFalseScorer, _BaseVideoScorer

A scorer that processes videos by extracting frames and scoring them using a true/false image scorer.

Aggregation Logic (hard-coded):

  • Frame scores are aggregated using OR: if ANY frame meets the objective, the visual score is True.

  • When audio_scorer is provided, the final score uses AND: BOTH visual (frames) AND audio must be True for the overall video score to be True.

This means:

  • Video-only scoring: True if any frame matches the objective

  • Video + Audio scoring: True only if both video frames AND audio transcript match their objectives

__init__(*, image_capable_scorer: TrueFalseScorer, audio_scorer: TrueFalseScorer | None = None, num_sampled_frames: int | None = None, validator: ScorerPromptValidator | None = None, image_objective_template: str | None = 'The objective is: {objective}\n\nThis request includes an image from the resultant video.', audio_objective_template: str | None = None) → None[source]#

Initialize the VideoTrueFalseScorer.

Parameters:

  • image_capable_scorer – A TrueFalseScorer capable of processing images.

  • audio_scorer – Optional TrueFalseScorer for scoring the video’s audio track. When provided, audio is extracted from the video and scored. The final score requires BOTH video frames AND audio to be True.

  • num_sampled_frames – Number of frames to extract from the video for scoring (default: 5).

  • validator – Validator for the scorer. Defaults to video_path data type validator.

  • image_objective_template – Template for formatting the objective when scoring image frames. Use {objective} as placeholder for the actual objective. Set to None to not pass objective to image scorer. Defaults to a template that provides context about the video frame.

  • audio_objective_template – Template for formatting the objective when scoring audio. Use {objective} as placeholder for the actual objective. Set to None to not pass objective to audio scorer. Defaults to None because video objectives typically describe visual content that doesn’t apply to audio.

Raises:

ValueError – If audio_scorer is provided and does not support audio_path data type.

Methods

__init__(*, image_capable_scorer[, ...])

Initialize the VideoTrueFalseScorer.

evaluate_async([file_mapping, ...])

Evaluate this scorer against human-labeled datasets.

get_identifier()

Get the typed identifier for this object.

get_scorer_metrics()

Get evaluation metrics for this scorer from the configured evaluation result file.

scale_value_float(value, min_value, max_value)

Scales a value from 0 to 1 based on the given min and max values.

score_async(message, *[, objective, ...])

Score the message, add the results to the database, and return a list of Score objects.

score_image_async(image_path, *[, objective])

Score the given image using the chat target.

score_image_batch_async(*, image_paths[, ...])

Score a batch of images asynchronously.

score_prompts_batch_async(*, messages[, ...])

Score multiple prompts in batches using the provided objectives.

score_response_async(*, response[, ...])

Score a response using an objective scorer and optional auxiliary scorers.

score_response_multiple_scorers_async(*, ...)

Score a response using multiple scorers in parallel.

score_text_async(text, *[, objective])

Scores the given text based on the task using the chat target.

validate_return_scores(scores)

Validate the scores returned by the scorer.

Attributes

evaluation_file_mapping

scorer_type

Get the scorer type based on class hierarchy.
Contents