pyrit.score.VideoFloatScaleScorer

pyrit.score.VideoFloatScaleScorer#

class VideoFloatScaleScorer(*, image_capable_scorer: ~pyrit.score.float_scale.float_scale_scorer.FloatScaleScorer, audio_scorer: ~pyrit.score.float_scale.float_scale_scorer.FloatScaleScorer | None = None, num_sampled_frames: int | None = None, validator: ~pyrit.score.scorer_prompt_validator.ScorerPromptValidator | None = None, score_aggregator: ~typing.Callable[[~typing.Iterable[~pyrit.models.score.Score]], ~typing.List[~pyrit.score.score_aggregator_result.ScoreAggregatorResult]] = <function _create_aggregator_by_category.<locals>.aggregator>, image_objective_template: str | None = 'The objective is: {objective}\n\nThis request includes an image from the resultant video.', audio_objective_template: str | None = None)[source]#

Bases: FloatScaleScorer, _BaseVideoScorer

A scorer that processes videos by extracting frames and scoring them using a float scale image scorer.

The VideoFloatScaleScorer breaks down a video into frames and uses a float scale scoring mechanism. Frame scores are aggregated using a FloatScaleAggregatorFunc.

By default, uses FloatScaleScorerByCategory.MAX which groups scores by category (useful for scorers like AzureContentFilterScorer that return multiple scores per frame). This returns one aggregated score per category (e.g., one for “Hate”, one for “Violence”, etc.).

For scorers that return a single score per frame, or to combine all categories together, use FloatScaleScoreAggregator.MAX, FloatScaleScorerAllCategories.MAX, etc.

Optionally, an audio_scorer can be provided to also score the video’s audio track. When provided, the audio is extracted, transcribed, and scored. The audio scores are included in the aggregation.

__init__(*, image_capable_scorer: ~pyrit.score.float_scale.float_scale_scorer.FloatScaleScorer, audio_scorer: ~pyrit.score.float_scale.float_scale_scorer.FloatScaleScorer | None = None, num_sampled_frames: int | None = None, validator: ~pyrit.score.scorer_prompt_validator.ScorerPromptValidator | None = None, score_aggregator: ~typing.Callable[[~typing.Iterable[~pyrit.models.score.Score]], ~typing.List[~pyrit.score.score_aggregator_result.ScoreAggregatorResult]] = <function _create_aggregator_by_category.<locals>.aggregator>, image_objective_template: str | None = 'The objective is: {objective}\n\nThis request includes an image from the resultant video.', audio_objective_template: str | None = None) → None[source]#

Initialize the VideoFloatScaleScorer.

Parameters:

  • image_capable_scorer – A FloatScaleScorer capable of processing images.

  • audio_scorer – Optional FloatScaleScorer for scoring the video’s audio track. When provided, audio is extracted from the video, transcribed to text, and scored. The audio scores are aggregated with frame scores.

  • num_sampled_frames – Number of frames to extract from the video for scoring (default: 5).

  • validator – Validator for the scorer. Defaults to video_path data type validator.

  • score_aggregator – Aggregator for combining frame scores. Defaults to FloatScaleScorerByCategory.MAX. Use FloatScaleScorerByCategory.MAX/AVERAGE/MIN for scorers that return multiple scores per frame (groups by category and returns one score per category). Use FloatScaleScorerAllCategories.MAX/AVERAGE/MIN to combine all scores regardless of category (returns single score with all categories combined). Use FloatScaleScoreAggregator.MAX/AVERAGE/MIN for simple aggregation preserving all categories (returns single score with all categories preserved).

  • image_objective_template – Template for formatting the objective when scoring image frames. Use {objective} as placeholder for the actual objective. Set to None to not pass objective to image scorer. Defaults to a template that provides context about the video frame.

  • audio_objective_template – Template for formatting the objective when scoring audio. Use {objective} as placeholder for the actual objective. Set to None to not pass objective to audio scorer. Defaults to None because video objectives typically describe visual content that doesn’t apply to audio.

Raises:

ValueError – If audio_scorer is provided and does not support audio_path data type.

Methods

__init__(*, image_capable_scorer[, ...])

Initialize the VideoFloatScaleScorer.

evaluate_async([file_mapping, ...])

Evaluate this scorer against human-labeled datasets.

get_identifier()

Get the typed identifier for this object.

get_scorer_metrics()

Get evaluation metrics for this scorer from the configured evaluation result file.

scale_value_float(value, min_value, max_value)

Scales a value from 0 to 1 based on the given min and max values.

score_async(message, *[, objective, ...])

Score the message, add the results to the database, and return a list of Score objects.

score_image_async(image_path, *[, objective])

Score the given image using the chat target.

score_image_batch_async(*, image_paths[, ...])

Score a batch of images asynchronously.

score_prompts_batch_async(*, messages[, ...])

Score multiple prompts in batches using the provided objectives.

score_response_async(*, response[, ...])

Score a response using an objective scorer and optional auxiliary scorers.

score_response_multiple_scorers_async(*, ...)

Score a response using multiple scorers in parallel.

score_text_async(text, *[, objective])

Scores the given text based on the task using the chat target.

validate_return_scores(scores)

Validate that the returned scores are within the valid range [0, 1].

Attributes

evaluation_file_mapping

scorer_type

Get the scorer type based on class hierarchy.
Contents