pyrit.score.VideoFloatScaleScorer

pyrit.score.VideoFloatScaleScorer#

class VideoFloatScaleScorer(*, image_capable_scorer: ~pyrit.score.float_scale.float_scale_scorer.FloatScaleScorer, num_sampled_frames: int | None = None, validator: ~pyrit.score.scorer_prompt_validator.ScorerPromptValidator | None = None, score_aggregator: ~typing.Callable[[~typing.Iterable[~pyrit.models.score.Score]], ~typing.List[~pyrit.score.score_aggregator_result.ScoreAggregatorResult]] = <function _create_aggregator_by_category.<locals>.aggregator>)[source]#

Bases: FloatScaleScorer, _BaseVideoScorer

A scorer that processes videos by extracting frames and scoring them using a float scale image scorer.

The VideoFloatScaleScorer breaks down a video into frames and uses a float scale scoring mechanism. Frame scores are aggregated using a FloatScaleAggregatorFunc.

By default, uses FloatScaleScorerByCategory.MAX which groups scores by category (useful for scorers like AzureContentFilterScorer that return multiple scores per frame). This returns one aggregated score per category (e.g., one for “Hate”, one for “Violence”, etc.).

For scorers that return a single score per frame, or to combine all categories together, use FloatScaleScoreAggregator.MAX, FloatScaleScorerAllCategories.MAX, etc.

__init__(*, image_capable_scorer: ~pyrit.score.float_scale.float_scale_scorer.FloatScaleScorer, num_sampled_frames: int | None = None, validator: ~pyrit.score.scorer_prompt_validator.ScorerPromptValidator | None = None, score_aggregator: ~typing.Callable[[~typing.Iterable[~pyrit.models.score.Score]], ~typing.List[~pyrit.score.score_aggregator_result.ScoreAggregatorResult]] = <function _create_aggregator_by_category.<locals>.aggregator>) → None[source]#

Initialize the VideoFloatScaleScorer.

Parameters:

  • image_capable_scorer – A FloatScaleScorer capable of processing images.

  • num_sampled_frames – Number of frames to extract from the video for scoring (default: 5).

  • validator – Validator for the scorer. Defaults to video_path data type validator.

  • score_aggregator – Aggregator for combining frame scores. Defaults to FloatScaleScorerByCategory.MAX. Use FloatScaleScorerByCategory.MAX/AVERAGE/MIN for scorers that return multiple scores per frame (groups by category and returns one score per category). Use FloatScaleScorerAllCategories.MAX/AVERAGE/MIN to combine all scores regardless of category (returns single score with all categories combined). Use FloatScaleScoreAggregator.MAX/AVERAGE/MIN for simple aggregation preserving all categories (returns single score with all categories preserved).

Methods

__init__(*, image_capable_scorer[, ...])

Initialize the VideoFloatScaleScorer.

get_identifier()

Get an identifier dictionary for the scorer for database storage.

get_scorer_metrics(dataset_name[, metrics_type])

Get evaluation statistics for the scorer using the dataset_name of the human labeled dataset.

scale_value_float(value, min_value, max_value)

Scales a value from 0 to 1 based on the given min and max values.

score_async(message, *[, objective, ...])

Score the message, add the results to the database, and return a list of Score objects.

score_image_async(image_path, *[, objective])

Score the given image using the chat target.

score_image_batch_async(*, image_paths[, ...])

Score a batch of images asynchronously.

score_prompts_batch_async(*, messages[, ...])

Score multiple prompts in batches using the provided objectives.

score_response_async(*, response[, ...])

Score a response using an objective scorer and optional auxiliary scorers.

score_response_multiple_scorers_async(*, ...)

Score a response using multiple scorers in parallel.

score_text_async(text, *[, objective])

Scores the given text based on the task using the chat target.

validate_return_scores(scores)

Validate that the returned scores are within the valid range [0, 1].

Attributes

scorer_identifier

Get the scorer identifier.

scorer_type

Contents