pyrit.score.BatchScorer

pyrit.score.BatchScorer#

class BatchScorer(*, batch_size: int = 10)[source]#

Bases: object

A utility class for scoring prompts in batches in a parallelizable and convenient way.

This class provides functionality to score existing prompts stored in memory without any target interaction, making it a pure scoring utility.

__init__(*, batch_size: int = 10) → None[source]#

Initialize the BatchScorer.

Parameters:

batch_size (int) – The (max) batch size for sending prompts. Defaults to 10. Note: If using a scorer that takes a prompt target, and providing max requests per minute on the target, this should be set to 1 to ensure proper rate limit management.

Methods

__init__(*[, batch_size])

Initialize the BatchScorer.

score_prompts_by_id_async(*, scorer, prompt_ids)

Score prompts using the Scorer for prompts with the prompt_ids.

score_responses_by_filters_async(*, scorer)

Score the responses that match the specified filters.
async score_prompts_by_id_async(*, scorer: Scorer, prompt_ids: list[str], responses_only: bool = False, task: str = '') → list[Score][source]#

Score prompts using the Scorer for prompts with the prompt_ids.

Use this function if you want to score prompt requests as well as prompt responses, or if you want more fine-grained control over the scorer tasks. If you only want to score prompt responses, use the score_responses_by_filters_async function.

Parameters:

  • scorer (Scorer) – The Scorer object to use for scoring.

  • prompt_ids (list[str]) – A list of prompt IDs correlating to the prompts to score.

  • responses_only (bool) – If True, only the responses (messages with role “assistant”) are scored. Defaults to False.

  • task (str) – A task is used to give the scorer more context on what exactly to score. A task might be the request prompt text or the original attack model’s objective. Note: the same task is to applied to all prompt_ids. Defaults to an empty string.

Returns:

A list of Score objects for the prompts with the prompt_ids.

Return type:

list[Score]

async score_responses_by_filters_async(*, scorer: Scorer, orchestrator_id: str | UUID | None = None, conversation_id: str | UUID | None = None, prompt_ids: list[str] | list[UUID] | None = None, labels: dict[str, str] | None = None, sent_after: datetime | None = None, sent_before: datetime | None = None, original_values: list[str] | None = None, converted_values: list[str] | None = None, data_type: str | None = None, not_data_type: str | None = None, converted_value_sha256: list[str] | None = None) → list[Score][source]#

Score the responses that match the specified filters.

Parameters:

  • scorer (Scorer) – The Scorer object to use for scoring.

  • orchestrator_id (Optional[str | uuid.UUID]) – The ID of the orchestrator. Defaults to None.

  • conversation_id (Optional[str | uuid.UUID]) – The ID of the conversation. Defaults to None.

  • prompt_ids (Optional[list[str] | list[uuid.UUID]]) – A list of prompt IDs. Defaults to None.

  • labels (Optional[dict[str, str]]) – A dictionary of labels. Defaults to None.

  • sent_after (Optional[datetime]) – Filter for prompts sent after this datetime. Defaults to None.

  • sent_before (Optional[datetime]) – Filter for prompts sent before this datetime. Defaults to None.

  • original_values (Optional[list[str]]) – A list of original values. Defaults to None.

  • converted_values (Optional[list[str]]) – A list of converted values. Defaults to None.

  • data_type (Optional[str]) – The data type to filter by. Defaults to None.

  • not_data_type (Optional[str]) – The data type to exclude. Defaults to None.

  • converted_value_sha256 (Optional[list[str]]) – A list of SHA256 hashes of converted values. Defaults to None.

Returns:

A list of Score objects for responses that match the specified filters.

Return type:

list[Score]

Raises:

ValueError – If no entries match the provided filters.

Contents