pyrit.orchestrator.ScoringOrchestrator

pyrit.orchestrator.ScoringOrchestrator#

class ScoringOrchestrator(batch_size: int = 10, verbose: bool = False)[source]#

Bases: Orchestrator

This orchestrator scores prompts in a parallelizable and convenient way.

__init__(batch_size: int = 10, verbose: bool = False) → None[source]#
Parameters:

batch_size (int, Optional) – The (max) batch size for sending prompts. Defaults to 10. Note: If using a scorer that takes a prompt target, and providing max requests per minute on the target, this should be set to 1 to ensure proper rate limit management.

Methods

__init__([batch_size, verbose])

dispose_db_engine()

Dispose database engine to release database connections and resources.

get_identifier()

get_memory()

Retrieves the memory associated with this orchestrator.

get_score_memory()

Retrieves the scores of the PromptRequestPieces associated with this orchestrator.

score_prompts_by_id_async(*, scorer, prompt_ids)

Scores prompts using the Scorer for prompts with the prompt_ids.

score_responses_by_filters_async(*, scorer)

Scores the responses that match the specified filters.
async score_prompts_by_id_async(*, scorer: Scorer, prompt_ids: list[str], responses_only: bool = False, task: str = '') → list[Score][source]#

Scores prompts using the Scorer for prompts with the prompt_ids. Use this function if you want to score prompt requests as well as prompt responses, or if you want more fine-grained control over the scorer tasks. If you only want to score prompt responses, use the score_responses_by_filters_async function.

Parameters:

  • scorer (Scorer) – The Scorer object to use for scoring.

  • prompt_ids (list[str]) – A list of prompt IDs correlating to the prompts to score.

  • responses_only (bool, optional) – If True, only the responses (messages with role “assistant”) are scored. Defaults to False.

  • task (str, optional) – A task is used to give the scorer more context on what exactly to score. A task might be the request prompt text or the original attack model’s objective. Note: the same task is to applied to all prompt_ids. Defaults to an empty string.

Returns:

A list of Score objects for the prompts with the prompt_ids.

Return type:

list[Score]

async score_responses_by_filters_async(*, scorer: Scorer, orchestrator_id: str | UUID | None = None, conversation_id: str | UUID | None = None, prompt_ids: list[str] | list[UUID] | None = None, labels: dict[str, str] | None = None, sent_after: datetime | None = None, sent_before: datetime | None = None, original_values: list[str] | None = None, converted_values: list[str] | None = None, data_type: str | None = None, not_data_type: str | None = None, converted_value_sha256: list[str] | None = None) → list[Score][source]#

Scores the responses that match the specified filters.

Parameters:

  • scorer (Scorer) – The Scorer object to use for scoring.

  • orchestrator_id (Optional[str | uuid.UUID], optional) – The ID of the orchestrator. Defaults to None.

  • conversation_id (Optional[str | uuid.UUID], optional) – The ID of the conversation. Defaults to None.

  • prompt_ids (Optional[list[str] | list[uuid.UUID]], optional) – A list of prompt IDs. Defaults to None.

  • labels (Optional[dict[str, str]], optional) – A dictionary of labels. Defaults to None.

  • sent_after (Optional[datetime], optional) – Filter for prompts sent after this datetime. Defaults to None.

  • sent_before (Optional[datetime], optional) – Filter for prompts sent before this datetime. Defaults to None.

  • original_values (Optional[list[str]], optional) – A list of original values. Defaults to None.

  • converted_values (Optional[list[str]], optional) – A list of converted values. Defaults to None.

  • data_type (Optional[str], optional) – The data type to filter by. Defaults to None.

  • not_data_type (Optional[str], optional) – The data type to exclude. Defaults to None.

  • converted_value_sha256 (Optional[list[str]], optional) – A list of SHA256 hashes of converted values. Defaults to None.

Returns:

A list of Score objects for responses that match the specified filters.

Return type:

list[Score]

Raises:

Exception – If there is an error retrieving the prompts, an exception is logged and an empty list is returned.