pyrit.models.ScenarioResult

pyrit.models.ScenarioResult#

class ScenarioResult(*, scenario_identifier: ScenarioIdentifier, objective_target_identifier: dict[str, Any] | ComponentIdentifier, attack_results: dict[str, list[AttackResult]], objective_scorer_identifier: dict[str, Any] | ComponentIdentifier, scenario_run_state: Literal['CREATED', 'IN_PROGRESS', 'COMPLETED', 'FAILED'] = 'CREATED', labels: dict[str, str] | None = None, completion_time: datetime | None = None, number_tries: int = 0, id: UUID | None = None, objective_scorer: Scorer | None = None)[source]#

Bases: object

Scenario result class for aggregating scenario results.

__init__(*, scenario_identifier: ScenarioIdentifier, objective_target_identifier: dict[str, Any] | ComponentIdentifier, attack_results: dict[str, list[AttackResult]], objective_scorer_identifier: dict[str, Any] | ComponentIdentifier, scenario_run_state: Literal['CREATED', 'IN_PROGRESS', 'COMPLETED', 'FAILED'] = 'CREATED', labels: dict[str, str] | None = None, completion_time: datetime | None = None, number_tries: int = 0, id: UUID | None = None, objective_scorer: Scorer | None = None) → None[source]#

Initialize a scenario result.

Parameters:

  • scenario_identifier (ScenarioIdentifier) – Identifier for the executed scenario.

  • objective_target_identifier (Union[Dict[str, Any], TargetIdentifier]) – Target identifier.

  • attack_results (dict[str, List[AttackResult]]) – Results grouped by atomic attack name.

  • objective_scorer_identifier (Union[Dict[str, Any], ScorerIdentifier]) – Objective scorer identifier.

  • scenario_run_state (ScenarioRunState) – Current scenario run state.

  • labels (Optional[dict[str, str]]) – Optional labels.

  • completion_time (Optional[datetime]) – Optional completion timestamp.

  • number_tries (int) – Number of run attempts.

  • id (Optional[uuid.UUID]) – Optional scenario result ID.

  • objective_scorer (Optional[Scorer]) – Deprecated scorer object parameter.

Methods

__init__(*, scenario_identifier, ...[, ...])

Initialize a scenario result.

get_objectives(*[, atomic_attack_name])

Get the list of unique objectives for this scenario.

get_scorer_evaluation_metrics()

Get the evaluation metrics for the scenario's scorer from the scorer evaluation registry.

get_strategies_used()

Get the list of strategies used in this scenario.

normalize_scenario_name(scenario_name)

Normalize a scenario name to match the stored class name format.

objective_achieved_rate(*[, atomic_attack_name])

Get the success rate of this scenario.
get_objectives(*, atomic_attack_name: str | None = None) → list[str][source]#

Get the list of unique objectives for this scenario.

Parameters:

atomic_attack_name (Optional[str]) – Name of specific atomic attack to include. If None, includes objectives from all atomic attacks. Defaults to None.

Returns:

Deduplicated list of objectives.

Return type:

List[str]

get_scorer_evaluation_metrics() → ScorerMetrics | None[source]#

Get the evaluation metrics for the scenario’s scorer from the scorer evaluation registry.

Returns:

The evaluation metrics object, or None if not found.

Return type:

ScorerMetrics

get_strategies_used() → list[str][source]#

Get the list of strategies used in this scenario.

Returns:

Atomic attack strategy names present in the results.

Return type:

List[str]

static normalize_scenario_name(scenario_name: str) → str[source]#

Normalize a scenario name to match the stored class name format.

Converts CLI-style snake_case names (e.g., “foundry” or “content_harms”) to PascalCase class names (e.g., “Foundry” or “ContentHarms”) for database queries. If the input is already in PascalCase or doesn’t match the snake_case pattern, it is returned unchanged.

This is the inverse of ScenarioRegistry._class_name_to_scenario_name().

Parameters:

scenario_name – The scenario name to normalize.

Returns:

The normalized scenario name suitable for database queries.

objective_achieved_rate(*, atomic_attack_name: str | None = None) → int[source]#

Get the success rate of this scenario.

Parameters:

atomic_attack_name (Optional[str]) – Name of specific atomic attack to calculate rate for. If None, calculates rate across all atomic attacks. Defaults to None.

Returns:

Success rate as a percentage (0-100).

Return type:

int

Contents