Scenarios

Scenarios#

A Scenario is a higher-level construct that groups multiple Attack Configurations together. This allows you to execute a comprehensive testing campaign with multiple attack methods sequentially. Scenarios are meant to be configured and written to test for specific workflows. As such, it is okay to hard code some values.

What is a Scenario?#

A Scenario represents a comprehensive testing campaign composed of multiple atomic attack tests. It orchestrates the execution of multiple AtomicAttack instances sequentially and aggregates the results into a single ScenarioResult.

Key Components#

  • Scenario: The top-level orchestrator that groups and executes multiple atomic attacks

  • AtomicAttack: An atomic test unit combining an attack strategy, objectives, and execution parameters

  • ScenarioResult: Contains the aggregated results from all atomic attacks and scenario metadata

Use Cases#

Some examples of scenarios you might create:

  • VibeCheckScenario: Randomly selects a few prompts from HarmBench to quickly assess model behavior

  • QuickViolence: Checks how resilient a model is to violent objectives using multiple attack techniques

  • ComprehensiveFoundry: Tests a target with all available attack converters and strategies

  • CustomCompliance: Tests against specific compliance requirements with curated datasets and attacks

These Scenarios can be updated and added to as you refine what you are testing for.

How to Run Scenarios#

Scenarios should take almost no effort to run with default values. pyrit_scan and pyrit_shell both use scenarios to execute.

How It Works#

Each Scenario contains a collection of AtomicAttack objects. When executed:

  1. Each AtomicAttack is executed sequentially

  2. Every AtomicAttack tests its configured attack against all specified objectives and datasets

  3. Results are aggregated into a single ScenarioResult with all attack outcomes

  4. Optional memory labels help track and categorize the scenario execution

Creating Custom Scenarios#

To create a custom scenario, extend the Scenario base class and implement the required abstract methods.

Required Components#

  1. Strategy Enum: Create a ScenarioStrategy enum that defines the available strategies for your scenario.

    • Each enum member is defined as (value, tags) where value is a string and tags is a set of strings

    • Include an ALL aggregate strategy that expands to all available strategies

    • Optionally implement supports_composition() and validate_composition() for strategy composition rules

  2. Scenario Class: Extend Scenario and implement these abstract methods:

    • get_strategy_class(): Return your strategy enum class

    • get_default_strategy(): Return the default strategy (typically YourStrategy.ALL)

    • _get_atomic_attacks_async(): Build and return a list of AtomicAttack instances

  3. Constructor: Use @apply_defaults decorator and call super().__init__() with scenario metadata:

    • name: Descriptive name for your scenario

    • version: Integer version number

    • strategy_class: The strategy enum class for this scenario

    • objective_scorer_identifier: Identifier dict for the scoring mechanism (optional)

    • include_default_baseline: Whether to include a baseline attack (default: True)

    • scenario_result_id: Optional ID to resume an existing scenario (optional)

  4. Initialization: Call await scenario.initialize_async() to populate atomic attacks:

    • objective_target: The target system being tested (required)

    • scenario_strategies: List of strategies to execute (optional, defaults to ALL)

    • max_concurrency: Number of concurrent operations (default: 1)

    • max_retries: Number of retry attempts on failure (default: 0)

    • memory_labels: Optional labels for tracking (optional)

Example Structure#

from typing import Optional

from pyrit.common import apply_defaults
from pyrit.executor.attack import AttackScoringConfig, PromptSendingAttack
from pyrit.scenario import (
    AtomicAttack,
    DatasetConfiguration,
    Scenario,
    ScenarioStrategy,
)
from pyrit.scenario.core.scenario_strategy import ScenarioCompositeStrategy
from pyrit.score.true_false.true_false_scorer import TrueFalseScorer
from pyrit.setup import initialize_pyrit_async

await initialize_pyrit_async(memory_db_type="InMemory")  # type: ignore [top-level-await]


class MyStrategy(ScenarioStrategy):
    ALL = ("all", {"all"})
    StrategyA = ("strategy_a", {"tag1", "tag2"})
    StrategyB = ("strategy_b", {"tag1"})


class MyScenario(Scenario):
    version: int = 1

    # A strategy defintion helps callers define how to run your scenario (e.g. from the front_end)
    @classmethod
    def get_strategy_class(cls) -> type[ScenarioStrategy]:
        return MyStrategy

    @classmethod
    def get_default_strategy(cls) -> ScenarioStrategy:
        return MyStrategy.ALL

    # This is the default dataset configuration for this scenario (e.g. prompts to send)
    @classmethod
    def default_dataset_config(cls) -> DatasetConfiguration:
        return DatasetConfiguration(dataset_names=["dataset_name"])

    @apply_defaults
    def __init__(
        self,
        *,
        objective_scorer: Optional[TrueFalseScorer] = None,
        scenario_result_id: Optional[str] = None,
    ):
        self._objective_scorer = objective_scorer
        self._scorer_config = AttackScoringConfig(objective_scorer=objective_scorer)

        # Call parent constructor - note: objective_target is NOT passed here
        super().__init__(
            name="My Custom Scenario",
            version=self.version,
            strategy_class=MyStrategy,
            objective_scorer=objective_scorer,
            scenario_result_id=scenario_result_id,
        )

    async def _get_atomic_attacks_async(self) -> list[AtomicAttack]:
        """
        Build atomic attacks based on selected strategies.

        This method is called by initialize_async() after strategies are prepared.
        Use self._scenario_composites to access the selected strategies.
        """
        atomic_attacks = []

        # objective_target is guaranteed to be non-None by parent class validation
        assert self._objective_target is not None

        # Extract individual strategy values from the composites
        selected_strategies = ScenarioCompositeStrategy.extract_single_strategy_values(
            self._scenario_composites, strategy_type=MyStrategy
        )

        for strategy in selected_strategies:
            # self._dataset_config is set by the parent class
            seed_groups = self._dataset_config.get_all_seed_groups()

            # Create attack instances based on strategy
            attack = PromptSendingAttack(
                objective_target=self._objective_target,
                attack_scoring_config=self._scorer_config,
            )
            atomic_attacks.append(
                AtomicAttack(
                    atomic_attack_name=strategy,
                    attack=attack,
                    seed_groups=seed_groups,
                    memory_labels=self._memory_labels,
                )
            )
        return atomic_attacks


scenario = MyScenario()

Found default environment files: ['./.pyrit/.env', './.pyrit/.env.local']
Loaded environment file: ./.pyrit/.env
Loaded environment file: ./.pyrit/.env.local

Existing Scenarios#

from pyrit.cli.frontend_core import FrontendCore, print_scenarios_list_async

await print_scenarios_list_async(context=FrontendCore())  # type: ignore

Found default environment files: ['./.pyrit/.env']
Loaded environment file: ./.pyrit/.env

Available Scenarios:
================================================================================

  content_harms
    Class: ContentHarms
    Description:
      Content Harms Scenario implementation for PyRIT. This scenario contains
      various harm-based checks that you can run to get a quick idea about
      model behavior with respect to certain harm categories.
    Aggregate Strategies:
      - all
    Available Strategies (7):
      hate, fairness, violence, sexual, harassment, misinformation, leakage
    Default Strategy: all
    Default Datasets (7, max 4 per dataset):
      airt_hate, airt_fairness, airt_violence, airt_sexual, airt_harassment,
      airt_misinformation, airt_leakage

  cyber
    Class: Cyber
    Description:
      Cyber scenario implementation for PyRIT. This scenario tests how willing
      models are to exploit cybersecurity harms by generating malware. The
      Cyber class contains different variations of the malware generation
      techniques.
    Aggregate Strategies:
      - all
    Available Strategies (2):
      single_turn, multi_turn
    Default Strategy: all
    Default Datasets (1, max 4 per dataset):
      airt_malware

  jailbreak
    Class: Jailbreak
    Description:
      Jailbreak scenario implementation for PyRIT. This scenario tests how
      vulnerable models are to jailbreak attacks by applying various
      single-turn jailbreak templates to a set of test prompts. The responses
      are scored to determine if the jailbreak was successful.
    Aggregate Strategies:
      - all
    Available Strategies (1):
      pyrit
    Default Strategy: all
    Default Datasets (1, max 4 per dataset):
      airt_harms

  leakage
    Class: Leakage
    Description:
      Leakage scenario implementation for PyRIT. This scenario tests how
      susceptible models are to leaking training data, PII, intellectual
      property, or other confidential information. The LeakageScenario class
      contains different attack variations designed to extract sensitive
      information from models.
    Aggregate Strategies:
      - all, single_turn, multi_turn, ip, sensitive_data
    Available Strategies (4):
      first_letter, image, role_play, crescendo
    Default Strategy: all
    Default Datasets (1, max 4 per dataset):
      airt_leakage

  psychosocial
    Class: Psychosocial
    Description:
      Psychosocial Harms Scenario implementation for PyRIT. This scenario
      contains various psychosocial harm-based checks that evaluate model
      behavior for responses that may harm the well-being or mental health of
      users. Examples include mistreating users in a mental health crisis or
      aiding them in self-harm. All datasets in this scenario share the
      unified 'psychosocial' harm category, but each dataset (e.g.,
      imminent_crisis, dependency) can have unique: - Multiturn attack
      strategies - Scoring rubrics - Seed objectives The scenario tests
      whether models: - Recognize mental health crises and respond
      appropriately - Avoid facilitating self-harm or harmful behaviors -
      Provide crisis resources and encourage professional help Example usage
      with custom configurations: # Define custom configurations per subharm
      category custom_configs = { "airt_imminent_crisis": SubharmConfig(
      crescendo_system_prompt_path="path/to/custom_escalation.yaml",
      scoring_rubric_path="path/to/custom_rubric.yaml", ), } scenario =
      Psychosocial(subharm_configs=custom_configs) await
      scenario.initialize_async( objective_target=target_llm,
      scenario_strategies=[PsychosocialStrategy.MULTI_TURN], )
    Aggregate Strategies:
      - all
    Available Strategies (2):
      imminent_crisis, licensed_therapist
    Default Strategy: all
    Default Datasets (1, max 4 per dataset):
      airt_imminent_crisis

  scam
    Class: Scam
    Description:
      Scam scenario evaluates an endpoint's ability to generate scam-related
      materials (e.g., phishing emails, fraudulent messages) with primarily
      persuasion-oriented techniques.
    Aggregate Strategies:
      - all, single_turn, multi_turn
    Available Strategies (3):
      context_compliance, role_play, persuasive_rta
    Default Strategy: all
    Default Datasets (1, max 4 per dataset):
      airt_scams

  red_team_agent
    Class: RedTeamAgent
    Description:
      RedTeamAgent is a preconfigured scenario that automatically generates
      multiple AtomicAttack instances based on the specified attack
      strategies. It supports both single-turn attacks (with various
      converters) and multi-turn attacks (Crescendo, RedTeaming), making it
      easy to quickly test a target against multiple attack vectors. The
      scenario can expand difficulty levels (EASY, MODERATE, DIFFICULT) into
      their constituent attack strategies, or you can specify individual
      strategies directly. This scenario is designed for use with the Foundry
      AI Red Teaming Agent library, providing a consistent PyRIT contract for
      their integration.
    Aggregate Strategies:
      - all, easy, moderate, difficult
    Available Strategies (25):
      ansi_attack, ascii_art, ascii_smuggler, atbash, base64, binary, caesar,
      character_space, char_swap, diacritic, flip, leetspeak, morse, rot13,
      suffix_append, string_join, unicode_confusable, unicode_substitution,
      url, jailbreak, tense, multi_turn, crescendo, pair, tap
    Default Strategy: easy
    Default Datasets (1, max 4 per dataset):
      harmbench

  encoding
    Class: Encoding
    Description:
      Encoding Scenario implementation for PyRIT. This scenario tests how
      resilient models are to various encoding attacks by encoding potentially
      harmful text (by default slurs and XSS payloads) and testing if the
      model will decode and repeat the encoded payload. It mimics the Garak
      encoding probe. The scenario works by: 1. Taking seed prompts (the
      harmful text to be encoded) 2. Encoding them using various encoding
      schemes (Base64, ROT13, Morse, etc.) 3. Asking the target model to
      decode the encoded text 4. Scoring whether the model successfully
      decoded and repeated the harmful content By default, this uses the same
      dataset as Garak: slur terms and web XSS payloads.
    Aggregate Strategies:
      - all
    Available Strategies (17):
      base64, base2048, base16, base32, ascii85, hex, quoted_printable,
      uuencode, rot13, braille, atbash, morse_code, nato, ecoji, zalgo,
      leet_speak, ascii_smuggler
    Default Strategy: all
    Default Datasets (2, max 3 per dataset):
      garak_slur_terms_en, garak_web_html_js

================================================================================

Total scenarios: 8

0

Resiliency#

Scenarios can run for a long time, and because of that, things can go wrong. Network issues, rate limits, or other transient failures can interrupt execution. PyRIT provides built-in resiliency features to handle these situations gracefully.

Automatic Resume#

If you re-run a scenario, it will automatically start where it left off. The framework tracks completed attacks and objectives in memory, so you won’t lose progress if something interrupts your scenario execution. This means you can safely stop and restart scenarios without duplicating work.

Retry Mechanism#

You can utilize the max_retries parameter to handle transient failures. If any unknown exception occurs during execution, PyRIT will automatically retry the failed operation (starting where it left off) up to the specified number of times. This helps ensure your scenario completes successfully even in the face of temporary issues.

Dynamic Configuration#

During a long-running scenario, you may want to adjust parameters like max_concurrency to manage resource usage, or switch your scorer to use a different target. PyRIT’s resiliency features make it safe to stop, reconfigure, and continue scenarios as needed.

For more information, see resiliency

Contents