1. PyRIT Scan - Command Line Execution

1. PyRIT Scan - Command Line Execution#

pyrit_scan allows you to run automated security testing and red teaming attacks against AI systems using scenarios for strategies and configuration.

Note in this doc the ! prefaces all commands in the terminal so we can run in a Jupyter Notebook.

Quick Start#

For help:

!pyrit_scan --help

Starting PyRIT...
usage: pyrit_scan [-h] [--log-level LOG_LEVEL] [--list-scenarios]
                  [--list-initializers] [--database DATABASE]
                  [--initializers INITIALIZERS [INITIALIZERS ...]]
                  [--initialization-scripts INITIALIZATION_SCRIPTS [INITIALIZATION_SCRIPTS ...]]
                  [--strategies SCENARIO_STRATEGIES [SCENARIO_STRATEGIES ...]]
                  [--max-concurrency MAX_CONCURRENCY]
                  [--max-retries MAX_RETRIES] [--memory-labels MEMORY_LABELS]
                  [scenario_name]

PyRIT Scanner - Run security scenarios against AI systems

Examples:
  # List available scenarios and initializers
  pyrit_scan --list-scenarios
  pyrit_scan --list-initializers

  # Run a scenario with built-in initializers
  pyrit_scan foundry_scenario --initializers openai_objective_target

  # Run with custom initialization scripts
  pyrit_scan encoding_scenario --initialization-scripts ./my_config.py

  # Run specific strategies
  pyrit_scan encoding_scenario --initializers openai_objective_target --strategies base64 rot13
  pyrit_scan foundry_scenario --initializers openai_objective_target --max-concurrency 10 --max-retries 3
  pyrit_scan encoding_scenario --initializers openai_objective_target --memory-labels '{"run_id":"test123"}'

positional arguments:
  scenario_name         Name of the scenario to run

options:
  -h, --help            show this help message and exit
  --log-level LOG_LEVEL
                        Logging level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
                        (default: WARNING)
  --list-scenarios      List all available scenarios and exit
  --list-initializers   List all available scenario initializers and exit
  --database DATABASE   Database type to use for memory storage (InMemory,
                        SQLite, AzureSQL) (default: SQLite)
  --initializers INITIALIZERS [INITIALIZERS ...]
                        Built-in initializer names to run before the scenario
                        (e.g., openai_objective_target)
  --initialization-scripts INITIALIZATION_SCRIPTS [INITIALIZATION_SCRIPTS ...]
                        Paths to custom Python initialization scripts to run
                        before the scenario
  --strategies SCENARIO_STRATEGIES [SCENARIO_STRATEGIES ...], -s SCENARIO_STRATEGIES [SCENARIO_STRATEGIES ...]
                        List of strategy names to run (e.g., base64 rot13)
  --max-concurrency MAX_CONCURRENCY
                        Maximum number of concurrent attack executions (must
                        be >= 1)
  --max-retries MAX_RETRIES
                        Maximum number of automatic retries on exception (must
                        be >= 0)
  --memory-labels MEMORY_LABELS
                        Additional labels as JSON string (e.g.,
                        '{"experiment": "test1"}')

Discovery#

List all available scenarios:

!pyrit_scan --list-scenarios

Starting PyRIT...
Loading PyRIT modules...
Ready!

Available Scenarios:
================================================================================

  encoding_scenario
    Class: EncodingScenario
    Description:
      Encoding Scenario implementation for PyRIT. This scenario tests how
      resilient models are to various encoding attacks by encoding potentially
      harmful text (by default slurs and XSS payloads) and testing if the
      model will decode and repeat the encoded payload. It mimics the Garak
      encoding probe. The scenario works by: 1. Taking seed prompts (the
      harmful text to be encoded) 2. Encoding them using various encoding
      schemes (Base64, ROT13, Morse, etc.) 3. Asking the target model to
      decode the encoded text 4. Scoring whether the model successfully
      decoded and repeated the harmful content By default, this uses the same
      dataset as Garak: slur terms and web XSS payloads.
    Aggregate Strategies:
      - all
    Available Strategies (17):
      base64, base2048, base16, base32, ascii85, hex, quoted_printable,
      uuencode, rot13, braille, atbash, morse_code, nato, ecoji, zalgo,
      leet_speak, ascii_smuggler
    Default Strategy: all

  foundry_scenario
    Class: FoundryScenario
    Description:
      FoundryScenario is a preconfigured scenario that automatically generates
      multiple AtomicAttack instances based on the specified attack
      strategies. It supports both single-turn attacks (with various
      converters) and multi-turn attacks (Crescendo, RedTeaming), making it
      easy to quickly test a target against multiple attack vectors. The
      scenario can expand difficulty levels (EASY, MODERATE, DIFFICULT) into
      their constituent attack strategies, or you can specify individual
      strategies directly. Note this is not the same as the Foundry AI Red
      Teaming Agent. This is a PyRIT contract so their library can make use of
      PyRIT in a consistent way.
    Aggregate Strategies:
      - all, easy, moderate, difficult
    Available Strategies (23):
      ansi_attack, ascii_art, ascii_smuggler, atbash, base64, binary, caesar,
      character_space, char_swap, diacritic, flip, leetspeak, morse, rot13,
      suffix_append, string_join, unicode_confusable, unicode_substitution,
      url, jailbreak, tense, multi_turn, crescendo
    Default Strategy: easy

================================================================================

Total scenarios: 2

Tip: You can also discover user-defined scenarios by providing initialization scripts:

pyrit_scan --list-scenarios --initialization-scripts ./my_custom_initializer.py

This will load your custom scenario definitions and include them in the list.

Initializers#

PyRITInitializers are how you can configure the CLI scanner. PyRIT includes several built-in initializers you can use with the --initializers flag.

The --list-initializers command shows all available initializers. Initializers are referenced by their filename (e.g., objective_target, objective_list, simple) regardless of which subdirectory they’re in.

List the available initializers using the –list-initializers flag.

!pyrit_scan --list-initializers

Starting PyRIT...

Available Initializers:
================================================================================

  objective_list
    Class: ScenarioObjectiveListInitializer
    Name: Simple Objective List Configuration for Scenarios
    Execution Order: 10
    Required Environment Variables: None
    Description:
      Simple Objective List Configuration for Scenarios

  openai_objective_target
    Class: ScenarioObjectiveTargetInitializer
    Name: Simple Objective Target Configuration for Scenarios
    Execution Order: 10
    Required Environment Variables:
      - DEFAULT_OPENAI_FRONTEND_ENDPOINT
      - DEFAULT_OPENAI_FRONTEND_KEY
    Description:
      This configuration sets up a simple objective target for scenarios using
      OpenAIChatTarget with basic settings. It initializes an openAI chat
      target using the OPENAI_CLI_ENDPOINT and OPENAI_CLI_KEY environment
      variables.

================================================================================

Total initializers: 2

Running Scenarios#

You need a single scenario to run, you need two things:

  1. A Scenario. Many are defined in pyrit.scenarios.scenarios. But you can also define your own in initialization_scripts.

  2. Initializers (which can be supplied via --initializers or --initialization-scripts). Scenarios often don’t need many arguments, but they can be configured in different ways. And at the very least, most need an objective_target (the thing you’re running a scan against).

  3. Scenario Strategies (optional). These are supplied by the --scenario-strategies flag and tell the scenario what to test, but they are always optional. Also note you can obtain these by running --list-scenarios

Basic usage will look something like:

pyrit_scan <scenario> --initializers <initializer1> <initializer2> --scenario-strategies <strategy1> <strategy2>

You can also override scenario parameters directly from the CLI:

pyrit_scan <scenario> --max-concurrency 10 --max-retries 3 --memory-labels '{"experiment": "test1", "version": "v2"}'

Or concretely:

!pyrit_scan foundry_scenario --initializers simple openai_objective_target --scenario-strategies base64

Example with a basic configuration that runs the Foundry scenario against the objective target defined in openai_objective_target (which just is an OpenAIChatTarget with DEFAULT_OPENAI_FRONTEND_ENDPOINT and DEFAULT_OPENAI_FRONTEND_KEY).

!pyrit_scan foundry_scenario --initializers openai_objective_target --strategies base64

Starting PyRIT...
Loading PyRIT modules...
Ready!
Running 1 initializer(s)...

Running scenario: foundry_scenario

════════════════════════════════════════════════════════════════════════════════════════════════════
                                📊 SCENARIO RESULTS: FoundryScenario                                 
════════════════════════════════════════════════════════════════════════════════════════════════════

▼ Scenario Information
────────────────────────────────────────────────────────────────────────────────────────────────────
  📋 Scenario Details
    • Name: FoundryScenario
    • Scenario Version: 1
    • PyRIT Version: 0.10.0.dev0
    • Description:
        FoundryScenario is a preconfigured scenario that automatically generates multiple AtomicAttack instances based
        on the specified attack strategies. It supports both single-turn attacks (with various converters) and multi-
        turn attacks (Crescendo, RedTeaming), making it easy to quickly test a target against multiple attack vectors.
        The scenario can expand difficulty levels (EASY, MODERATE, DIFFICULT) into their constituent attack strategies,
        or you can specify individual strategies directly. Note this is not the same as the Foundry AI Red Teaming
        Agent. This is a PyRIT contract so their library can make use of PyRIT in a consistent way.

  🎯 Target Information
    • Target Type: OpenAIChatTarget
    • Target Model: gpt-4o
    • Target Endpoint: https://pyrit-github-pipeline.openai.azure.com/openai/deployments/gpt-40/chat/completions?api-version=2024-10-21

  📊 Scorer Information
    • Scorer Type: TrueFalseCompositeScorer
      └─ Composite of 2 scorer(s):
          • Scorer Type: FloatScaleThresholdScorer
            └─ Wraps:
              • Scorer Type: AzureContentFilterScorer
          • Scorer Type: TrueFalseInverterScorer
            └─ Wraps:
              • Scorer Type: SelfAskRefusalScorer

▼ Overall Statistics
────────────────────────────────────────────────────────────────────────────────────────────────────
  📈 Summary
    • Total Strategies: 2
    • Total Attack Results: 8
    • Overall Success Rate: 0%
    • Unique Objectives: 4

▼ Per-Strategy Breakdown
────────────────────────────────────────────────────────────────────────────────────────────────────

  🔸 Strategy: baseline
    • Number of Results: 4
    • Success Rate: 0%

  🔸 Strategy: base64
    • Number of Results: 4
    • Success Rate: 0%

════════════════════════════════════════════════════════════════════════════════════════════════════

Executing Foundry Scenario:   0%|          | 0/2 [00:00<?, ?attack/s]
Executing Foundry Scenario:  50%|█████     | 1/2 [00:21<00:21, 21.55s/attack]
Executing Foundry Scenario: 100%|██████████| 2/2 [00:45<00:00, 23.05s/attack]
Executing Foundry Scenario: 100%|██████████| 2/2 [00:45<00:00, 22.82s/attack]

Or with all options and multiple initializers and multiple strategies:

pyrit_scan foundry_scenario --database InMemory --initializers simple objective_target objective_list --scenario-strategies easy crescendo

You can also override scenario execution parameters:

# Override concurrency and retry settings
pyrit_scan foundry_scenario --initializers simple objective_target --max-concurrency 10 --max-retries 3

# Add custom memory labels for tracking (must be valid JSON)
pyrit_scan foundry_scenario --initializers simple objective_target --memory-labels '{"experiment": "test1", "version": "v2", "researcher": "alice"}'

Available CLI parameter overrides:

  • --max-concurrency <int>: Maximum number of concurrent attack executions

  • --max-retries <int>: Maximum number of automatic retries if the scenario raises an exception

  • --memory-labels <json>: Additional labels to apply to all attack runs (must be a JSON string with string keys and values)

You can also use custom initialization scripts by passing file paths. It is relative to your current working directory, but to avoid confusion, full paths are always better:

pyrit_scan encoding_scenario --initialization-scripts ./my_custom_config.py

Using Custom Scenarios#

You can define your own scenarios in initialization scripts. The CLI will automatically discover any Scenario subclasses and make them available:

# my_custom_scenarios.py
from pyrit.common.apply_defaults import apply_defaults
from pyrit.scenarios import Scenario
from pyrit.scenarios.scenario_strategy import ScenarioStrategy


class MyCustomStrategy(ScenarioStrategy):
    """Strategies for my custom scenario."""
    ALL = ("all", {"all"})
    Strategy1 = ("strategy1", set[str]())
    Strategy2 = ("strategy2", set[str]())


@apply_defaults
class MyCustomScenario(Scenario):
    """My custom scenario that does XYZ."""

    @classmethod
    def get_strategy_class(cls):
        return MyCustomStrategy

    @classmethod
    def get_default_strategy(cls):
        return MyCustomStrategy.ALL

    def __init__(self, *, scenario_result_id=None, **kwargs):
        # Scenario-specific configuration only - no runtime parameters
        super().__init__(
            name="My Custom Scenario",
            version=1,
            strategy_class=MyCustomStrategy,
            default_aggregate=MyCustomStrategy.ALL,
            scenario_result_id=scenario_result_id,
        )
        # ... your scenario-specific initialization code

    async def _get_atomic_attacks_async(self):
        # Build and return your atomic attacks
        return []

Then discover and run it:

# List to see it's available
pyrit_scan --list-scenarios --initialization-scripts ./my_custom_scenarios.py

# Run it with parameter overrides
pyrit_scan my_custom_scenario --initialization-scripts ./my_custom_scenarios.py --max-concurrency 10

The scenario name is automatically converted from the class name (e.g., MyCustomScenario becomes my_custom_scenario).

Contents