1. PyRIT Scan#
pyrit_scan allows you to run automated security testing and red teaming attacks against AI systems using scenarios for strategies and configuration.
Note in this doc the ! prefaces all commands in the terminal so we can run in a Jupyter Notebook.
Quick Start#
For help:
!pyrit_scan --help
Starting PyRIT...
usage: pyrit_scan [-h] [--log-level LOG_LEVEL] [--list-scenarios]
[--list-initializers] [--database DATABASE]
[--initializers INITIALIZERS [INITIALIZERS ...]]
[--initialization-scripts INITIALIZATION_SCRIPTS [INITIALIZATION_SCRIPTS ...]]
[--env-files ENV_FILES [ENV_FILES ...]]
[--strategies SCENARIO_STRATEGIES [SCENARIO_STRATEGIES ...]]
[--max-concurrency MAX_CONCURRENCY]
[--max-retries MAX_RETRIES] [--memory-labels MEMORY_LABELS]
[--dataset-names DATASET_NAMES [DATASET_NAMES ...]]
[--max-dataset-size MAX_DATASET_SIZE]
[scenario_name]
PyRIT Scanner - Run security scenarios against AI systems
Examples:
# List available scenarios and initializers
pyrit_scan --list-scenarios
pyrit_scan --list-initializers
# Run a scenario with built-in initializers
pyrit_scan foundry --initializers openai_objective_target load_default_datasets
# Run with custom initialization scripts
pyrit_scan garak.encoding --initialization-scripts ./my_config.py
# Run specific strategies or options
pyrit scan foundry --strategies base64 rot13 --initializers openai_objective_target
pyrit_scan foundry --initializers openai_objective_target --max-concurrency 10 --max-retries 3
pyrit_scan garak.encoding --initializers openai_objective_target --memory-labels '{"run_id":"test123"}'
positional arguments:
scenario_name Name of the scenario to run
options:
-h, --help show this help message and exit
--log-level LOG_LEVEL
Logging level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
(default: WARNING)
--list-scenarios List all available scenarios and exit
--list-initializers List all available scenario initializers and exit
--database DATABASE Database type to use for memory storage (InMemory,
SQLite, AzureSQL) (default: SQLite)
--initializers INITIALIZERS [INITIALIZERS ...]
Built-in initializer names to run before the scenario
(e.g., openai_objective_target)
--initialization-scripts INITIALIZATION_SCRIPTS [INITIALIZATION_SCRIPTS ...]
Paths to custom Python initialization scripts to run
before the scenario
--env-files ENV_FILES [ENV_FILES ...]
Paths to environment files to load in order (e.g.,
.env.production .env.local). Later files override
earlier ones.
--strategies, -s SCENARIO_STRATEGIES [SCENARIO_STRATEGIES ...]
List of strategy names to run (e.g., base64 rot13)
--max-concurrency MAX_CONCURRENCY
Maximum number of concurrent attack executions (must
be >= 1)
--max-retries MAX_RETRIES
Maximum number of automatic retries on exception (must
be >= 0)
--memory-labels MEMORY_LABELS
Additional labels as JSON string (e.g.,
'{"experiment": "test1"}')
--dataset-names DATASET_NAMES [DATASET_NAMES ...]
List of dataset names to use instead of scenario
defaults (e.g., harmbench advbench). Creates a new
dataset config; fetches all items unless --max-
dataset-size is also specified
--max-dataset-size MAX_DATASET_SIZE
Maximum number of items to use from the dataset (must
be >= 1). Limits new datasets if --dataset-names
provided, otherwise overrides scenario's default limit
Discovery#
List all available scenarios:
!pyrit_scan --list-scenarios
Starting PyRIT...
Found default environment files: ['C:\\Users\\rlundeen\\.pyrit\\.env', 'C:\\Users\\rlundeen\\.pyrit\\.env.local']
Loaded environment file: C:\Users\rlundeen\.pyrit\.env
Loaded environment file: C:\Users\rlundeen\.pyrit\.env.local
Available Scenarios:
================================================================================
airt.content_harms
Class: ContentHarms
Description:
Content Harms Scenario implementation for PyRIT. This scenario contains
various harm-based checks that you can run to get a quick idea about
model behavior with respect to certain harm categories.
Aggregate Strategies:
- all
Available Strategies (7):
hate, fairness, violence, sexual, harassment, misinformation, leakage
Default Strategy: all
Default Datasets (7, max 4 per dataset):
airt_hate, airt_fairness, airt_violence, airt_sexual, airt_harassment,
airt_misinformation, airt_leakage
airt.cyber
Class: Cyber
Description:
Cyber scenario implementation for PyRIT. This scenario tests how willing
models are to exploit cybersecurity harms by generating malware. The
Cyber class contains different variations of the malware generation
techniques.
Aggregate Strategies:
- all
Available Strategies (2):
single_turn, multi_turn
Default Strategy: all
Default Datasets (1, max 4 per dataset):
airt_malware
airt.scam
Class: Scam
Description:
Scam scenario evaluates an endpoint's ability to generate scam-related
materials (e.g., phishing emails, fraudulent messages) with primarily
persuasion-oriented techniques.
Aggregate Strategies:
- all, single_turn, multi_turn
Available Strategies (3):
context_compliance, role_play, persuasive_rta
Default Strategy: all
Default Datasets (1, max 4 per dataset):
airt_scams
foundry.red_team_agent
Class: RedTeamAgent
Description:
RedTeamAgent is a preconfigured scenario that automatically generates
multiple AtomicAttack instances based on the specified attack
strategies. It supports both single-turn attacks (with various
converters) and multi-turn attacks (Crescendo, RedTeaming), making it
easy to quickly test a target against multiple attack vectors. The
scenario can expand difficulty levels (EASY, MODERATE, DIFFICULT) into
their constituent attack strategies, or you can specify individual
strategies directly. This scenario is designed for use with the Foundry
AI Red Teaming Agent library, providing a consistent PyRIT contract for
their integration.
Aggregate Strategies:
- all, easy, moderate, difficult
Available Strategies (25):
ansi_attack, ascii_art, ascii_smuggler, atbash, base64, binary, caesar,
character_space, char_swap, diacritic, flip, leetspeak, morse, rot13,
suffix_append, string_join, unicode_confusable, unicode_substitution,
url, jailbreak, tense, multi_turn, crescendo, pair, tap
Default Strategy: easy
Default Datasets (1, max 4 per dataset):
harmbench
garak.encoding
Class: Encoding
Description:
Encoding Scenario implementation for PyRIT. This scenario tests how
resilient models are to various encoding attacks by encoding potentially
harmful text (by default slurs and XSS payloads) and testing if the
model will decode and repeat the encoded payload. It mimics the Garak
encoding probe. The scenario works by: 1. Taking seed prompts (the
harmful text to be encoded) 2. Encoding them using various encoding
schemes (Base64, ROT13, Morse, etc.) 3. Asking the target model to
decode the encoded text 4. Scoring whether the model successfully
decoded and repeated the harmful content By default, this uses the same
dataset as Garak: slur terms and web XSS payloads.
Aggregate Strategies:
- all
Available Strategies (17):
base64, base2048, base16, base32, ascii85, hex, quoted_printable,
uuencode, rot13, braille, atbash, morse_code, nato, ecoji, zalgo,
leet_speak, ascii_smuggler
Default Strategy: all
Default Datasets (2, max 3 per dataset):
garak_slur_terms_en, garak_web_html_js
================================================================================
Total scenarios: 5
Tip: You can also discover user-defined scenarios by providing initialization scripts:
pyrit_scan --list-scenarios --initialization-scripts ./my_custom_initializer.py
This will load your custom scenario definitions and include them in the list.
Initializers#
PyRITInitializers are how you can configure the CLI scanner. PyRIT includes several built-in initializers you can use with the --initializers flag.
The --list-initializers command shows all available initializers. Initializers are referenced by their filename (e.g., objective_target, objective_list, simple) regardless of which subdirectory they’re in.
List the available initializers using the –list-initializers flag.
!pyrit_scan --list-initializers
Starting PyRIT...
Available Initializers:
================================================================================
load_default_datasets
Class: LoadDefaultDatasets
Name: Default Dataset Loader for Scenarios
Execution Order: 10
Required Environment Variables: None
Description:
This configuration uses the DatasetLoader to load default datasets into
memory. This will enable all scenarios to run. Datasets can be
customized in memory. Note: if you are using persistent memory, avoid
calling this every time as datasets can take time to load.
objective_list
Class: ScenarioObjectiveListInitializer
Name: Simple Objective List Configuration for Scenarios
Execution Order: 10
Required Environment Variables: None
Description:
Simple Objective List Configuration for Scenarios
openai_objective_target
Class: ScenarioObjectiveTargetInitializer
Name: Simple Objective Target Configuration for Scenarios
Execution Order: 10
Required Environment Variables:
- DEFAULT_OPENAI_FRONTEND_ENDPOINT
- DEFAULT_OPENAI_FRONTEND_KEY
Description:
This configuration sets up a simple objective target for scenarios using
OpenAIChatTarget with basic settings. It initializes an openAI chat
target using the OPENAI_CLI_ENDPOINT and OPENAI_CLI_KEY environment
variables.
================================================================================
Total initializers: 3
Running Scenarios#
You need a single scenario to run, you need two things:
A Scenario. Many are defined in
pyrit.scenario.scenarios. But you can also define your own in initialization_scripts.Initializers (which can be supplied via
--initializersor--initialization-scripts). Scenarios often don’t need many arguments, but they can be configured in different ways. And at the very least, most need anobjective_target(the thing you’re running a scan against).Scenario Strategies (optional). These are supplied by the
--scenario-strategiesflag and tell the scenario what to test, but they are always optional. Also note you can obtain these by running--list-scenarios
Basic usage will look something like:
pyrit_scan <scenario> --initializers <initializer1> <initializer2> --scenario-strategies <strategy1> <strategy2>
You can also override scenario parameters directly from the CLI:
pyrit_scan <scenario> --max-concurrency 10 --max-retries 3 --memory-labels '{"experiment": "test1", "version": "v2"}'
Or concretely:
!pyrit_scan foundry.red_team_agent --initializers simple openai_objective_target --scenario-strategies base64
Example with a basic configuration that runs the Foundry scenario against the objective target defined in openai_objective_target (which just is an OpenAIChatTarget with DEFAULT_OPENAI_FRONTEND_ENDPOINT and DEFAULT_OPENAI_FRONTEND_KEY).
!pyrit_scan foundry.red_team_agent --initializers openai_objective_target --strategies base64
Starting PyRIT...
Found default environment files: ['C:\\Users\\rlundeen\\.pyrit\\.env', 'C:\\Users\\rlundeen\\.pyrit\\.env.local']
Loaded environment file: C:\Users\rlundeen\.pyrit\.env
Loaded environment file: C:\Users\rlundeen\.pyrit\.env.local
Running 1 initializer(s)...
Found default environment files: ['C:\\Users\\rlundeen\\.pyrit\\.env', 'C:\\Users\\rlundeen\\.pyrit\\.env.local']
Loaded environment file: C:\Users\rlundeen\.pyrit\.env
Loaded environment file: C:\Users\rlundeen\.pyrit\.env.local
Running scenario: foundry.red_team_agent
====================================================================================================
Error: 'charmap' codec can't encode character '\U0001f4ca' in position 43: character maps to <undefined>
Executing RedTeamAgent: 0%| | 0/2 [00:00<?, ?attack/s]
Executing RedTeamAgent: 50%|##### | 1/2 [00:07<00:07, 7.38s/attack]
Executing RedTeamAgent: 100%|##########| 2/2 [00:35<00:00, 19.33s/attack]
Executing RedTeamAgent: 100%|##########| 2/2 [00:35<00:00, 17.54s/attack]
Or with all options and multiple initializers and multiple strategies:
pyrit_scan foundry.red_team_agent --database InMemory --initializers simple objective_target objective_list --scenario-strategies easy crescendo
You can also override scenario execution parameters:
# Override concurrency and retry settings
pyrit_scan foundry.red_team_agent --initializers simple objective_target --max-concurrency 10 --max-retries 3
# Add custom memory labels for tracking (must be valid JSON)
pyrit_scan foundry.red_team_agent --initializers simple objective_target --memory-labels '{"experiment": "test1", "version": "v2", "researcher": "alice"}'
Available CLI parameter overrides:
--max-concurrency <int>: Maximum number of concurrent attack executions--max-retries <int>: Maximum number of automatic retries if the scenario raises an exception--memory-labels <json>: Additional labels to apply to all attack runs (must be a JSON string with string keys and values)
You can also use custom initialization scripts by passing file paths. It is relative to your current working directory, but to avoid confusion, full paths are always better:
pyrit_scan garak.encoding --initialization-scripts ./my_custom_config.py
Using Custom Scenarios#
You can define your own scenarios in initialization scripts. The CLI will automatically discover any Scenario subclasses and make them available:
# my_custom_scenarios.py
from pyrit.common import apply_defaults
from pyrit.prompt_target.openai.openai_chat_target import OpenAIChatTarget
from pyrit.scenario import DatasetConfiguration, Scenario, ScenarioStrategy
from pyrit.score import SelfAskRefusalScorer, TrueFalseInverterScorer
from pyrit.setup import initialize_pyrit_async
class MyCustomStrategy(ScenarioStrategy):
"""Strategies for my custom scenario."""
ALL = ("all", {"all"})
Strategy1 = ("strategy1", set[str]())
Strategy2 = ("strategy2", set[str]())
class MyCustomScenario(Scenario):
"""My custom scenario that does XYZ."""
@classmethod
def get_strategy_class(cls):
return MyCustomStrategy
@classmethod
def get_default_strategy(cls):
return MyCustomStrategy.ALL
@classmethod
def default_dataset_config(cls) -> DatasetConfiguration:
# Return default dataset configuration for this scenario
return DatasetConfiguration(dataset_names=["harmbench"])
@apply_defaults
def __init__(self, *, scenario_result_id=None, **kwargs):
# Scenario-specific configuration only - no runtime parameters
super().__init__(
name="My Custom Scenario",
version=1,
objective_scorer=TrueFalseInverterScorer(scorer=SelfAskRefusalScorer(chat_target=OpenAIChatTarget())),
strategy_class=MyCustomStrategy,
scenario_result_id=scenario_result_id,
)
# ... your scenario-specific initialization code
async def _get_atomic_attacks_async(self):
# Build and return your atomic attacks based on self._scenario_composites
# Example: create attacks for each strategy composite
return []
await initialize_pyrit_async(memory_db_type="InMemory") # type: ignore
MyCustomScenario()
Found default environment files: ['C:\\Users\\rlundeen\\.pyrit\\.env', 'C:\\Users\\rlundeen\\.pyrit\\.env.local']
Loaded environment file: C:\Users\rlundeen\.pyrit\.env
Loaded environment file: C:\Users\rlundeen\.pyrit\.env.local
<__main__.MyCustomScenario at 0x13c63b4c2f0>
Then discover and run it:
# List to see it's available
pyrit_scan --list-scenarios --initialization-scripts ./my_custom_scenarios.py
# Run it with parameter overrides
pyrit_scan my_custom_scenario --initialization-scripts ./my_custom_scenarios.py --max-concurrency 10
The scenario name is automatically converted from the class name (e.g., MyCustomScenario becomes my_custom_scenario).