Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

pyrit.cli.frontend_core

Shared core logic for PyRIT Frontends.

This module contains all the business logic for:

Both pyrit_scan and pyrit_shell use these functions.

Functions

add_common_arguments

add_common_arguments(parser: argparse.ArgumentParser) → None

Add arguments shared between pyrit_shell and pyrit_scan.

format_initializer_metadata

format_initializer_metadata(initializer_metadata: InitializerMetadata) → None

Print formatted information about an initializer class.

ParameterTypeDescription
initializer_metadataInitializerMetadataDataclass containing initializer metadata.

format_scenario_metadata

format_scenario_metadata(scenario_metadata: ScenarioMetadata) → None

Print formatted information about a scenario class.

ParameterTypeDescription
scenario_metadataScenarioMetadataDataclass containing scenario metadata.

get_default_initializer_discovery_path

get_default_initializer_discovery_path() → Path

Get the default path for discovering initializers.

Returns:

initialize_pyrit_async

initialize_pyrit_async(memory_db_type: Union[MemoryDatabaseType, str], initialization_scripts: Optional[Sequence[Union[str, pathlib.Path]]] = None, initializers: Optional[Sequence[PyRITInitializer]] = None, env_files: Optional[Sequence[pathlib.Path]] = None, silent: bool = False, memory_instance_kwargs: Any = {}) → None

Initialize PyRIT with the provided memory instance and loads environment files.

ParameterTypeDescription
memory_db_typeMemoryDatabaseTypeThe MemoryDatabaseType string literal which indicates the memory instance to use for central memory. Options include “InMemory”, “SQLite”, and “AzureSQL”.
initialization_scriptsOptional[Sequence[Union[str, pathlib.Path]]]Optional sequence of Python script paths that contain PyRITInitializer classes. Each script must define either a get_initializers() function or an ‘initializers’ variable that returns/contains a list of PyRITInitializer instances. Defaults to None.
initializersOptional[Sequence[PyRITInitializer]]Optional sequence of PyRITInitializer instances to execute directly. These provide type-safe, validated configuration with clear documentation. Defaults to None.
env_filesOptional[Sequence[pathlib.Path]]Optional sequence of environment file paths to load in order. If not provided, will load default .env and .env.local files from PyRIT home if they exist. All paths must be valid pathlib.Path objects. Defaults to None.
silentboolIf True, suppresses print statements about environment file loading. Defaults to False. Defaults to False.
**memory_instance_kwargsOptional[Any]Additional keyword arguments to pass to the memory instance. Defaults to {}.

Raises:

list_initializers_async

list_initializers_async(context: FrontendCore, discovery_path: Optional[Path] = None) → Sequence[InitializerMetadata]

List metadata for all available initializers.

ParameterTypeDescription
contextFrontendCorePyRIT context with loaded registries.
discovery_pathOptional[Path]Optional path to discover initializers from. Defaults to None.

Returns:

list_scenarios_async

list_scenarios_async(context: FrontendCore) → list[ScenarioMetadata]

List metadata for all available scenarios.

ParameterTypeDescription
contextFrontendCorePyRIT context with loaded registries.

Returns:

parse_memory_labels

parse_memory_labels(json_string: str) → dict[str, str]

Parse memory labels from a JSON string.

ParameterTypeDescription
json_stringstrJSON string containing label key-value pairs.

Returns:

Raises:

parse_run_arguments

parse_run_arguments(args_string: str) → dict[str, Any]

Parse run command arguments from a string (for shell mode).

ParameterTypeDescription
args_stringstrSpace-separated argument string (e.g., “scenario_name --initializers foo --strategies bar”).

Returns:

Raises:

print_initializers_list_async(context: FrontendCore, discovery_path: Optional[Path] = None) → int

Print a formatted list of all available initializers.

ParameterTypeDescription
contextFrontendCorePyRIT context with loaded registries.
discovery_pathOptional[Path]Optional path to discover initializers from. Defaults to None.

Returns:

print_scenarios_list_async(context: FrontendCore) → int

Print a formatted list of all available scenarios.

ParameterTypeDescription
contextFrontendCorePyRIT context with loaded registries.

Returns:

resolve_env_files

resolve_env_files(env_file_paths: list[str]) → list[Path]

Resolve environment file paths to absolute Path objects.

ParameterTypeDescription
env_file_pathslist[str]List of environment file path strings.

Returns:

Raises:

resolve_initialization_scripts

resolve_initialization_scripts(script_paths: list[str]) → list[Path]

Resolve initialization script paths.

ParameterTypeDescription
script_pathslist[str]List of script path strings.

Returns:

Raises:

run_scenario_async

run_scenario_async(scenario_name: str, context: FrontendCore, scenario_strategies: Optional[list[str]] = None, max_concurrency: Optional[int] = None, max_retries: Optional[int] = None, memory_labels: Optional[dict[str, str]] = None, dataset_names: Optional[list[str]] = None, max_dataset_size: Optional[int] = None, print_summary: bool = True) → ScenarioResult

Run a scenario by name.

ParameterTypeDescription
scenario_namestrName of the scenario to run.
contextFrontendCorePyRIT context with loaded registries.
scenario_strategiesOptional[list[str]]Optional list of strategy names. Defaults to None.
max_concurrencyOptional[int]Max concurrent operations. Defaults to None.
max_retriesOptional[int]Max retry attempts. Defaults to None.
memory_labelsOptional[dict[str, str]]Labels to attach to memory entries. Defaults to None.
dataset_namesOptional[list[str]]Optional list of dataset names to use instead of scenario defaults. If provided, creates a new dataset configuration (fetches all items unless max_dataset_size is also specified). Defaults to None.
max_dataset_sizeOptional[int]Optional maximum number of items to use from the dataset. If dataset_names is provided, limits items from the new datasets. If only max_dataset_size is provided, overrides the scenario’s default limit. Defaults to None.
print_summaryboolWhether to print the summary after execution. Defaults to True. Defaults to True.

Returns:

Raises:

validate_database

validate_database(database: str) → str

Validate database type.

ParameterTypeDescription
databasestrDatabase type string.

Returns:

Raises:

validate_integer

validate_integer(value: str, name: str = 'value', min_value: Optional[int] = None) → int

Validate and parse an integer value.

Note: The ‘value’ parameter is positional (not keyword-only) to allow use with argparse lambdas like: lambda v: validate_integer(v, min_value=1). This is an exception to the PyRIT style guide for argparse compatibility.

ParameterTypeDescription
valuestrString value to parse.
namestrParameter name for error messages. Defaults to “value”. Defaults to 'value'.
min_valueOptional[int]Optional minimum value constraint. Defaults to None.

Returns:

Raises:

validate_log_level

validate_log_level(log_level: str) → int

Validate log level and convert to logging constant.

ParameterTypeDescription
log_levelstrLog level string (case-insensitive).

Returns:

Raises:

ConfigurationLoader

Bases: YamlLoadable

Loader for PyRIT configuration from YAML files.

This class loads configuration from a YAML file and provides methods to initialize PyRIT with the loaded configuration.

Methods:

from_dict

from_dict(data: dict[str, Any]) → ConfigurationLoader

Create a ConfigurationLoader from a dictionary.

ParameterTypeDescription
datadict[str, Any]Dictionary containing configuration values.

Returns:

get_default_config_path

get_default_config_path() → pathlib.Path

Get the default configuration file path.

Returns:

initialize_pyrit_async

initialize_pyrit_async() → None

Initialize PyRIT with the loaded configuration.

This method resolves all initializer names to instances and calls the core initialize_pyrit_async function.

Raises:

load_with_overrides

load_with_overrides(config_file: Optional[pathlib.Path] = None, memory_db_type: Optional[str] = None, initializers: Optional[Sequence[Union[str, dict[str, Any]]]] = None, initialization_scripts: Optional[Sequence[str]] = None, env_files: Optional[Sequence[str]] = None) → ConfigurationLoader

Load configuration with optional overrides.

This factory method implements a 3-layer configuration precedence:

  1. Default config file (~/.pyrit/.pyrit_conf) if it exists

  2. Explicit config_file argument if provided

  3. Individual override arguments (non-None values take precedence)

This is a staticmethod (not classmethod) because it’s a pure factory function that doesn’t need access to class state and can be reused by multiple interfaces (CLI, shell, programmatic API).

ParameterTypeDescription
config_fileOptional[pathlib.Path]Optional path to a YAML-formatted configuration file. Defaults to None.
memory_db_typeOptional[str]Override for database type (in_memory, sqlite, azure_sql). Defaults to None.
initializersOptional[Sequence[Union[str, dict[str, Any]]]]Override for initializer list. Defaults to None.
initialization_scriptsOptional[Sequence[str]]Override for initialization script paths. Defaults to None.
env_filesOptional[Sequence[str]]Override for environment file paths. Defaults to None.

Returns:

Raises:

ConsoleScenarioResultPrinter

Bases: ScenarioResultPrinter

Console printer for scenario results with enhanced formatting.

This printer formats scenario results for console display with optional color coding, proper indentation, and visual separators. Colors can be disabled for consoles that don’t support ANSI characters.

Constructor Parameters:

ParameterTypeDescription
widthintMaximum width for text wrapping. Must be positive. Defaults to 100. Defaults to 100.
indent_sizeintNumber of spaces for indentation. Must be non-negative. Defaults to 2. Defaults to 2.
enable_colorsboolWhether to enable ANSI color output. When False, all output will be plain text without colors. Defaults to True. Defaults to True.
scorer_printerOptional[ScorerPrinter]Printer for scorer information. If not provided, a ConsoleScorerPrinter with matching settings is created. Defaults to None.

Methods:

print_summary_async(result: ScenarioResult) → None

Print a summary of the scenario result with per-strategy breakdown.

Displays:

ParameterTypeDescription
resultScenarioResultThe scenario result to summarize

DatasetConfiguration

Configuration for scenario datasets.

This class provides a unified way to specify the dataset source for scenarios. Only ONE of seed_groups or dataset_names can be set.

Constructor Parameters:

ParameterTypeDescription
seed_groupsOptional[List[SeedGroup]]Explicit list of SeedGroup to use. Defaults to None.
dataset_namesOptional[List[str]]Names of datasets to load from memory. Defaults to None.
max_dataset_sizeOptional[int]If set, randomly samples up to this many SeedGroups (without replacement). Defaults to None.
scenario_compositesOptional[Sequence[ScenarioCompositeStrategy]]The scenario strategies being executed. Subclasses can use this to filter or customize which seed groups are loaded. Defaults to None.

Methods:

get_all_seed_attack_groups

get_all_seed_attack_groups() → list[SeedAttackGroup]

Resolve and return all seed groups as SeedAttackGroups in a flat list.

This is a convenience method that calls get_seed_attack_groups() and flattens the results into a single list. Use this for attack scenarios that need SeedAttackGroup functionality.

Returns:

Raises:

get_all_seed_groups

get_all_seed_groups() → list[SeedGroup]

Resolve and return all seed groups as a flat list.

This is a convenience method that calls get_seed_groups() and flattens the results into a single list. Use this when you don’t need to track which dataset each seed group came from.

Returns:

Raises:

get_all_seeds

get_all_seeds() → list[Seed]

Load all seed prompts from memory for all configured datasets.

This is a convenience method that retrieves SeedPrompt objects directly from memory for all configured datasets. If max_dataset_size is set, randomly samples up to that many prompts per dataset (without replacement).

Returns:

Raises:

get_default_dataset_names

get_default_dataset_names() → list[str]

Get the list of default dataset names for this configuration.

This is used by the CLI to display what datasets the scenario uses by default.

Returns:

get_seed_attack_groups

get_seed_attack_groups() → dict[str, list[SeedAttackGroup]]

Resolve and return seed groups as SeedAttackGroups, grouped by dataset.

This wraps get_seed_groups() and converts each SeedGroup to a SeedAttackGroup. Use this when you need attack-specific functionality like objectives, prepended conversations, or simulated conversation configuration.

Returns:

Raises:

get_seed_groups

get_seed_groups() → dict[str, list[SeedGroup]]

Resolve and return seed groups based on the configuration.

This method handles all resolution logic:

  1. If seed_groups is set, use those directly (under key ‘_explicit_seed_groups’)

  2. If dataset_names is set, load from memory using those names

In all cases, max_dataset_size is applied per dataset if set.

Subclasses can override this to filter or customize which seed groups are loaded based on the stored scenario_composites.

Returns:

Raises:

has_data_source

has_data_source() → bool

Check if this configuration has a data source configured.

Returns:

FrontendCore

Shared context for PyRIT operations.

This object holds all the registries and configuration needed to run scenarios. It can be created once (for shell) or per-command (for CLI).

Constructor Parameters:

ParameterTypeDescription
config_fileOptional[Path]Optional path to a YAML-formatted configuration file. The file uses .pyrit_conf extension but is YAML format. Defaults to None.
databaseOptional[str]Database type (InMemory, SQLite, or AzureSQL). Defaults to None.
initialization_scriptsOptional[list[Path]]Optional list of initialization script paths. Defaults to None.
initializer_namesOptional[list[Any]]Optional list of initializer entries. Each entry can be a string name (e.g., “simple”) or a dict with ‘name’ and optional ‘args’ (e.g., {“name”: “target”, “args”: {“tags”: “default,scorer”}}). Defaults to None.
env_filesOptional[list[Path]]Optional list of environment file paths to load in order. Defaults to None.
log_levelOptional[int]Logging level constant (e.g., logging.WARNING). Defaults to logging.WARNING. Defaults to None.

Methods:

initialize_async

initialize_async() → None

Initialize PyRIT and load registries (heavy operation).

Sets up memory and loads scenario/initializer registries. Initializers are NOT run here — they are run separately (per-scenario in pyrit_scan, or up-front in pyrit_backend).

InitializerMetadata

Bases: ClassRegistryEntry

Metadata describing a registered PyRITInitializer class.

Use get_class() to get the actual class.

InitializerRegistry

Bases: BaseClassRegistry['PyRITInitializer', InitializerMetadata]

Registry for discovering and managing available initializers.

This class discovers all PyRITInitializer subclasses from the pyrit/setup/initializers directory structure.

Initializers are identified by their filename (e.g., “objective_target”, “simple”). The directory structure is used for organization but not exposed to users.

Constructor Parameters:

ParameterTypeDescription
discovery_pathOptional[Path]The path to discover initializers from. If None, defaults to pyrit/setup/initializers (discovers all). To discover only scenarios, pass pyrit/setup/initializers/scenarios. Defaults to None.
lazy_discoveryboolIf True, discovery is deferred until first access. Defaults to False for backwards compatibility. Defaults to False.

Methods:

get_registry_singleton

get_registry_singleton() → InitializerRegistry

Get the singleton instance of the InitializerRegistry.

Returns:

resolve_initializer_paths

resolve_initializer_paths(initializer_names: list[str]) → list[Path]

Resolve initializer names to their file paths.

ParameterTypeDescription
initializer_nameslist[str]List of initializer names to resolve.

Returns:

Raises:

resolve_script_paths

resolve_script_paths(script_paths: list[str]) → list[Path]

Resolve and validate custom script paths.

ParameterTypeDescription
script_pathslist[str]List of script path strings to resolve.

Returns:

Raises:

ScenarioMetadata

Bases: ClassRegistryEntry

Metadata describing a registered Scenario class.

Use get_class() to get the actual class.

ScenarioRegistry

Bases: BaseClassRegistry['Scenario', ScenarioMetadata]

Registry for discovering and managing available scenario classes.

This class discovers all Scenario subclasses from:

  1. Built-in scenarios in pyrit.scenario.scenarios module

  2. User-defined scenarios from initialization scripts (set via globals)

Scenarios are identified by their simple name (e.g., “encoding”, “foundry”).

Constructor Parameters:

ParameterTypeDescription
lazy_discoveryboolIf True, discovery is deferred until first access. Defaults to True for performance. Defaults to True.

Methods:

discover_user_scenarios

discover_user_scenarios() → None

Discover user-defined scenarios from global variables.

After initialization scripts are executed, they may define Scenario subclasses and store them in globals. This method searches for such classes.

User scenarios will override built-in scenarios with the same name.

get_registry_singleton

get_registry_singleton() → ScenarioRegistry

Get the singleton instance of the ScenarioRegistry.

Returns:

ScenarioResult

Scenario result class for aggregating scenario results.

Constructor Parameters:

ParameterTypeDescription
scenario_identifierScenarioIdentifierIdentifier for the executed scenario.
objective_target_identifierUnion[Dict[str, Any], TargetIdentifier]Target identifier.
attack_resultsdict[str, List[AttackResult]]Results grouped by atomic attack name.
objective_scorer_identifierUnion[Dict[str, Any], ScorerIdentifier]Objective scorer identifier.
scenario_run_stateScenarioRunStateCurrent scenario run state. Defaults to 'CREATED'.
labelsOptional[dict[str, str]]Optional labels. Defaults to None.
completion_timeOptional[datetime]Optional completion timestamp. Defaults to None.
number_triesintNumber of run attempts. Defaults to 0.
idOptional[uuid.UUID]Optional scenario result ID. Defaults to None.
objective_scorerOptional[Scorer]Deprecated scorer object parameter. Defaults to None.

Methods:

get_objectives

get_objectives(atomic_attack_name: Optional[str] = None) → list[str]

Get the list of unique objectives for this scenario.

ParameterTypeDescription
atomic_attack_nameOptional[str]Name of specific atomic attack to include. If None, includes objectives from all atomic attacks. Defaults to None. Defaults to None.

Returns:

get_scorer_evaluation_metrics

get_scorer_evaluation_metrics() → Optional[ScorerMetrics]

Get the evaluation metrics for the scenario’s scorer from the scorer evaluation registry.

Returns:

get_strategies_used

get_strategies_used() → list[str]

Get the list of strategies used in this scenario.

Returns:

normalize_scenario_name

normalize_scenario_name(scenario_name: str) → str

Normalize a scenario name to match the stored class name format.

Converts CLI-style snake_case names (e.g., “foundry” or “content_harms”) to PascalCase class names (e.g., “Foundry” or “ContentHarms”) for database queries. If the input is already in PascalCase or doesn’t match the snake_case pattern, it is returned unchanged.

This is the inverse of ScenarioRegistry._class_name_to_scenario_name().

ParameterTypeDescription
scenario_namestrThe scenario name to normalize.

Returns:

objective_achieved_rate

objective_achieved_rate(atomic_attack_name: Optional[str] = None) → int

Get the success rate of this scenario.

ParameterTypeDescription
atomic_attack_nameOptional[str]Name of specific atomic attack to calculate rate for. If None, calculates rate across all atomic attacks. Defaults to None. Defaults to None.

Returns:

termcolor

Dummy termcolor fallback for colored printing if termcolor is not installed.

Methods:

cprint

cprint(text: str, color: str = None, attrs: list = None) → None

Print text without color.