Why Instance Registries?

Why Instance Registries?#

Some components need configuration that can’t easily be passed at instantiation time. For example, scorers often need:

  • A configured chat_target for LLM-based scoring

  • Specific prompt templates

  • Other dependencies

Instance registries let initializers register fully-configured instances that are ready to use.

Listing Available Instances#

Use get_names() to see registered instances, or list_metadata() for details.

from pyrit.prompt_target import OpenAIChatTarget
from pyrit.registry import ScorerRegistry
from pyrit.score import SelfAskRefusalScorer
from pyrit.setup import IN_MEMORY, initialize_pyrit_async

await initialize_pyrit_async(memory_db_type=IN_MEMORY)  # type: ignore

# Get the registry singleton
registry = ScorerRegistry.get_registry_singleton()

# Register a scorer instance for demonstration
chat_target = OpenAIChatTarget()
refusal_scorer = SelfAskRefusalScorer(chat_target=chat_target)
registry.register_instance(refusal_scorer)

# List what's available
names = registry.get_names()
print(f"Registered scorers: {names}")

Found default environment files: ['C:\\Users\\songjustin\\.pyrit\\.env']
Loaded environment file: C:\Users\songjustin\.pyrit\.env
Registered scorers: ['self_ask_refusal_scorer::94a582f5']

Getting an Instance#

Use get() to retrieve a pre-configured instance by name. The instance is ready to use immediately.

# Get the first registered scorer
if names:
    scorer_name = names[0]
    scorer = registry.get(scorer_name)
    print(f"Retrieved scorer: {scorer}")
    print(f"Scorer type: {type(scorer).__name__}")

Retrieved scorer: <pyrit.score.true_false.self_ask_refusal_scorer.SelfAskRefusalScorer object at 0x000001CB7FF57350>
Scorer type: SelfAskRefusalScorer

Inspecting Metadata#

Scorer metadata includes the scorer type and identifier for tracking.

from pyrit.score import ConsoleScorerPrinter

# Get metadata for all registered scorers
metadata = registry.list_metadata()
for item in metadata:
    print(f"\n{item.unique_name}:")
    print(f"  Class: {item.class_name}")
    print(f"  Type: {item.params.get('scorer_type', 'unknown')}")

    ConsoleScorerPrinter().print_objective_scorer(scorer_identifier=item)

self_ask_refusal_scorer::94a582f5:
  Class: SelfAskRefusalScorer
  Type: true_false
  Description: A self-ask scorer that detects refusal in AI responses. This...

  📊 Scorer Information
    ▸ Scorer Identifier
      • Scorer Type: SelfAskRefusalScorer
      • Target Model: gpt-4o
      • Temperature: None
      • Score Aggregator: OR_

    ▸ Performance Metrics
      Official evaluation has not been run yet for this specific configuration

Filtering#

Use list_metadata() with include_filters and exclude_filters dictionaries to filter scorers by any metadata property. include_filters requires ALL criteria to match (AND logic). exclude_filters excludes items matching ANY criteria. Filters use exact match for simple types and membership check for list types.

# Filter by scorer_type (based on isinstance check against TrueFalseScorer/FloatScaleScorer)
true_false_scorers = registry.list_metadata(include_filters={"scorer_type": "true_false"})
print(f"True/False scorers: {[m.unique_name for m in true_false_scorers]}")

# Filter by class_name
refusal_scorers = registry.list_metadata(include_filters={"class_name": "SelfAskRefusalScorer"})
print(f"Refusal scorers: {[m.unique_name for m in refusal_scorers]}")

# Combine multiple filters (AND logic)
specific_scorers = registry.list_metadata(
    include_filters={"scorer_type": "true_false", "class_name": "SelfAskRefusalScorer"}
)
print(f"True/False refusal scorers: {[m.unique_name for m in specific_scorers]}")

True/False scorers: ['self_ask_refusal_scorer::94a582f5']
Refusal scorers: ['self_ask_refusal_scorer::94a582f5']
True/False refusal scorers: ['self_ask_refusal_scorer::94a582f5']

Using Target Initializer#

You can optionally use the AIRTTargetInitializer to automatically configure and register targets that use commonly used environment variables (from .env_example). This initializer does not strictly require any environment variables - it simply registers whatever endpoints are available.

from pyrit.registry import TargetRegistry
from pyrit.setup import initialize_pyrit_async
from pyrit.setup.initializers import AIRTTargetInitializer

# Using built-in initializer
await initialize_pyrit_async(  # type: ignore
    memory_db_type="InMemory", initializers=[AIRTTargetInitializer()]
)

# Get the registry singleton
registry = TargetRegistry.get_registry_singleton()
# List registered targets
target_names = registry.get_names()
print(f"Registered targets after AIRT initialization: {target_names}")

Found default environment files: ['C:\\Users\\songjustin\\.pyrit\\.env']
Loaded environment file: C:\Users\songjustin\.pyrit\.env
Registered targets after AIRT initialization: ['azure_content_safety', 'azure_gpt4o_unsafe_chat', 'azure_gpt4o_unsafe_chat2', 'default_openai_frontend', 'openai_chat', 'openai_image', 'openai_realtime', 'openai_responses', 'openai_tts', 'openai_video']
Contents