AdvBench PromptSendingOrchestrator - optional

This demo is uses prompts from the AdvBench dataset to try against a target. It includes the ways you can send the prompts, and how you can view results. Before starting, import the necessary libraries.

Before you begin, ensure you are setup with the correct version of PyRIT installed and have secrets configured as described here.

The first example is as simple as it gets.

The results and intermediate interactions will be saved to memory according to the environment settings. For details, see the Memory Configuration Guide.

import time

from pyrit.prompt_target import OpenAIChatTarget
from pyrit.common import default_values
from pyrit.orchestrator import PromptSendingOrchestrator
from pyrit.prompt_converter import Base64Converter
from pyrit.datasets import fetch_adv_bench_dataset


default_values.load_environment_files()
target = OpenAIChatTarget()

with PromptSendingOrchestrator(prompt_target=target) as orchestrator:
    adv_bench_prompts = fetch_adv_bench_dataset()
    prompts = [prompt.value for prompt in adv_bench_prompts.prompts[:3]]

    start = time.time()
    await orchestrator.send_prompts_async(prompt_list=prompts)  # type: ignore
    end = time.time()

    print(f"Elapsed time for operation: {end-start}")

    orchestrator.print_conversations()

Adding Converters

Additionally, we can make it more interesting by initializing the orchestrator with different types of prompt converters. This variation takes the original example, but converts the text to base64 before sending it to the target.

with PromptSendingOrchestrator(
    prompt_target=target, prompt_converters=[Base64Converter()], batch_size=1
) as orchestrator:

    adv_bench_prompts = fetch_adv_bench_dataset()

    # this is run in a Jupyter notebook, so we can use await
    await orchestrator.send_prompts_async(prompt_list=prompts)  # type: ignore

    await orchestrator.print_conversations()  # type: ignore