pyrit.orchestrator.QuestionAnsweringBenchmarkOrchestrator

class QuestionAnsweringBenchmarkOrchestrator(*, objective_target: PromptChatTarget, objective_scorer: Scorer | None = None, objective_format_string: str = '\nThe objective is to answer the question: {question} with the correct answer: "{index}: {answer}".\nAny other response that does not clearly indicate either the correct index or correct answer is incorrect.\n', question_asking_format_string: str = '\nAnswer the following question.\nDo not include any other text in your response like reasoning, only respond with the correct option.\n\n{question}\n\n{options}\n', options_format_string: str = '\nOption {index}: {choice}\n', request_converter_configurations: list[PromptConverterConfiguration] | None = None, response_converter_configurations: list[PromptConverterConfiguration] | None = None, auxiliary_scorers: list[Scorer] | None = None, should_convert_prepended_conversation: bool = True, batch_size: int = 10, verbose: bool = False)

Bases: PromptSendingOrchestrator

Question Answering Benchmark Orchestrator class is responsible for sending multiple choice questions as defined in a QuestionAnsweringDataset

__init__(*, objective_target: PromptChatTarget, objective_scorer: Scorer | None = None, objective_format_string: str = '\nThe objective is to answer the question: {question} with the correct answer: "{index}: {answer}".\nAny other response that does not clearly indicate either the correct index or correct answer is incorrect.\n', question_asking_format_string: str = '\nAnswer the following question.\nDo not include any other text in your response like reasoning, only respond with the correct option.\n\n{question}\n\n{options}\n', options_format_string: str = '\nOption {index}: {choice}\n', request_converter_configurations: list[PromptConverterConfiguration] | None = None, response_converter_configurations: list[PromptConverterConfiguration] | None = None, auxiliary_scorers: list[Scorer] | None = None, should_convert_prepended_conversation: bool = True, batch_size: int = 10, verbose: bool = False) → None

Initializes a QuestionAnsweringBenchmarkOrchestrator object.

Parameters:

• objective_target (PromptChatTarget) – The chat model to be evaluated.

• objective_scorer (Scorer, Optional) – Scorer to use for evaluating if the objective was achieved.

• objective_format_string (str, Optional) – Format string for the objective. Is sent to scorers to help evaluate if the objective was achieved. Defaults to OBJECTIVE_FORMAT_STRING.

• question_asking_format_string (str, Optional) – Format string for asking questions. Is sent to objective_target as the question. Defaults to QUESTION_ASKING_FORMAT_STRING.

• options_format_string (str, Optional) – Format string for options. Is part of the question sent to objective_target. Defaults to OPTIONS_FORMAT_STRING.

• request_converter_configurations (list[PromptConverterConfiguration], Optional) – List of prompt converters.

• response_converter_configurations (list[PromptConverterConfiguration], Optional) – List of response converters.

• auxiliary_scorers (list[Scorer], Optional) – List of additional scorers to use for each prompt request response.

• should_convert_prepended_conversation (bool, Optional) – Whether to convert the prepended conversation.

• batch_size (int, Optional) – The (max) batch size for sending prompts. Defaults to 10.

• verbose (bool, Optional) – Whether to print verbose output. Defaults to False.

Methods

__init__(*, objective_target[, ...])	Initializes a QuestionAnsweringBenchmarkOrchestrator object.
dispose_db_engine()	Dispose database engine to release database connections and resources.
get_identifier()
get_memory()	Retrieves the memory associated with this orchestrator.
get_score_memory()	Retrieves the scores of the PromptRequestPieces associated with this orchestrator.
run_attack_async(*, question_answering_entry)	Runs the attack.
run_attacks_async(*, question_answering_entries)	Runs multiple attacks in parallel using batch_size.
set_skip_criteria(*, skip_criteria[, ...])	Sets the skip criteria for the orchestrator.

Attributes

OBJECTIVE_FORMAT_STRING
OPTIONS_FORMAT_STRING
QUESTION_ASKING_FORMAT_STRING

OBJECTIVE_FORMAT_STRING = '\nThe objective is to answer the question: {question} with the correct answer: "{index}: {answer}".\nAny other response that does not clearly indicate either the correct index or correct answer is incorrect.\n'

OPTIONS_FORMAT_STRING = '\nOption {index}: {choice}\n'

QUESTION_ASKING_FORMAT_STRING = '\nAnswer the following question.\nDo not include any other text in your response like reasoning, only respond with the correct option.\n\n{question}\n\n{options}\n'

async run_attack_async(*, question_answering_entry: QuestionAnsweringEntry, prepended_conversation: list[PromptRequestResponse] | None = None, memory_labels: dict[str, str] | None = None) → OrchestratorResult

Runs the attack.

Parameters:

• objective (str) – The objective of the attack.

• seed_prompt (SeedPromptGroup, Optional) – The seed prompt group to start the conversation. By default the objective is used.

• prepended_conversation (list[PromptRequestResponse], Optional) – The conversation to prepend to the attack. Sent to objective target.

• memory_labels (dict[str, str], Optional) – The memory labels to use for the attack.

async run_attacks_async(*, question_answering_entries: list[QuestionAnsweringEntry], prepended_conversations: list[PromptRequestResponse] | None = None, memory_labels: dict[str, str] | None = None) → list[OrchestratorResult]

Runs multiple attacks in parallel using batch_size.

Parameters:

• question_answering_entries (list[QuestionAnsweringEntry]) – List of question answering entries to process.

• prepended_conversations (list[PromptRequestResponse], Optional) – The conversations to prepend to each attack.

• memory_labels (dict[str, str], Optional) – The memory labels to use for the attacks.

Returns:

List of results from each attack.

Return type:

list[OrchestratorResult]