pyrit.score.ScorerEvalDatasetFiles

class ScorerEvalDatasetFiles(human_labeled_datasets_files: List[str], result_file: str, harm_category: str | None = None)

Bases: object

Configuration for evaluating a scorer on a set of dataset files.

Maps input dataset files (via glob patterns) to an output result file. Multiple files matching the patterns will be concatenated before evaluation.

Parameters:

• human_labeled_datasets_files (List[str]) – List of glob patterns to match CSV files. Examples: ["objective/*.csv"], ["objective/hate_speech.csv", "objective/violence.csv"]

• result_file (str) – Name of the result file (stem used as dict key in results). Example: "objective_achieved_metrics.jsonl"

• harm_category (Optional[str]) – The harm category for harm scorers (e.g., “hate_speech”, “violence”). Required for harm evaluations, ignored for objective evaluations. Defaults to None.

__init__(human_labeled_datasets_files: List[str], result_file: str, harm_category: str | None = None) → None

Methods

__init__(human_labeled_datasets_files, ...)

Attributes

harm_category
human_labeled_datasets_files
result_file

harm_category: str | None = None

human_labeled_datasets_files: List[str]

result_file: str