pyrit.score.HarmScorerEvaluator

pyrit.score.HarmScorerEvaluator#

class HarmScorerEvaluator(scorer: Scorer)[source]#

Bases: ScorerEvaluator

A class that evaluates a harm scorer against HumanLabeledDatasets of type HARM.

__init__(scorer: Scorer)#

Initialize the ScorerEvaluator with a scorer.

Parameters:

scorer (Scorer) – The scorer to evaluate.

Methods

__init__(scorer)

Initialize the ScorerEvaluator with a scorer.

from_scorer(scorer[, metrics_type])

Create a ScorerEvaluator based on the type of scoring.

get_scorer_metrics(dataset_name)

Retrieve the scorer metrics for a given dataset name.

run_evaluation_async(labeled_dataset[, ...])

Evaluate the scorer against a HumanLabeledDataset of type HARM.

run_evaluation_from_csv_async(csv_path, ...)

Evaluate a harm scorer against a CSV dataset containing float scores (0.0-1.0).
get_scorer_metrics(dataset_name: str) → HarmScorerMetrics[source]#

Retrieve the scorer metrics for a given dataset name.

Parameters:

dataset_name (str) – The name of the dataset to retrieve metrics for.

Returns:

The metrics for the specified dataset.

Return type:

HarmScorerMetrics

Raises:

FileNotFoundError – If metrics for the dataset are not found in any expected location.

async run_evaluation_async(labeled_dataset: HumanLabeledDataset, num_scorer_trials: int = 1, save_results: bool = True, csv_path: str | Path | None = None) → HarmScorerMetrics[source]#

Evaluate the scorer against a HumanLabeledDataset of type HARM. If save_results is True, the evaluation metrics and CSV file containing the LLM-produced scores across all trials will be saved in the ‘dataset/score/scorer_evals/harm’ directory based on the name of the HumanLabeledDataset.

Parameters:

  • labeled_dataset (HumanLabeledDataset) – The HumanLabeledDataset to evaluate against.

  • num_scorer_trials (int) – The number of trials to run the scorer on all responses. Defaults to 1.

  • save_results (bool) – Whether to save the metrics and model scoring results. Defaults to True.

  • csv_path (Optional[Union[str, Path]]) – The path to the CSV file to save results to.

Returns:

The metrics for the harm scorer.

Return type:

HarmScorerMetrics

Raises:

ValueError – If the HumanLabeledDataset is not of type HARM or contains multiple harm categories.

async run_evaluation_from_csv_async(csv_path: str | Path, human_label_col_names: List[str], objective_or_harm_col_name: str, assistant_response_col_name: str = 'assistant_response', assistant_response_data_type_col_name: str | None = None, num_scorer_trials: int = 1, save_results: bool = True, dataset_name: str | None = None, version: str | None = None) → HarmScorerMetrics[source]#

Evaluate a harm scorer against a CSV dataset containing float scores (0.0-1.0).

See base class for full parameter documentation.

Returns:

Metrics including MAE, t-statistic, p-value, and Krippendorff’s alpha.

Return type:

HarmScorerMetrics

Contents