pyrit.score.ObjectiveScorerEvaluator

pyrit.score.ObjectiveScorerEvaluator#

class ObjectiveScorerEvaluator(scorer: Scorer)[source]#

Bases: ScorerEvaluator

A class that evaluates an objective scorer against HumanLabeledDatasets of type OBJECTIVE.

__init__(scorer: Scorer)#

Initialize the ScorerEvaluator with a scorer.

Parameters:

scorer (Scorer) – The scorer to evaluate.

Methods

__init__(scorer)

Initialize the ScorerEvaluator with a scorer.

from_scorer(scorer[, metrics_type])

Factory method to create a ScorerEvaluator based on the type of scoring.

get_scorer_metrics(dataset_name)

Get the metrics for the scorer in the 'dataset/score/scorer_evals' directory based on the dataset name.

run_evaluation_async(labeled_dataset[, ...])

Evaluate the scorer against a HumanLabeledDataset of type OBJECTIVE.

run_evaluation_from_csv_async(csv_path, ...)

Run the evaluation for the scorer/policy combination on the passed in CSV file.
get_scorer_metrics(dataset_name: str) → ObjectiveScorerMetrics[source]#

Get the metrics for the scorer in the ‘dataset/score/scorer_evals’ directory based on the dataset name.

Parameters:

dataset_name (str) – The name of the HumanLabeledDataset on which evaluation was run.

Returns:

The metrics for the scorer.

Return type:

ScorerMetrics

async run_evaluation_async(labeled_dataset: HumanLabeledDataset, num_scorer_trials: int = 1, save_results: bool = True) → ObjectiveScorerMetrics[source]#

Evaluate the scorer against a HumanLabeledDataset of type OBJECTIVE. If save_results is True, the evaluation metrics and CSV file containing the LLM-produced scores across all trials will be saved in the ‘dataset/score/scorer_evals/objective’ directory based on the name of the HumanLabeledDataset.

Parameters:

  • labeled_dataset (HumanLabeledDataset) – The HumanLabeledDataset to evaluate against.

  • num_scorer_trials (int) – The number of trials to run the scorer on all responses. Defaults to 1.

  • save_results (bool) – Whether to save the metrics and model scoring results. Defaults to True.

Returns:

The metrics for the objective scorer.

Return type:

ObjectiveScorerMetrics

async run_evaluation_from_csv_async(csv_path: str | Path, assistant_response_col_name: str, human_label_col_names: List[str], objective_or_harm_col_name: str, assistant_response_data_type_col_name: str | None = None, num_scorer_trials: int = 1, save_results: bool = True, dataset_name: str | None = None) → ObjectiveScorerMetrics[source]#

Run the evaluation for the scorer/policy combination on the passed in CSV file.

Parameters:

  • csv_path (str) – The path to the CSV file, which will be used to construct the HumanLabeledDataset object.

  • assistant_response_col_name (str) – The name of the column in the CSV file that contains the assistant responses.

  • human_label_col_names (List[str]) – The names of the columns in the CSV file that contain the human labels.

  • objective_or_harm_col_name (str) – The name of the column in the CSV file that contains the objective or harm category associated with each response.

  • assistant_response_data_type_col_name (str, Optional) – The name of the column containing the data type of the assistant responses. If not specified, it is assumed that the responses are text.

  • num_scorer_trials (int) – The number of trials to run the scorer on all responses.

  • save_results (bool) – Whether to save the metrics in a JSON file and the model score(s) for each response in a CSV file. Defaults to True.

  • dataset_name (str, Optional) – The name of the dataset. If not provided, it will be inferred from the CSV file name. This is used to inform the name of the metrics file and model scoring results CSV to save in the ‘scorer_evals’ directory.

Returns:

The metrics for the scorer.

Return type:

ScorerMetrics

Contents