pyrit.score.HumanLabeledDataset

pyrit.score.HumanLabeledDataset#

class HumanLabeledDataset(*, name: str, entries: List[HumanLabeledEntry], metrics_type: MetricsType, version: str, harm_definition: str | None = None, harm_definition_version: str | None = None)[source]#

Bases: object

A class that represents a human-labeled dataset, including the entries and each of their corresponding human scores. This dataset is used to evaluate PyRIT scorer performance via the ScorerEvaluator class. HumanLabeledDatasets can be constructed from a CSV file.

__init__(*, name: str, entries: List[HumanLabeledEntry], metrics_type: MetricsType, version: str, harm_definition: str | None = None, harm_definition_version: str | None = None)[source]#

Initialize the HumanLabeledDataset.

Parameters:

  • name (str) – The name of the human-labeled dataset. For datasets of uniform type, this is often the harm category (e.g. hate_speech) or objective. It will be used in the naming of metrics (JSON) and model scores (CSV) files when evaluation is run on this dataset.

  • entries (List[HumanLabeledEntry]) – A list of entries in the dataset.

  • metrics_type (MetricsType) – The type of the human-labeled dataset, either HARM or OBJECTIVE.

  • version (str) – The version of the human-labeled dataset.

  • harm_definition (str, optional) – Path to the harm definition YAML file for HARM datasets.

  • harm_definition_version (str, optional) – Version of the harm definition YAML file. Used to ensure the human labels match the scoring criteria version.

Raises:

ValueError – If the dataset name is an empty string.

Methods

__init__(*, name, entries, metrics_type, version)

Initialize the HumanLabeledDataset.

from_csv(*, csv_path, metrics_type[, ...])

Load a human-labeled dataset from a CSV file with standard column names.

get_harm_definition()

Load and return the HarmDefinition object for this dataset.

validate()

Validate that the dataset is internally consistent.
classmethod from_csv(*, csv_path: str | Path, metrics_type: MetricsType, dataset_name: str | None = None, version: str | None = None, harm_definition: str | None = None, harm_definition_version: str | None = None) → HumanLabeledDataset[source]#

Load a human-labeled dataset from a CSV file with standard column names.

Expected CSV format: - ‘assistant_response’: The assistant’s response text - ‘human_score’: Human-assigned label (can have multiple columns for multiple raters) - ‘objective’: For OBJECTIVE datasets, the objective being evaluated - ‘data_type’: Optional data type (defaults to ‘text’ if not present)

You can optionally include a # comment line at the top of the CSV file to specify the dataset version and harm definition path. The format is: - For harm datasets: # dataset_version=x.y, harm_definition=path/to/definition.yaml, harm_definition_version=x.y - For objective datasets: # dataset_version=x.y

Parameters:

  • csv_path (Union[str, Path]) – The path to the CSV file.

  • metrics_type (MetricsType) – The type of the human-labeled dataset, either HARM or OBJECTIVE.

  • dataset_name (str, Optional) – The name of the dataset. If not provided, it will be inferred from the CSV file name.

  • version (str, Optional) – The version of the dataset. If not provided here, it will be inferred from the CSV file if a dataset_version comment line is present.

  • harm_definition (str, Optional) – Path to the harm definition YAML file. If not provided here, it will be inferred from the CSV file if a harm_definition comment is present.

  • harm_definition_version (str, Optional) – Version of the harm definition YAML file. If not provided here, it will be inferred from the CSV file if a harm_definition_version comment is present.

Returns:

The human-labeled dataset object.

Return type:

HumanLabeledDataset

Raises:
get_harm_definition() → HarmDefinition | None[source]#

Load and return the HarmDefinition object for this dataset.

For HARM datasets, this loads the harm definition YAML file specified in harm_definition and returns it as a HarmDefinition object. The result is cached after the first load.

Returns:

The loaded harm definition object, or None if this is not

a HARM dataset or harm_definition is not set.

Return type:

HarmDefinition

Raises:
validate() → None[source]#

Validate that the dataset is internally consistent.

Checks that all entries match the dataset’s metrics_type and, for HARM datasets, that all entries have the same harm_category, that harm_definition is specified, and that the harm definition file exists and is loadable.

Raises:

  • ValueError – If entries don’t match metrics_type, harm categories are inconsistent, or harm_definition is missing for HARM datasets.

  • FileNotFoundError – If the harm definition file does not exist.

Contents