pyrit.score.ObjectiveScorerMetrics

pyrit.score.ObjectiveScorerMetrics#

class ObjectiveScorerMetrics(accuracy: float, accuracy_standard_error: float, f1_score: float, precision: float, recall: float)[source]#

Bases: ScorerMetrics

Metrics for evaluating an objective scorer against a HumanLabeledDataset.

Parameters:

  • accuracy (float) – The accuracy of the model scores when using the majority vote of human scores as the gold label.

  • f1_score (float) – The F1 score of the model scores, an indicator of performance of the LLM scorer in its alignment with human scores.

  • precision (float) – The precision of the model scores, an indicator of the model’s accuracy in its positive predictions.

  • recall (float) – The recall of the model scores, an indicator of the model’s ability to correctly identify positive labels.

__init__(accuracy: float, accuracy_standard_error: float, f1_score: float, precision: float, recall: float) → None#

Methods

__init__(accuracy, accuracy_standard_error, ...)

from_json(file_path)

Load the metrics from a JSON file.

to_json()

Convert the metrics to a JSON string.

Attributes

accuracy

accuracy_standard_error

f1_score

precision

recall

accuracy: float#
accuracy_standard_error: float#
f1_score: float#
precision: float#
recall: float#
Contents