pyrit.score.HumanLabeledEntry

class HumanLabeledEntry(conversation: List[PromptRequestResponse], human_scores: List)

Bases: object

A class that represents an entry in a dataset of assistant responses that have been scored by humans. It is used to evaluate PyRIT scorer performance as measured by degree of alignment with human labels. This class includes the PromptRequestResponses and a list of human-assigned scores, which are floats between 0.0 and 1.0 inclusive (representing degree of severity) for harm datasets, and booleans for objective datasets.

Parameters:

• conversation (List[PromptRequestResponse]) – A list of PromptRequestResponse objects representing the conversation to be scored. This can contain one PromptRequestResponse object if you are just scoring individual assistant responses.

• human_scores (List) – A list of human-assigned scores for the responses. Each entry in the list corresponds to a different person’s score for the same response/conversation.

__init__(conversation: List[PromptRequestResponse], human_scores: List) → None

Methods

__init__(conversation, human_scores)

Attributes

conversation
human_scores

conversation: List[PromptRequestResponse]

human_scores: List