pyrit.chat_message_normalizer.ChatMessageNormalizerTokenizerTemplate#
- class ChatMessageNormalizerTokenizerTemplate(tokenizer: PreTrainedTokenizer | PreTrainedTokenizerFast)[source]#
Bases:
ChatMessageNormalizer[str]This class enables you to apply the chat template stored in a Hugging Face tokenizer to a list of chat messages. For more details, see https://huggingface.co/docs/transformers/main/en/chat_templating
- __init__(tokenizer: PreTrainedTokenizer | PreTrainedTokenizerFast)[source]#
Initializes an instance of the ChatMessageNormalizerTokenizerTemplate class.
- Parameters:
tokenizer (PreTrainedTokenizer | PreTrainedTokenizerFast) – A Hugging Face tokenizer.
Methods
__init__(tokenizer)Initializes an instance of the ChatMessageNormalizerTokenizerTemplate class.
normalize(messages)Applies the chat template stored in the tokenizer to a list of chat messages.
squash_system_message(messages, squash_function)Combines the system message into the first user request.
- normalize(messages: list[ChatMessage]) str[source]#
Applies the chat template stored in the tokenizer to a list of chat messages.
- Parameters:
messages (list[ChatMessage]) – A list of ChatMessage objects.
- Returns:
The formatted chat messages.
- Return type: