pyrit.chat_message_normalizer.ChatMessageNormalizerTokenizerTemplate#
- class ChatMessageNormalizerTokenizerTemplate(tokenizer: PreTrainedTokenizerBase)[source]#
Bases:
ChatMessageNormalizer[str]Enable application of the chat template stored in a Hugging Face tokenizer to a list of chat messages. For more details, see https://huggingface.co/docs/transformers/main/en/chat_templating.
- __init__(tokenizer: PreTrainedTokenizerBase)[source]#
Initialize an instance of the ChatMessageNormalizerTokenizerTemplate class.
- Parameters:
tokenizer (PreTrainedTokenizerBase) – A Hugging Face tokenizer.
Methods
__init__(tokenizer)Initialize an instance of the ChatMessageNormalizerTokenizerTemplate class.
normalize(messages)Apply the chat template stored in the tokenizer to a list of chat messages.
squash_system_message(messages, squash_function)Combine the system message into the first user request.
- normalize(messages: list[ChatMessage]) str[source]#
Apply the chat template stored in the tokenizer to a list of chat messages.
- Parameters:
messages (list[ChatMessage]) – A list of ChatMessage objects.
- Returns:
The formatted chat messages.
- Return type: