4 ms·
Are there tokens reserved for tool calls? If yes, I can see the equivalence. If not, not so much.
by perlgeek 6mo ago
Are there tokens reserved for tool calls? If yes, I can see the equivalence. If not, not so much.
- yorwba 6mo agoYes, typically the tags used for tool calls get their own special tokens, e.g. https://huggingface.co/google/gemma-4-E4B-it/blob/main/tokenizer_config.json https://huggingface.co/google/gemma-4-E4B-it/blob/main/token...