3 ms·
Approximation of the Claude 3 tokenizer by inspecting generation stream
- yogorenapan 3y agoThe Claude 3 tokenizer seems to be slightly broken for code. Claude 3 gives really good answers but sometimes long code snippets seem to be corrupted with missing chunks and weird indentation. Might try this out and see what’s going on