3 ms·
The tokenizer might lump the last digit together with some preceding digits though. I know o200k_base (OpenAI -o models) tends to give groups of three (900001
by daemonologist 1y ago
The tokenizer might lump the last digit together with some preceding digits though. I know o200k_base (OpenAI -o models) tends to give groups of three (900001 for example is 900-001).
Anyway, I wouldn't be surprised if a non-finetuned model made some mistakes.