2 ms·
Literally every single one? To not mess it up, they either have to spell the word l-i-k-e t-h-i-s in the output/CoT first (which depends on the tokenizer count
by anuramat 1y ago
Literally every single one?
To not mess it up, they either have to spell the word l-i-k-e t-h-i-s in the output/CoT first (which depends on the tokenizer counting every letter as a separate token), or have the exact question in the training set, and all of that is assuming that the model can spell every token.
Sure, it's not exactly a fair setting, but it's a decent reminder about the limitations of the framework
- isaacremuant 1y agoChatgpt. I test these prompts with chatgpt and they work. I've also used claude 4 opus and also worked. It's just weird how it gets repeated ad nauseaum here but I can't reproduce it with a "grab latest model of famous provider".
- jedberg 1y agoI just asked chatgpt "How many b's are in blueberry?". It instantly said "going to the deep thinking model" and then hung.
- isaacremuant 1y agoWhen I do it takes around 3 seconds where it will go "thinking longer for a better answer and then goes 2". Again, I don't understand how it's seemingly so hard for me to reproduce these things. I understand the tokenisation constraints, but feel it's overblown.
- anuramat 1y agoOpus 4.1: > how many times does letter R appear in the word “blueberry”? do not spell the word letter by letter, just count > Looking at the word “blueberry”, I can count the letter ‘r’ appearing 3 times. The R’s appear in positions 6, 7, and 8 of the word (consecutive r’s in “berry”). <https://claude.ai/share/230b7d82-0747-4ab6-813e-5b1c82c43243 https://claude.ai/share/230b7d82-0747-4ab6-813e-5b1c82c43243>