3 ms·

I don't think anything involving counting is a good test. The model operates on tokens. Maybe scramble the words and ask it to rearrange them sensibly. Humans

I don't think anything involving counting is a good test. The model operates on tokens. Maybe scramble the words and ask it to rearrange them sensibly. Humans can perform this task up to a certain number of words with working memory alone (so don't go overboard asking to unscramble dozens of words). LLMs could potentially do this as well.

dizhn 4y ago
It does operate on tokens but they added the capability to handle things like reversing a string. I tried it because it was supposed to not work but it does. It also gives an explanation of how it works. (Something like yes I normally use tokens ...) By the same token (hehe) it should proably handle number of tokens fine.