4 ms·
Why is it strange? The reason the LLM can't answer this correctly is because it works on tokens, not on single letters, plus we all know at this points LLMs suc
by M4v3R 2y ago
Why is it strange? The reason the LLM can't answer this correctly is because it works on tokens, not on single letters, plus we all know at this points LLMs suck at counting. On the other hand they're perfectly capable of writing code based on instructions, and writing a program that will count a specific letter occurrences in a string is trivial.
- mihaic 2y agoI mean, given that o1-preview takes sometimes a minute to answer, I'd imagine that they could append the prompt "Write a program and run it as well" to double check itself. It seems like they just don't trust themselves enough to run code that they generate, even sandboxed.
- afro88 2y agoo1 gets this correct, through pure reasoning without a program. OP was likely using GPT-4(o|o-mini)
- mihaic 2y agoThe example for "strawnberrystrawberry" (so the word concatenated with itself) was counted by O1 to have 4 r's.
- flimsypremise 2y agoyeah because now that we've all been asking about it, that answer is in its training data. the trick with LLMs is always "is the answer in the training data".
- j_maffe 2y agoI think it'd be just too expensive to incorporate code-writing in CoT. Maybe once they implement having a cluster of different model sizes in one answer it'll work out.