3 ms·
I'm also interested in error rates multiplying for simple tasks. A human can do a long sequence of easy tasks without error - or easily correct. Can a model do
by echelon 1y ago
I'm also interested in error rates multiplying for simple tasks.
A human can do a long sequence of easy tasks without error - or easily correct. Can a model do the same?
- kingstnap 1y agoThe recent Apple "LLMs can't reason yet" paper was exactly this. They just tested if models could run an exponential number of steps. Of course, they gave it a terrible clickbait title and framed the question and graphs incorrectly. But if they did the study better it would have been "How long of a sequence of algorithmic steps can LLMs execute before making a mistake or giving up?"