3 ms·
I mean, if you don’t want to include tests that LLMs are, by definition, bad at, why don’t we do the same thing for humans?
by moomin 1y ago
I mean, if you don’t want to include tests that LLMs are, by definition, bad at, why don’t we do the same thing for humans?
- cma 1y agoBecause this is a gotcha test and they are trained on a different tokenized transform of the data that makes it hard. As a system they can output it as a string and have an interpreter, as part of the system, count the letters. So no real meaningful deficiency except they maybe don't waste system prompt space to tell it to do that, as a choice.