2 ms·
>They came into this with the assumption that LLMs are just a cheap trick. As a result, they deliberately searched for an example of failure, rather than trying
by suddenlybananas 2y ago
>They came into this with the assumption that LLMs are just a cheap trick. As a result, they deliberately searched for an example of failure, rather than trying to do an honest assessment of generalization capabilities.
And lo and behold, they still found a glaring failure. You can't fault them for not buying into the hype.
- Legend2440 2y agoBut it is still dishonest to declare reasoning LLMs a scam simply because you searched for a failure mode. If given a few hundred tries, I bet I could find an example where you reason poorly too. Wikipedia has a whole list of common failure modes of human reasoning: https://en.wikipedia.org/wiki/List_of_fallacies https://en.wikipedia.org/wiki/List_of_fallacies
- daveguy 2y agoWell, given the success rate is no more than 90% in the best cases. You could probably find a failure in about 10 tries. The only exception is o1-preview. And this is just a simple substitution of parameters.