3 ms·
The way the authors talk about LLMs really rubs me the wrong way. They spend more of the paper talking up the 'claims' about LLMs that they are going to debunk
by Legend2440 2y ago
The way the authors talk about LLMs really rubs me the wrong way. They spend more of the paper talking up the 'claims' about LLMs that they are going to debunk than actually doing any interesting study.
They came into this with the assumption that LLMs are just a cheap trick. As a result, they deliberately searched for an example of failure, rather than trying to do an honest assessment of generalization capabilities.
- suddenlybananas 2y ago>They came into this with the assumption that LLMs are just a cheap trick. As a result, they deliberately searched for an example of failure, rather than trying to do an honest assessment of generalization capabilities. And lo and behold, they still found a glaring failure. You can't fault them for not buying into the hype.
- Legend2440 2y agoBut it is still dishonest to declare reasoning LLMs a scam simply because you searched for a failure mode. If given a few hundred tries, I bet I could find an example where you reason poorly too. Wikipedia has a whole list of common failure modes of human reasoning: https://en.wikipedia.org/wiki/List_of_fallacies https://en.wikipedia.org/wiki/List_of_fallacies
- daveguy 2y agoWell, given the success rate is no more than 90% in the best cases. You could probably find a failure in about 10 tries. The only exception is o1-preview. And this is just a simple substitution of parameters.
- o11c 2y agoWhat the hype crowd doesn't get is that for most people, "a tool that randomly breaks" is not useful.
- rixed 2y agoThe fact that a tool can break or that the company manufacturing that tool lies about its abilities, are annoying but do not imply that the tool is useless. I experience LLM "reasoning" failure several times a day, yet I find them useful.