4 ms·

Thanks, but I can't see that in the article by Anthropic. AFAICT the "650 ideas" that were wrong were all generated independently of each other. I don't agree

Thanks, but I can't see that in the article by Anthropic. AFAICT the "650 ideas" that were wrong were all generated independently of each other. I don't agree that LLMs can evaluate an experiment. There's nothing in LLM training that makes them capable of telling what is e.g. a correct hypothesis from an incorrect one. I know that is a common claim particularly encouraged by AI companies but whenever that claim has been studied systematically and carefully the result is that self-verification doesn't work. For example, see: On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks https://arxiv.org/abs/2402.08115 https://arxiv.org/abs/2402.08115 Note also that basically all the mathematical results published so far have to be checked by an external verifier, either human mathematicians or a proof assistant like Lean, or both, and some systems explicitly couple an LLM generator to a traditional solver, like e.g. AlphaProof. None of this would be needed if LLMs could really evaluate their own results in any reliably correct manner.

red75prime 2mo ago
> On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks An empirical paper from 2024 that doesn't give a principled reason why LLMs will always be bad at self-verification.
- YeGoblynQueenne 2mo ago
  "An empirical paper that doesn't give a principled reason" is a great description for 99% of machine learning papers since the 1980's. I don't see anyone batting an eyelid e.g. that "Attention is All you Need" or "Language Models are Few Shot Learners" have no theoretical results. Which is to say, I think your criticism is unfair and an attempt to avoid engaging with the arguments in the paper.