5 ms·
Surely the reason LLMs fail here is because this is an adaptation of a common word problem, except your version has been tweaked so that there is a trivial answ
by ProjectArcturis 3y ago
Surely the reason LLMs fail here is because this is an adaptation of a common word problem, except your version has been tweaked so that there is a trivial answer.
- sp332 3y agoYes, that's the point of the question. We want to know if it's actually doing some reasoning, or if it has just memorized an answer.
- deleted 3y ago[deleted]
- ProjectArcturis 3y agoIt's the latter. For every LLM out there. They are trained to memorize, not reason. It will take radically different training techniques to make these networks reason in a human-like way.
- jdkee 3y agoThen how do you explain zero-shot performance?
- ben_w 3y agoMemorising is so trivial we've been doing it by default since forever, regardless of if that means magnetic core memory, the Jacquard Loom, the Gutenberg press, the ceramic movable type China had for a few centuries before Gutenberg, or using a stick to smudge words into soft clay tablets that were accidentally made permanent by a house fire. AI like this aren't just memorisation. They almost certainly don't think like us — even if they did at a low level, the training regime would take the equivalent of hundreds of human lifetimes, and the number of parameters in the larger models is a thousandth of the number in a human brain.