3 ms·
Pretty serious flaws in the original paper. 1. Scoring unsolvable challenges as incorrect 2. Not accounting for token span 3. Not allowing LLMs to code as pa
by dr_dshiv 1y ago
Pretty serious flaws in the original paper.
1. Scoring unsolvable challenges as incorrect
2. Not accounting for token span
3. Not allowing LLMs to code as part of solution.
I tend to see Apple’s paper as an excuse for not having competitive products.
- throwfaraway4 1y agoSounds like confirmation bias in action
- ForHackernews 1y agoA bot that outputs plausible gibberish instead of "this is unsolvable" has given the incorrect answer. A bot that regurgitates correct code from its training set is not reasoning. This is the difference between someone who has memorized leetcode solutions and someone who can work through a novel problem.
- thefz 1y ago> I tend to see Apple’s paper as an excuse for not having competitive products. Until they will manage to, then claim they invented AI