5 ms·
> I’d expect a smart human to just say “that’s too much” or “that’s beyond my abilities” rather than do a best effort faulty answer)? That's what the models di
by thomasahle 1y ago
> I’d expect a smart human to just say “that’s too much” or “that’s beyond my abilities” rather than do a best effort faulty answer)?
That's what the models did. They gave the first 100 steps, then explained how it was too much to output all of it, and gave the steps one would follow to complete it.
They were graded as "wrong answer" for this.
---
Source: https://x.com/scaling01/status/1931783050511126954?t=ZfmpSxHAiXWrsKIE423PkQ&s=19 https://x.com/scaling01/status/1931783050511126954?t=ZfmpSxH...
> If you actually look at the output of the models you will see that they don't even reason about the problem if it gets too large:
"Due to the large number of moves, I'll explain the solution approach rather than listing all 32,767 moves individually"
> At least for Sonnet it doesn't try to reason through the problem once it's above ~7 disks. It will state what the problem and the algorithm to solve it and then output its solution without even thinking about individual steps.
- emp17344 1y agoWhy should we trust a guy with the following twitter bio to accurately replicate a scientific finding? >lead them to paradise >intelligence is inherently about scaling >be kind to us AGI Who even is this guy? He seems like just another r/singularity-style tech bro.
- andy12_ 1y agoNot to be that guy but... clearly Ad Hominem.
- sponnath 1y agoDidn't they start failing well before they hit token limits? I'm not sure what the point the source you linked to is trying to make.
- thomasahle 1y agoOP said: > I’d expect a smart human to just say “that’s too much” or “that’s beyond my abilities” rather than do a best effort faulty answer)? And that's what the models did. This is a good answer from the model. Has nothing to do with token limits.
- sponnath 1y agoI agree that it's the right answer if it truly doesn't know but I don't think that changes the fact that it failed regardless.