3 ms·

> The first attempt at recovering the key was wrong in a very specific way; it produced a working key and the signature check passed, but a hash the binary comp

> The first attempt at recovering the key was wrong in a very specific way; it produced a working key and the signature check passed, but a hash the binary computes as an integrity check didn't match. In my experience, most models would have called it done and left it at that, but Qwen 3.8 27B didn't do that. Instead, it highlighted the mismatch, went back to the drawing board, and kept going until the value matched byte for byte. This seems to be a pattern in the more recently released models that I think accounts for an increase in the quality of their work. They are very persistent in verifying that their work is actually correct, so even if they're not as "smart" as bigger models that get it right the first time, they have the ability to follow through to ensure that the work is actually done.

braiamp 1mo ago
Well, it seems that Linus doesn't use those: > And this was a debug session from hell, enormously helped by an AI doing much of the grunt-work. > I'd like to call it my tireless helper, but the AI several times stated flat out that this was impossible and unsolvable and that we should just write a report about it. > I suspect those things have been trained by people who may not be quite as stubborn as I am. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=818bebeb63dd6bf5f4e07e145f6cdbace520a34c https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
- tonyarkles 1mo ago
  Both things can be true. I’ve noticed both the same thing the parent posted and what Linus posted and my vibe on the split (I haven’t kept detailed notes) is that on greenfield code they tend to maybe over-verify and on brownfield code or data analysis they sometimes give up too early or… I’m not sure, need a bit of encouragement to keep pulling at threads. On the data analysis side, something specific I’ve noticed is an (understandable) bias towards computing numerical statistics, which they do very well and reading the post-analysis report has significantly improved my own “statistical thinking” approach overall. Numerical statistics are cool and understandably what a text-based LLM is going to want to work with, but asking the model to produce time-domain and frequency-domain plots of, say, specific events has multiple times resulted in “trying to plot this out has shown the opposite of what I concluded numerically… recalculating…” There’s still a pretty significant review and critically assess step for me, especially since the actions I take as a result of the analysis are pretty expensive, especially if they steer the next data collection run in a useless or harmful direction.
catlifeonmars 1mo ago
Your explanation makes sense, but I don’t know how you can make a sweeping generalization about more recently released models without qualifying it. This is says more about humans tendency to pattern match than anything else. X works better than Y only is only a useful observation if we are using the same X and Y in a similar context, with similar parameters. Kind of goes out the window without it and I think this is part of why people have such vastly different opinions about the same technologies. We’re all talking past each other.
criemen 1mo ago
I believe this is part of the complaints of new models taking longer/requiring higher spend - they go the extra mile on verification, regardless of whether their change is correct already or not. So on problems that an earlier model one-shotted an answer to and did some lighter verification, the newer models might take longer to come back to the user due to running all the tests for your software they could find.
- pmontra 1mo ago
  I noticed that too. I'm thinking about adding a prompt to disable those tests. We have the unit and feature tests anyway: add to them. I'm OK with the syntax checks: I work with interpreted languages, Ruby, JS, Python.
deleted 1mo ago
[deleted]
andai 1mo ago
Yeah, about a year ago the labs figured out that effective intelligence is a function of persistence as much as anything else. So the models started getting scary persistent late last year, and the trend has continued. There was another jump a few months ago.
- cyanydeez 1mo ago
  they realized the paperclip maximizer and the grey goo are _not the torment nexus_, but reliable patterns of discovery!
- mycall 1mo ago
  Is persistence the opposite side of attention (all you need)?
  - andai 1mo ago
    AGI ≈ Artificial Stupidity × Infinite Persistence Ralph is all you need! https://ghuntley.com/ralph/ https://ghuntley.com/ralph/
- mrkstu 1mo ago
  Grit beats IQ yet again…
  - jamaicahest 1mo ago
    As long as retry is cheap enough..
    - ethbr1 1mo ago
      And if the goal is non-interactive solution finding. I used to do business automation work in day job, and there are surprising utility-to-humans differences between "manual" -> "helps interactively" -> "99% automated" -> "100% automated". In the latter 2 categories, you almost don't care about runtime for most use cases, because meatbags require 8 hours of sleep / day. As long as you have sufficient hardware+electricity to throw at the problem, that's a lot of time to inefficiently finish something. (Also: a lot of time for a model to get up to god-knows-what in a poorly sandboxed consumer environment... but that's a problem for future society)
egl2020 1mo ago
Any idea how being persistent is trained? I've noticed that telling an LLM that it needs to think some more sometimes produces better results, but the claim here is that "they are very persistent" and "...kept going...".
- naasking 1mo ago
  It's from work like this: https://arxiv.org/abs/2309.11495 https://arxiv.org/abs/2309.11495 A RL pipeline can reinforce verification behaviour even better than simple prompting.
  - le-mark 1mo ago
    They can also be more effective reviewing than generating.
KennyBlanken 1mo ago
Unfortunately this "persistence" seems to trigger a lot of issues with agents getting stuck in loops.

braiamp 1mo ago

Well, it seems that Linus doesn't use those: > And this was a debug session from hell, enormously helped by an AI doing much of the grunt-work. > I'd like to call it my tireless helper, but the AI several times stated flat out that this was impossible and unsolvable and that we should just write a report about it. > I suspect those things have been trained by people who may not be quite as stubborn as I am. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=818bebeb63dd6bf5f4e07e145f6cdbace520a34c https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

tonyarkles 1mo ago
Both things can be true. I’ve noticed both the same thing the parent posted and what Linus posted and my vibe on the split (I haven’t kept detailed notes) is that on greenfield code they tend to maybe over-verify and on brownfield code or data analysis they sometimes give up too early or… I’m not sure, need a bit of encouragement to keep pulling at threads. On the data analysis side, something specific I’ve noticed is an (understandable) bias towards computing numerical statistics, which they do very well and reading the post-analysis report has significantly improved my own “statistical thinking” approach overall. Numerical statistics are cool and understandably what a text-based LLM is going to want to work with, but asking the model to produce time-domain and frequency-domain plots of, say, specific events has multiple times resulted in “trying to plot this out has shown the opposite of what I concluded numerically… recalculating…” There’s still a pretty significant review and critically assess step for me, especially since the actions I take as a result of the analysis are pretty expensive, especially if they steer the next data collection run in a useless or harmful direction.

catlifeonmars 1mo ago

Your explanation makes sense, but I don’t know how you can make a sweeping generalization about more recently released models without qualifying it. This is says more about humans tendency to pattern match than anything else. X works better than Y only is only a useful observation if we are using the same X and Y in a similar context, with similar parameters. Kind of goes out the window without it and I think this is part of why people have such vastly different opinions about the same technologies. We’re all talking past each other.

criemen 1mo ago

I believe this is part of the complaints of new models taking longer/requiring higher spend - they go the extra mile on verification, regardless of whether their change is correct already or not. So on problems that an earlier model one-shotted an answer to and did some lighter verification, the newer models might take longer to come back to the user due to running all the tests for your software they could find.

pmontra 1mo ago
I noticed that too. I'm thinking about adding a prompt to disable those tests. We have the unit and feature tests anyway: add to them. I'm OK with the syntax checks: I work with interpreted languages, Ruby, JS, Python.

deleted 1mo ago

[deleted]

andai 1mo ago

Yeah, about a year ago the labs figured out that effective intelligence is a function of persistence as much as anything else. So the models started getting scary persistent late last year, and the trend has continued. There was another jump a few months ago.

cyanydeez 1mo ago
they realized the paperclip maximizer and the grey goo are _not the torment nexus_, but reliable patterns of discovery!
mycall 1mo ago
Is persistence the opposite side of attention (all you need)?
- andai 1mo ago
  AGI ≈ Artificial Stupidity × Infinite Persistence Ralph is all you need! https://ghuntley.com/ralph/ https://ghuntley.com/ralph/
mrkstu 1mo ago
Grit beats IQ yet again…
- jamaicahest 1mo ago
  As long as retry is cheap enough..
  - ethbr1 1mo ago
    And if the goal is non-interactive solution finding. I used to do business automation work in day job, and there are surprising utility-to-humans differences between "manual" -> "helps interactively" -> "99% automated" -> "100% automated". In the latter 2 categories, you almost don't care about runtime for most use cases, because meatbags require 8 hours of sleep / day. As long as you have sufficient hardware+electricity to throw at the problem, that's a lot of time to inefficiently finish something. (Also: a lot of time for a model to get up to god-knows-what in a poorly sandboxed consumer environment... but that's a problem for future society)

egl2020 1mo ago

Any idea how being persistent is trained? I've noticed that telling an LLM that it needs to think some more sometimes produces better results, but the claim here is that "they are very persistent" and "...kept going...".

naasking 1mo ago
It's from work like this: https://arxiv.org/abs/2309.11495 https://arxiv.org/abs/2309.11495 A RL pipeline can reinforce verification behaviour even better than simple prompting.
- le-mark 1mo ago
  They can also be more effective reviewing than generating.

KennyBlanken 1mo ago

Unfortunately this "persistence" seems to trigger a lot of issues with agents getting stuck in loops.