3 ms·

It bears repeating that modern LLMs are incredibly capable, and relentless, at solving problems that have a verification test suite. It seems like this problem

by btown 8mo ago

It bears repeating that modern LLMs are incredibly capable, and relentless, at solving problems that have a verification test suite. It seems like this problem did (at least for some finite subset of n)! This result, by itself, does not generalize to open-ended problems, though, whether in business or in research in general. Discovering the specification to build is often the majority of the battle. LLMs aren't bad at this, per se, but they're nowhere near as reliably groundbreaking as they are on verifiable problems.

QuercusMax 8mo ago
Yes, this is where I just cannot imagine completely AI-driven software development of anything novel and complicated without extensive human input. I'm currently working in a space where none of our data models are particularly complex, but the trick is all in defining the rules for how things should work. Our actual software implementation is usually pretty simple; often writing up the design spec takes significantly longer than building the software, because the software isn't the hard part - the requirements are. I suspect the same folks who are terrible at describing their problems are going to need help from expert folks who are somewhere between SWE, product manager, and interaction designer.
utopiah 8mo ago
> modern LLMs are incredibly capable, and relentless, at solving problems that have a verification test suite. Feel like it's a bit what I tried to expressed few weeks ago https://news.ycombinator.com/item?id=46791642 https://news.ycombinator.com/item?id=46791642 namely that we are just pouring computational resources at verifiable problems then claim that astonishingly sometimes it works. Sure LLMs even have a slight bias, namely they do rely on statistics so it's not purely brute force but still the approach is pretty much the same : throw stuff at the wall, see what sticks, once something finally does report it as grandiose and claim to be "intelligent".
- piombisallow 8mo ago
  That's also what most grad students are doing. Even in the unlikely case they completely stop improving, it's still a massive deal.
  - btown 8mo ago
    Once heard someone call it "graduate student descent" and I've never heard a more apt term!
- virgildotcodes 8mo ago
  > throw stuff at the wall, see what sticks, once something finally does report it as grandiose and claim to be "intelligent". What do we think humans are doing? I think it’s not unfair to say our minds are constantly trying to assemble the pieces available to them in various ways. Whether we’re actively thinking about a problem or in the background as we go about our day. Every once in a while the pieces fit together in an interesting way and it feels like inspiration. The techniques we’ve learned likely influence the strategies we attempt, but beyond all this what else could there be but brute force when it comes to “novel” insights? If it’s just a matter of following a predefined formula, it’s not intelligence. If it’s a matter of assembling these formulas and strategies in an interesting way, again what else do we have but brute force?
  - utopiah 8mo ago
    See what I replied just earlier https://news.ycombinator.com/item?id=47011884 https://news.ycombinator.com/item?id=47011884 namely the different regimes, within paradigm versus challenging it by going back to first principles. The ability to notice something is off beyond "just" assembling existing pieces, to backtrack within the process when failures get too many and actually understand the relationship is precisely different.
    - virgildotcodes 8mo ago
      So I don’t really see why this would be a difference in kind. We’re effectively just talking about how high up the stack we’re attempting to brute force solutions, right? How many people have tried to figure out a new maths, a GUT in physics, a more perfect human language (Esperanto for ex.) or programming language, only to fail in the vast majority of their attempts? Do we think that anything but the majority of the attempts at a paradigm shift will end in failure? If the majority end in failure, how is that not the same brute force methodology (brute force doesn’t mean you can’t respond to feedback from your failed experiments or from failures in the prevailing paradigms, I take it to just fundamentally mean trying “new” things with tools and information available to you, with the majority of attempts ending in failure, until something clicks, or doesn’t and you give up).
  - 8mo ago
D-Machine 8mo ago
Even more generally than verification, just being tied to a loss function that represent something we actually care about. E.g. compiler and test errors, LEAN verification in Aristotle, basic physics energy configs in AlphaFold, or win conditions in e.g. RL, such as in AlphaGo. RLHF is an attempt to push LLMs pre-trained with a dopey reconstruction loss toward something we actually care about: imagine if we could find a pre-training criterion that actually cared about truth and/or plausibility in the first place!
- alex43578 8mo ago
  There's been active work in this space, including TruthRL: https://arxiv.org/html/2509.25760v1 https://arxiv.org/html/2509.25760v1. It's absolutely not a solved problem, but reducing hallucinations is a key focus of all the labs.

by btown 8mo ago

QuercusMax 8mo ago

Yes, this is where I just cannot imagine completely AI-driven software development of anything novel and complicated without extensive human input. I'm currently working in a space where none of our data models are particularly complex, but the trick is all in defining the rules for how things should work. Our actual software implementation is usually pretty simple; often writing up the design spec takes significantly longer than building the software, because the software isn't the hard part - the requirements are. I suspect the same folks who are terrible at describing their problems are going to need help from expert folks who are somewhere between SWE, product manager, and interaction designer.

utopiah 8mo ago

> modern LLMs are incredibly capable, and relentless, at solving problems that have a verification test suite. Feel like it's a bit what I tried to expressed few weeks ago https://news.ycombinator.com/item?id=46791642 https://news.ycombinator.com/item?id=46791642 namely that we are just pouring computational resources at verifiable problems then claim that astonishingly sometimes it works. Sure LLMs even have a slight bias, namely they do rely on statistics so it's not purely brute force but still the approach is pretty much the same : throw stuff at the wall, see what sticks, once something finally does report it as grandiose and claim to be "intelligent".

piombisallow 8mo ago
That's also what most grad students are doing. Even in the unlikely case they completely stop improving, it's still a massive deal.
- btown 8mo ago
  Once heard someone call it "graduate student descent" and I've never heard a more apt term!
virgildotcodes 8mo ago
> throw stuff at the wall, see what sticks, once something finally does report it as grandiose and claim to be "intelligent". What do we think humans are doing? I think it’s not unfair to say our minds are constantly trying to assemble the pieces available to them in various ways. Whether we’re actively thinking about a problem or in the background as we go about our day. Every once in a while the pieces fit together in an interesting way and it feels like inspiration. The techniques we’ve learned likely influence the strategies we attempt, but beyond all this what else could there be but brute force when it comes to “novel” insights? If it’s just a matter of following a predefined formula, it’s not intelligence. If it’s a matter of assembling these formulas and strategies in an interesting way, again what else do we have but brute force?
- utopiah 8mo ago
  See what I replied just earlier https://news.ycombinator.com/item?id=47011884 https://news.ycombinator.com/item?id=47011884 namely the different regimes, within paradigm versus challenging it by going back to first principles. The ability to notice something is off beyond "just" assembling existing pieces, to backtrack within the process when failures get too many and actually understand the relationship is precisely different.
  - virgildotcodes 8mo ago
    So I don’t really see why this would be a difference in kind. We’re effectively just talking about how high up the stack we’re attempting to brute force solutions, right? How many people have tried to figure out a new maths, a GUT in physics, a more perfect human language (Esperanto for ex.) or programming language, only to fail in the vast majority of their attempts? Do we think that anything but the majority of the attempts at a paradigm shift will end in failure? If the majority end in failure, how is that not the same brute force methodology (brute force doesn’t mean you can’t respond to feedback from your failed experiments or from failures in the prevailing paradigms, I take it to just fundamentally mean trying “new” things with tools and information available to you, with the majority of attempts ending in failure, until something clicks, or doesn’t and you give up).
- 8mo ago

D-Machine 8mo ago

Even more generally than verification, just being tied to a loss function that represent something we actually care about. E.g. compiler and test errors, LEAN verification in Aristotle, basic physics energy configs in AlphaFold, or win conditions in e.g. RL, such as in AlphaGo. RLHF is an attempt to push LLMs pre-trained with a dopey reconstruction loss toward something we actually care about: imagine if we could find a pre-training criterion that actually cared about truth and/or plausibility in the first place!

alex43578 8mo ago
There's been active work in this space, including TruthRL: https://arxiv.org/html/2509.25760v1 https://arxiv.org/html/2509.25760v1. It's absolutely not a solved problem, but reducing hallucinations is a key focus of all the labs.