9 ms·

This is a misunderstanding. Modern LLMs are trained with RL to actually write good programs. They aren't just spewing tokens out.

cma 1y ago
They are also trained with RL to write code to pass unit tests and Claude does have a big problem with trying to cheat the test or request pretty quickly after running into issues, making manual edit approval more important. It usually still tells what it is trying to do wrong so you can often find out from its summary before having to scan the diff.
godelski 1y ago
No, YOU misunderstand. This isn't a thing RL can fix https://news.ycombinator.com/item?id=44163194 https://news.ycombinator.com/item?id=44068943 It doesn't optimize "good programs". It interprets "humans interpretation of good programs." More accurately, "it optimizes what low paid over worked humans believe are good programs." Are you hiring your best and brightest to code review the LLMs? Even if you do, it still optimizes tricking them. It will also optimize writing good programs, but you act like that's a well defined and measurable thing.
- mindwok 1y ago
  This is just semantics. What's the difference between a "human interpretation of a good program" and a "good program" when we (humans) are the ones using it? If the model can write code that passes tests, and meets my requirements, then it's a good programmer. I would expect nothing more or less out of a human programmer.
  - otabdeveloper4 1y ago
    > What's the difference between a "human interpretation of a good program" and a "good program" when we (humans) are the ones using it? Correctness. > and meets my requirements It can't do that. "My requirements" wasn't part of the training set.
    - mindwok 1y ago
      "Correctness" in what sense? It sounds like it's being expanded to an abstract academic definition here. For practical purposes, correct means whatever the person using it deems to be correct. > It can't do that. "My requirements" wasn't part of the training set. Neither are mine, the art of building these models is that they are generalisable enough that they can tackle tasks that aren't in their dataset. They have proven, at least for some classes of tasks, they can do exactly that.
      - godelski 1y ago
        > to an abstract academic definition here Besides the fact that your statement is self contradicting, there is actually a solid definition [0]. You should click the link on specification too. Or better yet, go talk to one of those guys that did their PhD in programming languages. > They have proven Have they? Or did you just assume? Yeah, I know they got good scores on those benchmarks but did you look at the benchmarks? Look at the question and look what is required to pass it. Then take a moment and think. For the love of God, take a moment and think about how you can pass those tests. Don't just take a pass at face value and move on. If you do, well I got a bridge to sell you. [0] https://en.wikipedia.org/wiki/Correctness_(computer_science) https://en.wikipedia.org/wiki/Correctness_(computer_science)
        mindwok 1y ago
        Sure, > In theoretical computer science, an algorithm is correct with respect to a specification if it behaves as specified. "As specified" here being the key phrase. This is defined however you want, and ranges from a person saying "yep, behaves as specified", to a formal proof. Modern language language models are trained under RL for both sides of this spectrum, from "Hey man looks good", to formal theorem proving. See https://arxiv.org/html/2502.08908v1 https://arxiv.org/html/2502.08908v1. So I'll return to my original point: LLMs are not just generating outputs that look plausible, they are generating outputs that satisfy (or at least attempt to satisfy) lots of different objectives across a wide range of requirements. They are explicitly trained to do this. So while you argue over the semantics of "correctness", the rest of us will be building stuff with LLMs that is actually useful and fun.
  - godelski 1y ago
    Is your grandma qualified to determine what is good code? > If the model can write code that passes tests You think tests make code good? Oh my sweet summer child. TDD has been tried many times and each time it failed worse than the last.
    - pydry 1y ago
      Good to know something i've been doing for 10 years consistently could never work.
      - godelski 1y ago
        It's okay, lots of people's code is always buggy. I know people that suck at coding and have been doing it for 50 years. It's not uncommon I'm not saying don't make tests. But I am saying you're not omniscient. Until you are, your tests are going to be incomplete. They are helpful guides, but they should not drive development. If you really think you can test for every bug then I suggest you apply to be Secretary for health. https://hackernoon.com/test-driven-development-is-fundamentally-wrong-hor3z4d https://hackernoon.com/test-driven-development-is-fundamenta... https://geometrian.com/projects/blog/test_driven_development_is_stupid.html https://geometrian.com/projects/blog/test_driven_development...
        pydry 1y ago
        Ive worked with people who write tests afterwards on production code and it's pretty inevitable that they: * End up missing tests for edge cases they built and forgot about. Those edge cases often have bugs. * They forget and cover the same edge cases twice if theyre being thorough with test-after. This is a waste. * They usually end up spending almost as much time manually testing in the end to verify the code change they just made worked whereas I would typically just deploy straight to prod. It doesnt prevent all bugs it just prevents enough to make the teams around us who dont do it look bad by comparison even though they do manual checks too. Ive heard loads of good reasons to not write tests at all, Ive yet to hear a good reason to not write one before if you are going to write one. Both of your articles raise pretty typical straw men. One is "what if im not sure what the customer wants?" (thats fine but i hope you arent writing production code at this point) and the other is the peculiar but common notion that TDD can only be done with a low level unit test which is dangerous bullshit.
- tptacek 1y ago
  I don't know if any of this applies to the arguments in my article, but most of the point of it is that progress in code production from LLMs is not a consequence of better models (or fine tuning or whatever), but rather on a shift in how LLMs are used, in agent loops with access to ground truth about whether things compile and pass automatic acceptance. And I'm not claiming that closed-loop agents reliably produce mergeable code, only that they've broken through a threshold where they produce enough mergeable code that they significantly accelerate development.
  - godelski 1y ago
    > I don't know if any of this applies to the arguments > with access to ground truth There's the connection. You think you have ground truth. No such thing exists
    - rfrey 1y ago
      Yes it does. Ground truth is what 30 years of experience says constitutes mergeable code. Ground truth doesn't mean "perfect, provably correct code", it means whatever your best benchmark for acceptable code is. In medical AI, where I'm currently working, "ground truth" is usually whatever human experts say about a medical image, and is rarely perfect. The goal is always to do better than whatever the current ground truth is.
      - godelski 1y ago
        I understand how you interpreted my comment as this. That's my bad. But even when taking state of the art knowledge as a ground truth aligning to that is incredibly hard. Medicine is a great example. You're trying to create a causal graph in a highly noisy environment. You ask 10 doctors and you'll get 12 diagnoses. The problem is subtle things become incredibly important. Which is exactly what makes measurements so fucking hard. There is no state of the art in a well defined sense. The point is that in most domains this is how things are. Even in programming. Getting the right answer isn't enough
    - tptacek 1y ago
- tuhlatte 1y ago
  Those links mostly discuss the original RLHF used to train e.g. ChatGPT 3.5. Current paradigms are shifting towards RLVR (reinforcement learning with verifiable rewards), which absolutely can optimize good programs. You can definitely still run into some of the problems eluded to in the first link. Think hacking unit tests, deception, etc -- but the bar is less "create a perfect RL environment" than "create an RL environment where solving the problem is easier than reward hacking." It might be possible to exploit a bug in the Lean 4 proof assistant to prove a mathematical statement, but I suspect it will usually be easier for an LLM to just write a correct proof. Current RL environments aren't as watertight as Lean 4, but there's certainly work to make them more watertight. This is in no way a "solved" problem, but I do see it as a counter to your assertion that "This isn't a thing RL can fix." RL is powerful.
  - godelski 1y ago
    > Current paradigms are shifting towards RLVR, which absolutely can optimize good programs I think you've misunderstood. RL is great. Hell, RLHF has done a lot of good. I'm not saying LLM are useless. But no, it's much more complex than you claim. RLVM can optimize for correct answers in the narrow domains where there are correct answers but it can't optimize good programs. There's a big difference. You're right that Lean, Coq, and other ATPs can prove mathematical statements, but they also don't ensure that a program is good. There's frequently an infinite number of proofs that are correct, but most of those are terrible proofs. This is the same problem all the coding benchmarks face. Even if the LLM isn't cheating, testing isn't enough. If it was we'd never do code review lol. I can pass a test with an algorithm that's O(n^3) despite there being an O(1) solution. You're right that it makes it better, but it doesn't fix the underlying problem I'm discussing. Not everything is verifiable. Verifiability isn't enough. If you'd like to prove me wrong in the former you're going to need to demonstrate that there are provably true statements to lots of things. I'm not expecting you to defy my namesake, nor will I ask you prove correctness and solve the related halting problem. You can't prove an image is high fidelity. You can't prove a song sounds good. You can't prove a poem is a poem. You can't prove this sentence is English. The world is messy as fuck and most things are highly subjective. But the problem isn't binary, it is continuous. I said we're using Justice Potter optimization, you can't even define what porn is. These definitions change over time, often rapidly! You're forgetting about the tyrannical of metrics. Metrics are great, powerful tools that are incredibly useful. But if you think they're perfectly aligned with what you intend to measure then they become tools that work against you. Goodhart's Law. Metrics only work as guides. They're no different than any other powerful tool, if you use it wrong you get hurt. If you really want to understand this I really encourage you to deep dive into this stuff. You need to get into the math. Into the weeds. You'll find a lot of help with metamathematics (i.e. my namesake), metaphysics (Ian Hacking is a good start), and such. It isn't enough to know the math, you need to know what the math means.
otabdeveloper4 1y ago
"Good" is the context of LLMs means "plausible". Not "correct". If you can't code then the distinction is lost on you, but in fact the "correct" part is why programmers get paid. If "plausible" were good enough then the profession of programmer wouldn't exist.
- mindwok 1y ago
  Not necessarily. If the RL objective is passing tests then in the context of LLMs it means "correct", or at least "correct based on the tests".
  - otabdeveloper4 1y ago
    Unfortunately that doesn't solve the problem in any way. We don't have an Oracle machine for testing software. If we did, we could autogenerate code even without an LLM.

cma 1y ago

They are also trained with RL to write code to pass unit tests and Claude does have a big problem with trying to cheat the test or request pretty quickly after running into issues, making manual edit approval more important. It usually still tells what it is trying to do wrong so you can often find out from its summary before having to scan the diff.

godelski 1y ago

No, YOU misunderstand. This isn't a thing RL can fix https://news.ycombinator.com/item?id=44163194 https://news.ycombinator.com/item?id=44068943 It doesn't optimize "good programs". It interprets "humans interpretation of good programs." More accurately, "it optimizes what low paid over worked humans believe are good programs." Are you hiring your best and brightest to code review the LLMs? Even if you do, it still optimizes tricking them. It will also optimize writing good programs, but you act like that's a well defined and measurable thing.

mindwok 1y ago
This is just semantics. What's the difference between a "human interpretation of a good program" and a "good program" when we (humans) are the ones using it? If the model can write code that passes tests, and meets my requirements, then it's a good programmer. I would expect nothing more or less out of a human programmer.
- otabdeveloper4 1y ago
  > What's the difference between a "human interpretation of a good program" and a "good program" when we (humans) are the ones using it? Correctness. > and meets my requirements It can't do that. "My requirements" wasn't part of the training set.
  - mindwok 1y ago
    "Correctness" in what sense? It sounds like it's being expanded to an abstract academic definition here. For practical purposes, correct means whatever the person using it deems to be correct. > It can't do that. "My requirements" wasn't part of the training set. Neither are mine, the art of building these models is that they are generalisable enough that they can tackle tasks that aren't in their dataset. They have proven, at least for some classes of tasks, they can do exactly that.
    - godelski 1y ago
      > to an abstract academic definition here Besides the fact that your statement is self contradicting, there is actually a solid definition [0]. You should click the link on specification too. Or better yet, go talk to one of those guys that did their PhD in programming languages. > They have proven Have they? Or did you just assume? Yeah, I know they got good scores on those benchmarks but did you look at the benchmarks? Look at the question and look what is required to pass it. Then take a moment and think. For the love of God, take a moment and think about how you can pass those tests. Don't just take a pass at face value and move on. If you do, well I got a bridge to sell you. [0] https://en.wikipedia.org/wiki/Correctness_(computer_science) https://en.wikipedia.org/wiki/Correctness_(computer_science)
      - mindwok 1y ago
        Sure, > In theoretical computer science, an algorithm is correct with respect to a specification if it behaves as specified. "As specified" here being the key phrase. This is defined however you want, and ranges from a person saying "yep, behaves as specified", to a formal proof. Modern language language models are trained under RL for both sides of this spectrum, from "Hey man looks good", to formal theorem proving. See https://arxiv.org/html/2502.08908v1 https://arxiv.org/html/2502.08908v1. So I'll return to my original point: LLMs are not just generating outputs that look plausible, they are generating outputs that satisfy (or at least attempt to satisfy) lots of different objectives across a wide range of requirements. They are explicitly trained to do this. So while you argue over the semantics of "correctness", the rest of us will be building stuff with LLMs that is actually useful and fun.
- godelski 1y ago
  Is your grandma qualified to determine what is good code? > If the model can write code that passes tests You think tests make code good? Oh my sweet summer child. TDD has been tried many times and each time it failed worse than the last.
  - pydry 1y ago
    Good to know something i've been doing for 10 years consistently could never work.
    - godelski 1y ago
      It's okay, lots of people's code is always buggy. I know people that suck at coding and have been doing it for 50 years. It's not uncommon I'm not saying don't make tests. But I am saying you're not omniscient. Until you are, your tests are going to be incomplete. They are helpful guides, but they should not drive development. If you really think you can test for every bug then I suggest you apply to be Secretary for health. https://hackernoon.com/test-driven-development-is-fundamentally-wrong-hor3z4d https://hackernoon.com/test-driven-development-is-fundamenta... https://geometrian.com/projects/blog/test_driven_development_is_stupid.html https://geometrian.com/projects/blog/test_driven_development...
      - pydry 1y ago
        Ive worked with people who write tests afterwards on production code and it's pretty inevitable that they: * End up missing tests for edge cases they built and forgot about. Those edge cases often have bugs. * They forget and cover the same edge cases twice if theyre being thorough with test-after. This is a waste. * They usually end up spending almost as much time manually testing in the end to verify the code change they just made worked whereas I would typically just deploy straight to prod. It doesnt prevent all bugs it just prevents enough to make the teams around us who dont do it look bad by comparison even though they do manual checks too. Ive heard loads of good reasons to not write tests at all, Ive yet to hear a good reason to not write one before if you are going to write one. Both of your articles raise pretty typical straw men. One is "what if im not sure what the customer wants?" (thats fine but i hope you arent writing production code at this point) and the other is the peculiar but common notion that TDD can only be done with a low level unit test which is dangerous bullshit.
tptacek 1y ago
I don't know if any of this applies to the arguments in my article, but most of the point of it is that progress in code production from LLMs is not a consequence of better models (or fine tuning or whatever), but rather on a shift in how LLMs are used, in agent loops with access to ground truth about whether things compile and pass automatic acceptance. And I'm not claiming that closed-loop agents reliably produce mergeable code, only that they've broken through a threshold where they produce enough mergeable code that they significantly accelerate development.
- godelski 1y ago
  > I don't know if any of this applies to the arguments > with access to ground truth There's the connection. You think you have ground truth. No such thing exists
  - rfrey 1y ago
    Yes it does. Ground truth is what 30 years of experience says constitutes mergeable code. Ground truth doesn't mean "perfect, provably correct code", it means whatever your best benchmark for acceptable code is. In medical AI, where I'm currently working, "ground truth" is usually whatever human experts say about a medical image, and is rarely perfect. The goal is always to do better than whatever the current ground truth is.
    - godelski 1y ago
      I understand how you interpreted my comment as this. That's my bad. But even when taking state of the art knowledge as a ground truth aligning to that is incredibly hard. Medicine is a great example. You're trying to create a causal graph in a highly noisy environment. You ask 10 doctors and you'll get 12 diagnoses. The problem is subtle things become incredibly important. Which is exactly what makes measurements so fucking hard. There is no state of the art in a well defined sense. The point is that in most domains this is how things are. Even in programming. Getting the right answer isn't enough
  - tptacek 1y ago
tuhlatte 1y ago
Those links mostly discuss the original RLHF used to train e.g. ChatGPT 3.5. Current paradigms are shifting towards RLVR (reinforcement learning with verifiable rewards), which absolutely can optimize good programs. You can definitely still run into some of the problems eluded to in the first link. Think hacking unit tests, deception, etc -- but the bar is less "create a perfect RL environment" than "create an RL environment where solving the problem is easier than reward hacking." It might be possible to exploit a bug in the Lean 4 proof assistant to prove a mathematical statement, but I suspect it will usually be easier for an LLM to just write a correct proof. Current RL environments aren't as watertight as Lean 4, but there's certainly work to make them more watertight. This is in no way a "solved" problem, but I do see it as a counter to your assertion that "This isn't a thing RL can fix." RL is powerful.
- godelski 1y ago
  > Current paradigms are shifting towards RLVR, which absolutely can optimize good programs I think you've misunderstood. RL is great. Hell, RLHF has done a lot of good. I'm not saying LLM are useless. But no, it's much more complex than you claim. RLVM can optimize for correct answers in the narrow domains where there are correct answers but it can't optimize good programs. There's a big difference. You're right that Lean, Coq, and other ATPs can prove mathematical statements, but they also don't ensure that a program is good. There's frequently an infinite number of proofs that are correct, but most of those are terrible proofs. This is the same problem all the coding benchmarks face. Even if the LLM isn't cheating, testing isn't enough. If it was we'd never do code review lol. I can pass a test with an algorithm that's O(n^3) despite there being an O(1) solution. You're right that it makes it better, but it doesn't fix the underlying problem I'm discussing. Not everything is verifiable. Verifiability isn't enough. If you'd like to prove me wrong in the former you're going to need to demonstrate that there are provably true statements to lots of things. I'm not expecting you to defy my namesake, nor will I ask you prove correctness and solve the related halting problem. You can't prove an image is high fidelity. You can't prove a song sounds good. You can't prove a poem is a poem. You can't prove this sentence is English. The world is messy as fuck and most things are highly subjective. But the problem isn't binary, it is continuous. I said we're using Justice Potter optimization, you can't even define what porn is. These definitions change over time, often rapidly! You're forgetting about the tyrannical of metrics. Metrics are great, powerful tools that are incredibly useful. But if you think they're perfectly aligned with what you intend to measure then they become tools that work against you. Goodhart's Law. Metrics only work as guides. They're no different than any other powerful tool, if you use it wrong you get hurt. If you really want to understand this I really encourage you to deep dive into this stuff. You need to get into the math. Into the weeds. You'll find a lot of help with metamathematics (i.e. my namesake), metaphysics (Ian Hacking is a good start), and such. It isn't enough to know the math, you need to know what the math means.

otabdeveloper4 1y ago

"Good" is the context of LLMs means "plausible". Not "correct". If you can't code then the distinction is lost on you, but in fact the "correct" part is why programmers get paid. If "plausible" were good enough then the profession of programmer wouldn't exist.

mindwok 1y ago
Not necessarily. If the RL objective is passing tests then in the context of LLMs it means "correct", or at least "correct based on the tests".
- otabdeveloper4 1y ago
  Unfortunately that doesn't solve the problem in any way. We don't have an Oracle machine for testing software. If we did, we could autogenerate code even without an LLM.