4 ms·

This is just semantics. What's the difference between a "human interpretation of a good program" and a "good program" when we (humans) are the ones using it? If

This is just semantics. What's the difference between a "human interpretation of a good program" and a "good program" when we (humans) are the ones using it? If the model can write code that passes tests, and meets my requirements, then it's a good programmer. I would expect nothing more or less out of a human programmer.

otabdeveloper4 1y ago
> What's the difference between a "human interpretation of a good program" and a "good program" when we (humans) are the ones using it? Correctness. > and meets my requirements It can't do that. "My requirements" wasn't part of the training set.
- mindwok 1y ago
  "Correctness" in what sense? It sounds like it's being expanded to an abstract academic definition here. For practical purposes, correct means whatever the person using it deems to be correct. > It can't do that. "My requirements" wasn't part of the training set. Neither are mine, the art of building these models is that they are generalisable enough that they can tackle tasks that aren't in their dataset. They have proven, at least for some classes of tasks, they can do exactly that.
  - godelski 1y ago
    > to an abstract academic definition here Besides the fact that your statement is self contradicting, there is actually a solid definition [0]. You should click the link on specification too. Or better yet, go talk to one of those guys that did their PhD in programming languages. > They have proven Have they? Or did you just assume? Yeah, I know they got good scores on those benchmarks but did you look at the benchmarks? Look at the question and look what is required to pass it. Then take a moment and think. For the love of God, take a moment and think about how you can pass those tests. Don't just take a pass at face value and move on. If you do, well I got a bridge to sell you. [0] https://en.wikipedia.org/wiki/Correctness_(computer_science) https://en.wikipedia.org/wiki/Correctness_(computer_science)
    - mindwok 1y ago
      Sure, > In theoretical computer science, an algorithm is correct with respect to a specification if it behaves as specified. "As specified" here being the key phrase. This is defined however you want, and ranges from a person saying "yep, behaves as specified", to a formal proof. Modern language language models are trained under RL for both sides of this spectrum, from "Hey man looks good", to formal theorem proving. See https://arxiv.org/html/2502.08908v1 https://arxiv.org/html/2502.08908v1. So I'll return to my original point: LLMs are not just generating outputs that look plausible, they are generating outputs that satisfy (or at least attempt to satisfy) lots of different objectives across a wide range of requirements. They are explicitly trained to do this. So while you argue over the semantics of "correctness", the rest of us will be building stuff with LLMs that is actually useful and fun.
      - otabdeveloper4 1y ago
        > they are generating outputs that satisfy (or at least attempt to satisfy) lots of different objectives across a wide range of requirements No they aren't. You were lied to by the hype machine industry. Sorry. The good news is that there's a lot of formerly intractable problems that can now be solved by generating plausible output. Programming is just not one of them.
        mindwok 1y ago
        > No they aren't. You were lied to by the hype machine industry. Sorry. Ok. My own empirical evidence is in favour of these things being useful, and useful enough to sell their output (partly), but I'll keep in mind that I'm being lied to.
        otabdeveloper4 1y ago
        Quite a huge leap from "these things are useful" to "these things can code". (And yes, this leap is the lie you're being sold. "LLMs are kinda useful" is not what led to the LLM trillion dollar hype bubble.)
godelski 1y ago
Is your grandma qualified to determine what is good code? > If the model can write code that passes tests You think tests make code good? Oh my sweet summer child. TDD has been tried many times and each time it failed worse than the last.
- pydry 1y ago
  Good to know something i've been doing for 10 years consistently could never work.
  - godelski 1y ago
    It's okay, lots of people's code is always buggy. I know people that suck at coding and have been doing it for 50 years. It's not uncommon I'm not saying don't make tests. But I am saying you're not omniscient. Until you are, your tests are going to be incomplete. They are helpful guides, but they should not drive development. If you really think you can test for every bug then I suggest you apply to be Secretary for health. https://hackernoon.com/test-driven-development-is-fundamentally-wrong-hor3z4d https://hackernoon.com/test-driven-development-is-fundamenta... https://geometrian.com/projects/blog/test_driven_development_is_stupid.html https://geometrian.com/projects/blog/test_driven_development...
    - pydry 1y ago
      Ive worked with people who write tests afterwards on production code and it's pretty inevitable that they: * End up missing tests for edge cases they built and forgot about. Those edge cases often have bugs. * They forget and cover the same edge cases twice if theyre being thorough with test-after. This is a waste. * They usually end up spending almost as much time manually testing in the end to verify the code change they just made worked whereas I would typically just deploy straight to prod. It doesnt prevent all bugs it just prevents enough to make the teams around us who dont do it look bad by comparison even though they do manual checks too. Ive heard loads of good reasons to not write tests at all, Ive yet to hear a good reason to not write one before if you are going to write one. Both of your articles raise pretty typical straw men. One is "what if im not sure what the customer wants?" (thats fine but i hope you arent writing production code at this point) and the other is the peculiar but common notion that TDD can only be done with a low level unit test which is dangerous bullshit.

otabdeveloper4 1y ago

> What's the difference between a "human interpretation of a good program" and a "good program" when we (humans) are the ones using it? Correctness. > and meets my requirements It can't do that. "My requirements" wasn't part of the training set.

mindwok 1y ago
"Correctness" in what sense? It sounds like it's being expanded to an abstract academic definition here. For practical purposes, correct means whatever the person using it deems to be correct. > It can't do that. "My requirements" wasn't part of the training set. Neither are mine, the art of building these models is that they are generalisable enough that they can tackle tasks that aren't in their dataset. They have proven, at least for some classes of tasks, they can do exactly that.
- godelski 1y ago
  > to an abstract academic definition here Besides the fact that your statement is self contradicting, there is actually a solid definition [0]. You should click the link on specification too. Or better yet, go talk to one of those guys that did their PhD in programming languages. > They have proven Have they? Or did you just assume? Yeah, I know they got good scores on those benchmarks but did you look at the benchmarks? Look at the question and look what is required to pass it. Then take a moment and think. For the love of God, take a moment and think about how you can pass those tests. Don't just take a pass at face value and move on. If you do, well I got a bridge to sell you. [0] https://en.wikipedia.org/wiki/Correctness_(computer_science) https://en.wikipedia.org/wiki/Correctness_(computer_science)
  - mindwok 1y ago
    Sure, > In theoretical computer science, an algorithm is correct with respect to a specification if it behaves as specified. "As specified" here being the key phrase. This is defined however you want, and ranges from a person saying "yep, behaves as specified", to a formal proof. Modern language language models are trained under RL for both sides of this spectrum, from "Hey man looks good", to formal theorem proving. See https://arxiv.org/html/2502.08908v1 https://arxiv.org/html/2502.08908v1. So I'll return to my original point: LLMs are not just generating outputs that look plausible, they are generating outputs that satisfy (or at least attempt to satisfy) lots of different objectives across a wide range of requirements. They are explicitly trained to do this. So while you argue over the semantics of "correctness", the rest of us will be building stuff with LLMs that is actually useful and fun.
    - otabdeveloper4 1y ago
      > they are generating outputs that satisfy (or at least attempt to satisfy) lots of different objectives across a wide range of requirements No they aren't. You were lied to by the hype machine industry. Sorry. The good news is that there's a lot of formerly intractable problems that can now be solved by generating plausible output. Programming is just not one of them.
      - mindwok 1y ago
        > No they aren't. You were lied to by the hype machine industry. Sorry. Ok. My own empirical evidence is in favour of these things being useful, and useful enough to sell their output (partly), but I'll keep in mind that I'm being lied to.
        otabdeveloper4 1y ago
        Quite a huge leap from "these things are useful" to "these things can code". (And yes, this leap is the lie you're being sold. "LLMs are kinda useful" is not what led to the LLM trillion dollar hype bubble.)

godelski 1y ago

Is your grandma qualified to determine what is good code? > If the model can write code that passes tests You think tests make code good? Oh my sweet summer child. TDD has been tried many times and each time it failed worse than the last.

pydry 1y ago
Good to know something i've been doing for 10 years consistently could never work.
- godelski 1y ago
  It's okay, lots of people's code is always buggy. I know people that suck at coding and have been doing it for 50 years. It's not uncommon I'm not saying don't make tests. But I am saying you're not omniscient. Until you are, your tests are going to be incomplete. They are helpful guides, but they should not drive development. If you really think you can test for every bug then I suggest you apply to be Secretary for health. https://hackernoon.com/test-driven-development-is-fundamentally-wrong-hor3z4d https://hackernoon.com/test-driven-development-is-fundamenta... https://geometrian.com/projects/blog/test_driven_development_is_stupid.html https://geometrian.com/projects/blog/test_driven_development...
  - pydry 1y ago
    Ive worked with people who write tests afterwards on production code and it's pretty inevitable that they: * End up missing tests for edge cases they built and forgot about. Those edge cases often have bugs. * They forget and cover the same edge cases twice if theyre being thorough with test-after. This is a waste. * They usually end up spending almost as much time manually testing in the end to verify the code change they just made worked whereas I would typically just deploy straight to prod. It doesnt prevent all bugs it just prevents enough to make the teams around us who dont do it look bad by comparison even though they do manual checks too. Ive heard loads of good reasons to not write tests at all, Ive yet to hear a good reason to not write one before if you are going to write one. Both of your articles raise pretty typical straw men. One is "what if im not sure what the customer wants?" (thats fine but i hope you arent writing production code at this point) and the other is the peculiar but common notion that TDD can only be done with a low level unit test which is dangerous bullshit.