3 ms·

> How the heck are we gonna deal with it when we cannot use LLMs for sanity checking LLM code? Unit testing. LLM's are very good at writing tests and writing c

> How the heck are we gonna deal with it when we cannot use LLMs for sanity checking LLM code? Unit testing. LLM's are very good at writing tests and writing code that is testable (as long as you ask it), and if you just check that the tests are actually calling the code and doing so with all the obvious edge cases and that the results are correct, that's actually quite fast to review -- faster than reviewing the code. And you can include things like performance testing in tests as well. We're moving to a world where we work with definitions and tests and are less concerned with the precise details of how code is written within functions. Which is a big shift in mindset.

verelo 1y ago
I’ve been criticized for this by my coworkers in the past, but I strongly believe that this is generally true and has been for quite a while. Developers, myself included, like to think their code is special, set in stone and going to last forever. Most the code we write struggles to live a few years yet we treat all of it like it’s going to last forever. I’ve been an advocate for flipping that and treating it like our code will not last long, and when we identify the components that will, going back and optimizing them. I’m pretty confident that most developers, again including myself, just really enjoy knowing something is done well. Being able to separate yourself from the code and fixate solely on the outcomes can sometimes get me past this.
- lemagedurage 1y ago
  I think this is true for the edges, but if you build on top of software that's not done well, it's a bad time.
  - verelo 1y ago
    Sadly i find most software I am building on top of is pretty awful...but i'm working in the real estate world right now, so that is unavoidable.
ori_b 1y ago
> Unit testing. LLM's are very good at writing tests and writing code that is testable (as long as you ask it) The unit tests LLMs generate are also often crap, testing tautologies, making sure that your dependencies act as specified without testing the actual code, etc.
- crazygringo 1y ago
  Not in my experience. Maybe if your instructions are super unclear?
rmonvfer 1y ago
The only way this might work (IMO) is writing the tests yourself (but of course, this requires you to plan and design very meticulously in advance) and doing some kind of “blind TDD” where the LLM is not able to see the tests, only run them and act on the results. Even then, I’ve had Claude (Opus 4.1) bypass tests by hardcoding conditions as it found them so I’d say reliability for this method is not 100%. Having the LLM write the tests is… well, a recipe for destruction unless you babysit it and give it extremely specific restrictions (again, I’ve done this in mid to large sized projects with fairly comprehensive documentation on testing conventions and results have been mixed: sometimes the LLM does an okay job but tests obvious things, sometimes it ignores the instructions, sometimes it hardcodes or disables conditions…)
- didericis 1y ago
  I’ve been saying this for years now: you can’t avoid communicating what you want a computer to do. The specific requirements have to be made somewhere. Inferring intent from plain english prompts and context is a powerful way for computers to guess what you want from underspecified requirements, but the problem of defining what you want specifically always requires you to convey some irreducible amount of information. Whether it’s code, highly specific plain english, or detailed tests, if you care about correctness they all basically converge to the same thing and the same amount of work.
  - crazygringo 1y ago
    > if you care about correctness they all basically converge to the same thing and the same amount of work. That's the part I'd push back on. They're not the same amount of work. When I'm writing the code myself, it's basically a ton of "plumbing" of loops and ifs and keeping track of counters and making sure I'm not making off-by-one errors and not making punctuation mistakes and all the rest. It actually takes quite a lot of brain energy and time to get that all perfect. It saves a lot of time to write the function definition in plain English, have the LLM generate a bunch of tests that you verify are the correct definition... and then let the LLM take care of all the loops and indexing and punctuation and plumbing. I regularly cut what used to be an entire afternoon or day's worth of work down into 30 minutes. I spend 10 minutes writing the design for what will be 500-1,000 lines of code, 5 minutes answering the LLM's questions about it, 5 minutes skimming the code to make sure it all looks vaguely plausible (no obvious red flags), 5 minutes ensuring the unit tests cover everything I can think of (almost always, the LLM has thought of a bunch of edge cases I never would have bothered to test), and another 5 minutes telling it to fix things, like its unit tests make me suddenly realize there's an edge case that should be defined differently. The idea that it's the "same amount of work" is crazy to me. It's so much more efficient. And in all honesty, the code is more reliable too because it tests things that I usually wouldn't bother with, because writing all the tests is so boring.
smrtinsert 1y ago
Doesn't work if black box. You still have to inspect code performing the operation
majormajor 1y ago
They're OK at it. I usually get more thoroughness of scenarios than a mediocre human engineer (which is great!) but less thoroughness of validation and output checking than a good human engineer (which is less so). But if you have a lot of unit tests and need to make a cross-cutting refactor you run into the same problem that you always have if all your coverage is at the unit level. Now your unit boundary is fundamentally different and you need to know how to lift and shift all the relevant tests to the relevant new places. And so far I've been less impressed by the "agents"' attempts at cross-cutting integration testing since this usually requires selective and clever interface setup and refactoring. LLMs have a habit of creating one-off things for particular unit test scenarios that doesn't scale well to that problem.
simultsop 1y ago
Your first response sounded like: You got more diabetes? Use more insulin :x (insulins are very good handling diabetes) (analogy). Seniors would tell: the more you get in seniority the more you delete code. So I don't think, more cushion for higher jumping is the solution, sometimes you don't need to jump from that high. We're moving to Junior Generative Juniors, recursively.
adamddev1 1y ago
Everyone keeps talking about unit testing as the answer to this problem. But we need to remember, as Dijkstra famously explained, that tests cannot prove the absence of bugs. Tests can only prove that bugs exist.

verelo 1y ago

I’ve been criticized for this by my coworkers in the past, but I strongly believe that this is generally true and has been for quite a while. Developers, myself included, like to think their code is special, set in stone and going to last forever. Most the code we write struggles to live a few years yet we treat all of it like it’s going to last forever. I’ve been an advocate for flipping that and treating it like our code will not last long, and when we identify the components that will, going back and optimizing them. I’m pretty confident that most developers, again including myself, just really enjoy knowing something is done well. Being able to separate yourself from the code and fixate solely on the outcomes can sometimes get me past this.

lemagedurage 1y ago
I think this is true for the edges, but if you build on top of software that's not done well, it's a bad time.
- verelo 1y ago
  Sadly i find most software I am building on top of is pretty awful...but i'm working in the real estate world right now, so that is unavoidable.

ori_b 1y ago

> Unit testing. LLM's are very good at writing tests and writing code that is testable (as long as you ask it) The unit tests LLMs generate are also often crap, testing tautologies, making sure that your dependencies act as specified without testing the actual code, etc.

crazygringo 1y ago
Not in my experience. Maybe if your instructions are super unclear?

rmonvfer 1y ago

The only way this might work (IMO) is writing the tests yourself (but of course, this requires you to plan and design very meticulously in advance) and doing some kind of “blind TDD” where the LLM is not able to see the tests, only run them and act on the results. Even then, I’ve had Claude (Opus 4.1) bypass tests by hardcoding conditions as it found them so I’d say reliability for this method is not 100%. Having the LLM write the tests is… well, a recipe for destruction unless you babysit it and give it extremely specific restrictions (again, I’ve done this in mid to large sized projects with fairly comprehensive documentation on testing conventions and results have been mixed: sometimes the LLM does an okay job but tests obvious things, sometimes it ignores the instructions, sometimes it hardcodes or disables conditions…)

didericis 1y ago
I’ve been saying this for years now: you can’t avoid communicating what you want a computer to do. The specific requirements have to be made somewhere. Inferring intent from plain english prompts and context is a powerful way for computers to guess what you want from underspecified requirements, but the problem of defining what you want specifically always requires you to convey some irreducible amount of information. Whether it’s code, highly specific plain english, or detailed tests, if you care about correctness they all basically converge to the same thing and the same amount of work.
- crazygringo 1y ago
  > if you care about correctness they all basically converge to the same thing and the same amount of work. That's the part I'd push back on. They're not the same amount of work. When I'm writing the code myself, it's basically a ton of "plumbing" of loops and ifs and keeping track of counters and making sure I'm not making off-by-one errors and not making punctuation mistakes and all the rest. It actually takes quite a lot of brain energy and time to get that all perfect. It saves a lot of time to write the function definition in plain English, have the LLM generate a bunch of tests that you verify are the correct definition... and then let the LLM take care of all the loops and indexing and punctuation and plumbing. I regularly cut what used to be an entire afternoon or day's worth of work down into 30 minutes. I spend 10 minutes writing the design for what will be 500-1,000 lines of code, 5 minutes answering the LLM's questions about it, 5 minutes skimming the code to make sure it all looks vaguely plausible (no obvious red flags), 5 minutes ensuring the unit tests cover everything I can think of (almost always, the LLM has thought of a bunch of edge cases I never would have bothered to test), and another 5 minutes telling it to fix things, like its unit tests make me suddenly realize there's an edge case that should be defined differently. The idea that it's the "same amount of work" is crazy to me. It's so much more efficient. And in all honesty, the code is more reliable too because it tests things that I usually wouldn't bother with, because writing all the tests is so boring.

smrtinsert 1y ago

Doesn't work if black box. You still have to inspect code performing the operation

majormajor 1y ago

They're OK at it. I usually get more thoroughness of scenarios than a mediocre human engineer (which is great!) but less thoroughness of validation and output checking than a good human engineer (which is less so). But if you have a lot of unit tests and need to make a cross-cutting refactor you run into the same problem that you always have if all your coverage is at the unit level. Now your unit boundary is fundamentally different and you need to know how to lift and shift all the relevant tests to the relevant new places. And so far I've been less impressed by the "agents"' attempts at cross-cutting integration testing since this usually requires selective and clever interface setup and refactoring. LLMs have a habit of creating one-off things for particular unit test scenarios that doesn't scale well to that problem.

simultsop 1y ago

Your first response sounded like: You got more diabetes? Use more insulin :x (insulins are very good handling diabetes) (analogy). Seniors would tell: the more you get in seniority the more you delete code. So I don't think, more cushion for higher jumping is the solution, sometimes you don't need to jump from that high. We're moving to Junior Generative Juniors, recursively.

adamddev1 1y ago

Everyone keeps talking about unit testing as the answer to this problem. But we need to remember, as Dijkstra famously explained, that tests cannot prove the absence of bugs. Tests can only prove that bugs exist.