7 ms·

I asked Codex to write some unit tests for Redux today. At first glance it looked fine, and I continued on. I then went back to add a test by hand, and after lo

I asked Codex to write some unit tests for Redux today. At first glance it looked fine, and I continued on. I then went back to add a test by hand, and after looking more closely at the output there were like 50 wtf worthy things scattered in there. Sure they ran, but it was bad in all sorts of ways. And this was just writing something very basic. This has been my experience almost every time I use AI: superficially it seems fine, once I go to extend the code I realize it's a disaster and I have to clean it up. The problem with "code is cheap" is that, it's not. GENERATING code is now cheap (while the LLMs are subsidized by endless VC dollars, anyway), but the cost of owning that code is not. Every line of code is a liability, and generating thousands of lines a day is like running up a few thousand dollars of debt on a credit card thinking you're getting free stuff and then being surprised when it gets declined.

jjimmy2k 9mo ago
A better agent can fix it in the future.
acedTrex 9mo ago
I've always said every line is a liability, its our job to limit liabilities. That has largely gone out the window these days.
- chr15m 9mo ago
  > every line is a liability, its our job to limit liabilities. Hard agree!
  - ares623 9mo ago
    But more code from AI means stocks go up. Stocks are assets. If you generate enough code the assets will outnumber the liabilities. It’s accounting 101. /s
    - chr15m 9mo ago
      Ha ha NGU!
- nomel 9mo ago
  The only people I've known that share this perspective are those that hate abstraction. Going back to their code, to extend it in some way, almost always requires a rewrite, because they wrote it with the goal of minimum viable complexity rather than understanding the realities of the real world problem they're solving, like "we all know we need these other features, but we have a deadline!" For one off, this is fine. For anything maintainable, that needs to survive the realities of time, this is truly terrible. Related, my friend works in a performance critical space. He can't use abstractions, because the direct, bare metal, "exact fit" implementation will perform best. They can't really add features, because it'll throw the timing of others things off to much, so usually have to re-architect. But, that's the reality of their problem space.
  - johnmwilkinson 9mo ago
    I believe this is conflating abstraction with encapsulation. The former is about semantic levels, the later about information hiding.
    - nomel 9mo ago
      Maybe I am? How is it possible to abstract without encapsulation? And also, how is it possible to encapsulate without abstracting some concept (intentionally or not) contained in that encapsulation? I can't really differentiate them, in the context of naming/referencing some list of CPU operations.
      - Retric 9mo ago
        > How is it possible to abstract without encapsulation. Historically pure machine code with jumps etc lacked any from of encapsulation as any data can be accessed and updated by anything. However, you would still use abstractions. If you pretend the train is actually going 80.2 MPH instead of somewhere between 80.1573 MPH to 80.2485 MPH which you got from different sensors you don’t need to do every calculation that follows twice.
        9mo ago
- cs_sorcerer 9mo ago
  Followed by even better is code is no code, and best is deleting code. It’s one of those things which has always strikes me funny about programming is how less usually really is more
- elgenie 9mo ago
  No code is as easy to maintain as no code. No code runs as fast as no code.
- doug_durham 9mo ago
  A better formulation is "every feature is a liability". Taking it to the line of code level is too prescriptive. Occasionally writing more verbose code is preferable if it makes it easier to understand.
  - phi-go 9mo ago
    This sounds wrong, features have to be the value of your code. The required maintenance and slow down to build more features (technical debt) are the liability, which is how I understood the relationship to "lines of code" anyway.
    - TeMPOraL 9mo ago
      Wrong or not, the industry embraced it. I can sort of understand it if I squint: every feature is a maintenance burden, and a risk of looking bad in front of users when you break or remove it, even if those users didn't use this feature. It's really a burden to be avoided when the point of your product is to grow its user base, not to actually be useful. Which explains why even Fischer-Price toys look more feature-ful and ergonomic than most new software products.
  - catdog 9mo ago
    > A better formulation is "every feature is a liability". Taking it to the line of code level is too prescriptive. Amount of code is a huge factor but maybe not the best wording here. It's more a thing of complexity where amount of code is a contributing metric but not the only one. You can very easily have a feature implemented in a too complex way and with too much code (esp. if an LLM generated the code but also with human developers). Also not every feature is equal. > Occasionally writing more verbose code is preferable if it makes it easier to understand. Think this is more a classic case of "if the metric becomes a goal it ceases to be a metric" than it being a bad metric per se.
- ky3 9mo ago
  EWD 1036: On the cruelty of really teaching computing science (1988) “My point today is that, if we wish to count lines of code, we should not regard them as ‘lines produced’ but as ‘lines spent’: the current conventional wisdom is so foolish as to book that count on the wrong side of the ledger.”
- ramraj07 9mo ago
  Agreed, every line you ship,whether you wrote it or not, you are responsible. In that regard, while I write a lot of code completely with AI, I still endeavor to keep the lines as minimal as possible. This means you never write both the main code and the tests using AI. Id rather have no tests than AI tests (we have QA team writing that up). This kinda works.
sjsizjhaha 9mo ago
Generating code was always cheap. That’s part of the reason this tech has to be forced on teams. Similar to the move to cloud, it’s the kind of cost that’s only gonna show up later - faster than the cloud move, I think. Though, in some cases it will be the correct choice.
zamalek 9mo ago
The main issue I've seen is it writing passing tests, the code being correct is a big (and often incorrect) assumption.
- 0x696C6961 9mo ago
  The majority of devs do the same thing.
akst 9mo ago
ATM I feel like LLM writing tests can be a bit dangerous at times, there are cases where it's fine there are cases where it's not. I don't really think I could articulate a systemised basis for identifying either case, but I know it when I see it I guess. Like the the other day, I gave it a bunch of use cases to write tests for, the use cases were correct the code was not, it saw one of the tests broken so it sought to rewrite the test. You risking suboptimal results when an agent is dictating its own success criteria. At one point I did try and use seperate Claude instances to write tests, then I'd get the other instance to write the implementation unaware of the tests. But it's a bit to much setup.
- icedchai 9mo ago
  I work with individuals who attempt to use LLMs to write tests. More than once, it's added nonsensical, useless test cases. Admittedly, humans do this, too, to a lesser extent. Additionally, if their code has broken existing tests, it "fixes" them by not fixing the code under test, but changing the tests... (assert status == 200 becomes 500 and deleting code.) Tests "pass." PR is opened. Reviewers wade through slop...
  - sigotirandolas 9mo ago
    The most annoying thing is that even after cleaning up all the nonsense, the tests still contain all sort of fanfare and it’s essentially impossible to get the submitter to trim them because it’s death by a thousand cuts (and you better not say "do it as if you didn’t use AI" in the current climate..)
    - icedchai 9mo ago
      Yep. We've had to throw PRs away and ask them to start over with a smaller set of changes since it became impossible to manage. Reviews went on for weeks. The individual couldn't justify why things were done (and apparently their AI couldn't, either!)
      - sigotirandolas 9mo ago
cheema33 9mo ago
This is how you do things if you are new to this game. Get two other, different, LLMs to thoroughly review the code. If you don’t have an automated way to do all of this, you will struggle and eventually put yourself out of a job. If you do use this approach, you will get code that is better than what most software devs put out. And that gives you a good base to work with if you need to add polish to it.
- 3kkdd 9mo ago
  Im sick and tired of these empty posts. SHOW AN EXAMPLE OF YOU ACTUALLY DOING WHAT YOU SAY!
  - cheema33 9mo ago
    The setup is much simpler than you might think. I have 4 CLI tools I use for this setup. Claude Code, Codex, Copilot and Cursor CLI. I asked Claude Code to create a code reviewer "skill" that uses the other 3 CLI tools to review changes in detail and provide feedback. I then ask Claude Code to use this skill to review any changes in code or even review plan documents. It is very very effective. Is it perfect? No. Nothing is. But, as I stated before, this produces results that are better than what an average developer sends in for PR review. Far far better in my own experience. In addition to that, we do use CodeRabbit plugin on GitHub to perform a 4th code review. And we tell all of our agents to not get into gold-plating mode. You can choose not to use modern tools like these to write software. You can also choose to write software in binary.
  - Foreignborn 9mo ago
    these two posts (the parent and then the OP) seem equally empty? by level of compute spend, it might look like: - ask an LLM in the same query/thread to write code AND tests (not good) - ask the LLM in different threads (meh) - ask the LLM in a separate thread to critique said tests (too brittle, testing guidelines, testing implementation and not out behavior, etc). fix those. (decent) - ask the LLM to spawn multiple agents to review the code and tests. Fix those. Spawn agents to critique again. Fix again. - Do the same as above, but spawn agents from different families (so Claude calls Gemini and Codex). —- these are usually set up as /slash commands like /tests or /review so you aren’t doing this manually. since this can take some time, people might work on multiple features at once.
visarga 9mo ago
> "write some unit tests for Redux today" The equivalent of "draw me a dog" -> not a masterpiece!? who would have thought? You need to come up with a testing methodology, write it down, and then ask the model to go through it. It likes to make assumptions on unspecified things, so you got to be careful. More fundamentally I think testing is becoming the core component we need to think about. We should not vibe-check AI code, we should code-check it. Of course it will write the actual test code, but your main priority is to think about "how do I test this?" You can only know the value of a code up to the level of its testing. You can't commit your eyes into the repo, so don't do "LGTM" vibe-testing of AI code, it's walking a motorcycle.
- overgard 9mo ago
  I think you're assuming a lot about my prompting. I know to be specific with LLMs
acemarke 9mo ago
Hi, I'm the primary Redux maintainer. I'd love to see some examples of what got generated! (Doubt there's anything we could do to _influence_ this, but curious what happened here.) FWIW we do have our docs on testing approaches here, and have recommended a more integrated-style approach to testing for a while: - https://redux.js.org/usage/writing-tests https://redux.js.org/usage/writing-tests
- overgard 9mo ago
  Unfortunately I think I cleaned up the code before committing it, so I don't have an exact example! I did actually read that usage page though after looking at those tests, and that helped me in fixing the tests (maybe in retrospect I should have pointed the AI at the docs page first). I think the main issue I was having with it was reusing the store object instead of creating a new one for each test. The other issue I was seeing was that it was creating mock objects and API's for things that weren't even being tested (lot of scope creep), and one of those API's was basically copy-pasted between two files (code duplication). It was also just testing things that weren't really necessary (ie, testing Redux itself, instead of my usage of it). Another issue was just taking a complex approach to fixing something that could be more easily solved. For instance, I had debug: true turned on for redux-undo so I was seeing some unnecessary log messages in the test. Codex identified this, and asked if I wanted to disable them, so I said yes. What it did though, was instead of setting debug: false, or disabling it on tests, it patched console.log to look for redux-undo prefixes. Technically worked, but kind of byzantine! None of this was a terrible disaster or anything, especially since I started pretty small, but I think what made me miss some of the issues at first glance is this is my first usage of Redux in a non-toy project so while I understand the fundamentals fine, it was easy to sneak working-but-bad stuff past me until I sat down with the code to write a test on my own and started to see the issues.
pmarreck 9mo ago
nobody is going to care about your bespoke codework if there is no downstream measurable difference
hahahahhaah 9mo ago
Living off only deep fried potatoes and large cola bottles is cheap...

jjimmy2k 9mo ago

A better agent can fix it in the future.

acedTrex 9mo ago

I've always said every line is a liability, its our job to limit liabilities. That has largely gone out the window these days.

chr15m 9mo ago
> every line is a liability, its our job to limit liabilities. Hard agree!
- ares623 9mo ago
  But more code from AI means stocks go up. Stocks are assets. If you generate enough code the assets will outnumber the liabilities. It’s accounting 101. /s
  - chr15m 9mo ago
    Ha ha NGU!
nomel 9mo ago
The only people I've known that share this perspective are those that hate abstraction. Going back to their code, to extend it in some way, almost always requires a rewrite, because they wrote it with the goal of minimum viable complexity rather than understanding the realities of the real world problem they're solving, like "we all know we need these other features, but we have a deadline!" For one off, this is fine. For anything maintainable, that needs to survive the realities of time, this is truly terrible. Related, my friend works in a performance critical space. He can't use abstractions, because the direct, bare metal, "exact fit" implementation will perform best. They can't really add features, because it'll throw the timing of others things off to much, so usually have to re-architect. But, that's the reality of their problem space.
- johnmwilkinson 9mo ago
  I believe this is conflating abstraction with encapsulation. The former is about semantic levels, the later about information hiding.
  - nomel 9mo ago
    Maybe I am? How is it possible to abstract without encapsulation? And also, how is it possible to encapsulate without abstracting some concept (intentionally or not) contained in that encapsulation? I can't really differentiate them, in the context of naming/referencing some list of CPU operations.
    - Retric 9mo ago
      > How is it possible to abstract without encapsulation. Historically pure machine code with jumps etc lacked any from of encapsulation as any data can be accessed and updated by anything. However, you would still use abstractions. If you pretend the train is actually going 80.2 MPH instead of somewhere between 80.1573 MPH to 80.2485 MPH which you got from different sensors you don’t need to do every calculation that follows twice.
      - 9mo ago
cs_sorcerer 9mo ago
Followed by even better is code is no code, and best is deleting code. It’s one of those things which has always strikes me funny about programming is how less usually really is more
elgenie 9mo ago
No code is as easy to maintain as no code. No code runs as fast as no code.
doug_durham 9mo ago
A better formulation is "every feature is a liability". Taking it to the line of code level is too prescriptive. Occasionally writing more verbose code is preferable if it makes it easier to understand.
- phi-go 9mo ago
  This sounds wrong, features have to be the value of your code. The required maintenance and slow down to build more features (technical debt) are the liability, which is how I understood the relationship to "lines of code" anyway.
  - TeMPOraL 9mo ago
    Wrong or not, the industry embraced it. I can sort of understand it if I squint: every feature is a maintenance burden, and a risk of looking bad in front of users when you break or remove it, even if those users didn't use this feature. It's really a burden to be avoided when the point of your product is to grow its user base, not to actually be useful. Which explains why even Fischer-Price toys look more feature-ful and ergonomic than most new software products.
- catdog 9mo ago
  > A better formulation is "every feature is a liability". Taking it to the line of code level is too prescriptive. Amount of code is a huge factor but maybe not the best wording here. It's more a thing of complexity where amount of code is a contributing metric but not the only one. You can very easily have a feature implemented in a too complex way and with too much code (esp. if an LLM generated the code but also with human developers). Also not every feature is equal. > Occasionally writing more verbose code is preferable if it makes it easier to understand. Think this is more a classic case of "if the metric becomes a goal it ceases to be a metric" than it being a bad metric per se.
ky3 9mo ago
EWD 1036: On the cruelty of really teaching computing science (1988) “My point today is that, if we wish to count lines of code, we should not regard them as ‘lines produced’ but as ‘lines spent’: the current conventional wisdom is so foolish as to book that count on the wrong side of the ledger.”
ramraj07 9mo ago
Agreed, every line you ship,whether you wrote it or not, you are responsible. In that regard, while I write a lot of code completely with AI, I still endeavor to keep the lines as minimal as possible. This means you never write both the main code and the tests using AI. Id rather have no tests than AI tests (we have QA team writing that up). This kinda works.

sjsizjhaha 9mo ago

Generating code was always cheap. That’s part of the reason this tech has to be forced on teams. Similar to the move to cloud, it’s the kind of cost that’s only gonna show up later - faster than the cloud move, I think. Though, in some cases it will be the correct choice.

zamalek 9mo ago

The main issue I've seen is it writing passing tests, the code being correct is a big (and often incorrect) assumption.

0x696C6961 9mo ago
The majority of devs do the same thing.

akst 9mo ago

ATM I feel like LLM writing tests can be a bit dangerous at times, there are cases where it's fine there are cases where it's not. I don't really think I could articulate a systemised basis for identifying either case, but I know it when I see it I guess. Like the the other day, I gave it a bunch of use cases to write tests for, the use cases were correct the code was not, it saw one of the tests broken so it sought to rewrite the test. You risking suboptimal results when an agent is dictating its own success criteria. At one point I did try and use seperate Claude instances to write tests, then I'd get the other instance to write the implementation unaware of the tests. But it's a bit to much setup.

icedchai 9mo ago
I work with individuals who attempt to use LLMs to write tests. More than once, it's added nonsensical, useless test cases. Admittedly, humans do this, too, to a lesser extent. Additionally, if their code has broken existing tests, it "fixes" them by not fixing the code under test, but changing the tests... (assert status == 200 becomes 500 and deleting code.) Tests "pass." PR is opened. Reviewers wade through slop...
- sigotirandolas 9mo ago
  The most annoying thing is that even after cleaning up all the nonsense, the tests still contain all sort of fanfare and it’s essentially impossible to get the submitter to trim them because it’s death by a thousand cuts (and you better not say "do it as if you didn’t use AI" in the current climate..)
  - icedchai 9mo ago
    Yep. We've had to throw PRs away and ask them to start over with a smaller set of changes since it became impossible to manage. Reviews went on for weeks. The individual couldn't justify why things were done (and apparently their AI couldn't, either!)
    - sigotirandolas 9mo ago

cheema33 9mo ago

This is how you do things if you are new to this game. Get two other, different, LLMs to thoroughly review the code. If you don’t have an automated way to do all of this, you will struggle and eventually put yourself out of a job. If you do use this approach, you will get code that is better than what most software devs put out. And that gives you a good base to work with if you need to add polish to it.

3kkdd 9mo ago
Im sick and tired of these empty posts. SHOW AN EXAMPLE OF YOU ACTUALLY DOING WHAT YOU SAY!
- cheema33 9mo ago
  The setup is much simpler than you might think. I have 4 CLI tools I use for this setup. Claude Code, Codex, Copilot and Cursor CLI. I asked Claude Code to create a code reviewer "skill" that uses the other 3 CLI tools to review changes in detail and provide feedback. I then ask Claude Code to use this skill to review any changes in code or even review plan documents. It is very very effective. Is it perfect? No. Nothing is. But, as I stated before, this produces results that are better than what an average developer sends in for PR review. Far far better in my own experience. In addition to that, we do use CodeRabbit plugin on GitHub to perform a 4th code review. And we tell all of our agents to not get into gold-plating mode. You can choose not to use modern tools like these to write software. You can also choose to write software in binary.
- Foreignborn 9mo ago
  these two posts (the parent and then the OP) seem equally empty? by level of compute spend, it might look like: - ask an LLM in the same query/thread to write code AND tests (not good) - ask the LLM in different threads (meh) - ask the LLM in a separate thread to critique said tests (too brittle, testing guidelines, testing implementation and not out behavior, etc). fix those. (decent) - ask the LLM to spawn multiple agents to review the code and tests. Fix those. Spawn agents to critique again. Fix again. - Do the same as above, but spawn agents from different families (so Claude calls Gemini and Codex). —- these are usually set up as /slash commands like /tests or /review so you aren’t doing this manually. since this can take some time, people might work on multiple features at once.

visarga 9mo ago

> "write some unit tests for Redux today" The equivalent of "draw me a dog" -> not a masterpiece!? who would have thought? You need to come up with a testing methodology, write it down, and then ask the model to go through it. It likes to make assumptions on unspecified things, so you got to be careful. More fundamentally I think testing is becoming the core component we need to think about. We should not vibe-check AI code, we should code-check it. Of course it will write the actual test code, but your main priority is to think about "how do I test this?" You can only know the value of a code up to the level of its testing. You can't commit your eyes into the repo, so don't do "LGTM" vibe-testing of AI code, it's walking a motorcycle.

overgard 9mo ago
I think you're assuming a lot about my prompting. I know to be specific with LLMs

acemarke 9mo ago

Hi, I'm the primary Redux maintainer. I'd love to see some examples of what got generated! (Doubt there's anything we could do to _influence_ this, but curious what happened here.) FWIW we do have our docs on testing approaches here, and have recommended a more integrated-style approach to testing for a while: - https://redux.js.org/usage/writing-tests https://redux.js.org/usage/writing-tests

overgard 9mo ago
Unfortunately I think I cleaned up the code before committing it, so I don't have an exact example! I did actually read that usage page though after looking at those tests, and that helped me in fixing the tests (maybe in retrospect I should have pointed the AI at the docs page first). I think the main issue I was having with it was reusing the store object instead of creating a new one for each test. The other issue I was seeing was that it was creating mock objects and API's for things that weren't even being tested (lot of scope creep), and one of those API's was basically copy-pasted between two files (code duplication). It was also just testing things that weren't really necessary (ie, testing Redux itself, instead of my usage of it). Another issue was just taking a complex approach to fixing something that could be more easily solved. For instance, I had debug: true turned on for redux-undo so I was seeing some unnecessary log messages in the test. Codex identified this, and asked if I wanted to disable them, so I said yes. What it did though, was instead of setting debug: false, or disabling it on tests, it patched console.log to look for redux-undo prefixes. Technically worked, but kind of byzantine! None of this was a terrible disaster or anything, especially since I started pretty small, but I think what made me miss some of the issues at first glance is this is my first usage of Redux in a non-toy project so while I understand the fundamentals fine, it was easy to sneak working-but-bad stuff past me until I sat down with the code to write a test on my own and started to see the issues.

pmarreck 9mo ago

nobody is going to care about your bespoke codework if there is no downstream measurable difference

hahahahhaah 9mo ago

Living off only deep fried potatoes and large cola bottles is cheap...