3 ms·
Why isn't Claude doing QA testing for you?
by OsrsNeedsf2P 8mo ago
Why isn't Claude doing QA testing for you?
- slopinthebag 8mo agoI can't tell if this is sarcasm, but if not, you cant rely on the thing that produced invalid output to validate it's own output. That is fundementally insufficient, despite it potentially catching some errors.
- _2d30 8mo agoDamn. Guess I'll stop QAing my own work from now.
- Spivak 8mo agoI mean there is some wisdom to that, most teams separate dev and qa and writers aren't their own editors precisely because it's hard for the author of a thing to spot their own mistakes. When you merge them into one it's usually a cost saving measure accepting that quality control will take a hit.
- meheleventyone 8mo agoUh, yeah, thh hi is has been considered bad practice for decades.
- ryan_n 8mo agoThat is often how software development has been done the past several decades yea... Not to say that you don't review your own work, but it's good practice for others (or at least one other person) to review it/QA it as well.
- slopinthebag 8mo agoYou're making a false equivalence between a human being with agency and intelligence, and a machine.
- rhubarbtree 8mo agoAre humans not machines?
- sarchertech 8mo agoThat’s something that more than half of humans would disagree with (exact numbers vary but most polls show that more than 75% of people globally believe that humans have a soul or spirit). But ignoring that, if humans are machines, they are sufficiently advanced machines that we have only a very modest understanding of and no way to replicate. Our understanding of ourselves is so limited that we might as well be magic.
- jatari 8mo ago>if humans are machines, they are sufficiently advanced machines that we have only a very modest understanding of and no way to replicate Well, ignoring the whole literal replication thing humans do.
- sarchertech 8mo agoObviously by replicate I meant building a synthetic human.
- rhubarbtree 8mo agoSo good we're magic. So bad we think we're magic.
- slopinthebag 8mo agoObviously not? Like how is that a serious question...
- habinero 8mo agoYeah, someone should invent code review.
- alistairSH 8mo agoYes. That’s not a best practice. That’s why PRs and peer reviews and test automation suite exist.
- akdev1l 8mo agoI think it is common for one to write their own tests tho
- alistairSH 8mo agoHe said QA. QA is more than just unit tests.
- akdev1l 8mo agoWhatever level of automated testing, it’s all usually done by the same people who wrote the software to begin with
- lioeters 8mo agoThis but unironically. Of course review your own work. But QA is best done by people other than those who develop the product. Having another set of eyes to check your work is as old as science.
- iagooar 8mo agoWhat if "the thing" is a human and another human validating the output. Is that its own output (= that of a human) or not? Doesn't this apply to LLMs - you do not review the code within the same session that you used to generate the code?
- slopinthebag 8mo agoI think a human and an LLM are fundamentally different things, so no. Otherwise you could make the argument that only something extra-terrestrial could validate our work, since LLM's like all machines are also our outputs.
- koolba 8mo agoThe problem now is that it’s a human using Claude to write the code and another using Claude to review it.
- charcircuit 8mo agoProducts don't have to be perfect. If they can be less buggy than before AI. You can't call that anything but a win.
- Nition 8mo agoThat's why you get Codex to do it. /s
- latchkey 8mo ago> you cant rely on the thing that produced invalid output to validate it's own output I've been coding an app with the help of AI. At first it created some pretty awful unit tests and then over time, as more tests were created, it got better and better at creating tests. What I noticed was that AI would use the context from the tests to create valid output. When I'd find bugs it created, and have AI fix the bugs (with more tests), it would then do it the right way. So it actually was validating the invalid output because it could rely on other behaviors in the tests to find its own issues. The project is now at the point that I've pretty much stopped writing the tests myself. I'm sure it isn't perfect, but it feels pretty comprehensive at 693 tests. Feel free to look at the code yourself [0]. [0] https://github.com/OrangeJuiceExtension/OrangeJuice/actions/runs/22045717903/job/63694063998#step:6:88 https://github.com/OrangeJuiceExtension/OrangeJuice/actions/...
- slopinthebag 8mo agoI'm not saying you can't do it, I'm just saying it's not sufficient on its own. I run my code through an LLM and it occasionally catches stuff I missed.
- latchkey 8mo agoThanks for the clarification. That's the difference though, I don't need it to catch stuff I missed, I catch stuff it misses and I tell it to add it, which it dutifully does.
- CamperBob2 8mo agoI can't tell if that is sarcasm. Of course you can use the same model to write tests. That's a different problem altogether, with a different series of prompts altogether! When it comes to code review, though, it can be a good idea to pit multiple models against each other. I've relied on that trick from day 1.
- huslage 8mo agoI have had other LLMs QA the work of Claude Code and they find bugs. It's a good cycle, but the bugs almost never get fixed in one-shot without causing chaos in the codebase or vast swaths of rewritten code for no reason.
- PunchyHamster 8mo agoWhy isn't it doing it for Anthropic ?
- ffsm8 8mo agoWhat makes you think it isn't? They just have a lot of users doing QA to, and ignore any of their issues like true champs