8 ms·
Because code isn't free. I can see it in my team. We've all been using Claude a lot for the last 6 months. It's hard to measure the impact, but I can tell our
by yodsanklai 8mo ago
Because code isn't free.
I can see it in my team. We've all been using Claude a lot for the last 6 months. It's hard to measure the impact, but I can tell our systems are as buggy as ever. AI isn't a silver bullet.
- OsrsNeedsf2P 8mo agoWhy isn't Claude doing QA testing for you?
- slopinthebag 8mo agoI can't tell if this is sarcasm, but if not, you cant rely on the thing that produced invalid output to validate it's own output. That is fundementally insufficient, despite it potentially catching some errors.
- _2d30 8mo agoDamn. Guess I'll stop QAing my own work from now.
- Spivak 8mo agoI mean there is some wisdom to that, most teams separate dev and qa and writers aren't their own editors precisely because it's hard for the author of a thing to spot their own mistakes. When you merge them into one it's usually a cost saving measure accepting that quality control will take a hit.
- meheleventyone 8mo agoUh, yeah, thh hi is has been considered bad practice for decades.
- ryan_n 8mo agoThat is often how software development has been done the past several decades yea... Not to say that you don't review your own work, but it's good practice for others (or at least one other person) to review it/QA it as well.
- slopinthebag 8mo agoYou're making a false equivalence between a human being with agency and intelligence, and a machine.
- rhubarbtree 8mo agoAre humans not machines?
- sarchertech 8mo agoThat’s something that more than half of humans would disagree with (exact numbers vary but most polls show that more than 75% of people globally believe that humans have a soul or spirit). But ignoring that, if humans are machines, they are sufficiently advanced machines that we have only a very modest understanding of and no way to replicate. Our understanding of ourselves is so limited that we might as well be magic.
- jatari 8mo ago>if humans are machines, they are sufficiently advanced machines that we have only a very modest understanding of and no way to replicate Well, ignoring the whole literal replication thing humans do.
- sarchertech 8mo agoObviously by replicate I meant building a synthetic human.
- rhubarbtree 8mo agoSo good we're magic. So bad we think we're magic.
- slopinthebag 8mo agoObviously not? Like how is that a serious question...
- habinero 8mo agoYeah, someone should invent code review.
- alistairSH 8mo agoYes. That’s not a best practice. That’s why PRs and peer reviews and test automation suite exist.
- akdev1l 8mo agoI think it is common for one to write their own tests tho
- alistairSH 8mo agoHe said QA. QA is more than just unit tests.
- akdev1l 8mo agoWhatever level of automated testing, it’s all usually done by the same people who wrote the software to begin with
- lioeters 8mo agoThis but unironically. Of course review your own work. But QA is best done by people other than those who develop the product. Having another set of eyes to check your work is as old as science.
- iagooar 8mo agoWhat if "the thing" is a human and another human validating the output. Is that its own output (= that of a human) or not? Doesn't this apply to LLMs - you do not review the code within the same session that you used to generate the code?
- slopinthebag 8mo agoI think a human and an LLM are fundamentally different things, so no. Otherwise you could make the argument that only something extra-terrestrial could validate our work, since LLM's like all machines are also our outputs.
- koolba 8mo agoThe problem now is that it’s a human using Claude to write the code and another using Claude to review it.
- charcircuit 8mo agoProducts don't have to be perfect. If they can be less buggy than before AI. You can't call that anything but a win.
- Nition 8mo agoThat's why you get Codex to do it. /s
- latchkey 8mo ago> you cant rely on the thing that produced invalid output to validate it's own output I've been coding an app with the help of AI. At first it created some pretty awful unit tests and then over time, as more tests were created, it got better and better at creating tests. What I noticed was that AI would use the context from the tests to create valid output. When I'd find bugs it created, and have AI fix the bugs (with more tests), it would then do it the right way. So it actually was validating the invalid output because it could rely on other behaviors in the tests to find its own issues. The project is now at the point that I've pretty much stopped writing the tests myself. I'm sure it isn't perfect, but it feels pretty comprehensive at 693 tests. Feel free to look at the code yourself [0]. [0] https://github.com/OrangeJuiceExtension/OrangeJuice/actions/runs/22045717903/job/63694063998#step:6:88 https://github.com/OrangeJuiceExtension/OrangeJuice/actions/...
- slopinthebag 8mo agoI'm not saying you can't do it, I'm just saying it's not sufficient on its own. I run my code through an LLM and it occasionally catches stuff I missed.
- latchkey 8mo agoThanks for the clarification. That's the difference though, I don't need it to catch stuff I missed, I catch stuff it misses and I tell it to add it, which it dutifully does.
- CamperBob2 8mo agoI can't tell if that is sarcasm. Of course you can use the same model to write tests. That's a different problem altogether, with a different series of prompts altogether! When it comes to code review, though, it can be a good idea to pit multiple models against each other. I've relied on that trick from day 1.
- huslage 8mo agoI have had other LLMs QA the work of Claude Code and they find bugs. It's a good cycle, but the bugs almost never get fixed in one-shot without causing chaos in the codebase or vast swaths of rewritten code for no reason.
- PunchyHamster 8mo agoWhy isn't it doing it for Anthropic ?
- ffsm8 8mo agoWhat makes you think it isn't? They just have a lot of users doing QA to, and ignore any of their issues like true champs
- reconnecting 8mo agoAnd after 12 months, most probably no one from your team will understand what the result of half of those bugs is. When devs outsource their thinking to AI, they lose the mental map, and without it, control over the entire system.
- Dig1t 8mo agoOnly if they are supremely lazy. It’s possible to use these tools in a diligent way, where you maintain understanding and control of the system but outsource the implementation of tasks to the LLM. An engineer should be code reviewing every line written by an LLM, in the same way that every line is normally code reviewed when written by a human. Maybe this changes the original argument from software being “free”, but we could just change that to mean “super cheap”.
- collinvandyck76 8mo agoThere's a pretty big difference between the understanding that comes with reviewing code versus writing it, for most people I think.
- deleted 8mo ago[deleted]
- macintux 8mo agoDefinitely true for me. What’s particularly problematic is code I need to review but can’t effectively test due to environmental challenges.
- mapontosevenths 8mo agoThats a tough situation. How do you handle the testing with human code?
- macintux 8mo ago
- neal_jones 8mo agoDude, I blame all bugs on ai at this point. I suspect one could roughly identify AI’s entry into the game based on some metric of large system outages. Assume someone has already done this but…probably doesn’t matter.
- XenophileJKO 8mo agoI love the fact that we just got a model really capable of doing sustained coding (let me check my notes here...) 3 months ago, with a significant bump 15 days ago. And now the comments are "If it is so great why isn't everything already written from scratch with it?"
- hyperpape 8mo agoAh, so it's free, but you still have to wait 3 months. Just a question...what are you waiting for? Of course the answer is all the things that aren't free, refinement, testing, bug fixes, etc, like the parent post and the article suggested.
- kavok 8mo agoI feel like people have been saying AI was great for years now?
- rubenflamshep 8mo agoPeople are getting caught up in the "fast (but slow) diffusion)" that Dario has spoken to. Adoption of these tools has been fast but not instant but people will poke holes via "well, it hasn't done x yet". For my own work I've focused on using the agents to help clean up our CICD and make it more robust, specifically because the rest of the company is using agents more broadly. Seems like a way to leverage the technology in a non-slop oriented way
- Sateeshm 8mo agoWell the company keeps saying coding is a solved problem.
- LtWorf 8mo agoAnd you presume they are being completely honest about their capabilities why?
- AceJohnny2 8mo agoI'm reminded of the viral comic "I'm stupid faster" (2019?) by Shen https://imgur.com/gallery/i-m-stupid-faster-u8crXcq https://imgur.com/gallery/i-m-stupid-faster-u8crXcq (sorry for Imgur link, but Shen's web presence is a mess and it's hard to find a canonical source) I'm not saying this is completely the case for AI coding agents, whose capabilities and trustworthiness have seen a meteoric rise in the past year.
- deleted 8mo ago[deleted]