3 ms·
This last week: * One developer tried to refactor a bunch of graph ql with an LLM and ended up checking in a bunch of completely broken code. Thankfully there
by Verdex 10mo ago
This last week:
* One developer tried to refactor a bunch of graph ql with an LLM and ended up checking in a bunch of completely broken code. Thankfully there were api tests.
* One developer has an LLM making his PRs. He slurped up my unfinished branch, PRed it, and merged (!) it. One can only guess that the approved was also using an LLM. When I asked him why he did it, he was completely baffled and assured me he would never. Source control tells a different story.
* And I forgot to turn off LLM auto complete after setting up my new machine. The LLM wouldn't stop hallucinating non-existent constructors for non-existent classes. Bog standard intellisense did in seconds what I needed after turning off LLM auto complete.
LLMs sometimes save me some time. But overall I'm sitting at a pretty big amount of time wasted by them that the savings have not yet offset.
- brabel 10mo ago> One developer tried to refactor a bunch of graph ql with an LLM and ended up checking in a bunch of completely broken code. Thankfully there were api tests. So the LLM was not told how to run the tests? Without that they cannot know if what they did works, and they are a bit like humans, they try something and then they need to check if that does the right thing. Without a test cycle you definitely don’t get a lot out of LLMs.
- Capricorn2481 10mo agoYou guys always find a way to say "you can be an LLM maximalist too, you just skipped a step." The bigger story here is not that they forgot to tell the LLM to run tests, it's that agentic use has been so normalized and overhyped that an entire PR was attempted without any QA. Even if you're personally against this, this is how most people talk about agents online. You don't always have the privilege of working on a project with tests, and rarely are they so thorough that they catch everything. Blindly trusting LLM output without QA or Review shouldn't be normalized.
- blitz_skull 10mo agoWho is normalizing merging ANYTHING, LLM-generated or human-generated, without QA or review? You should be reviewing everything that touches your codebase regardless of source.
- Capricorn2481 10mo agoA LOT of people, if you're paying attention. Why do you think that happened at their company? It's not hard to find comments from people vibe coding apps without understanding the code, even apps handling sensitive data. And it's not hard to find comments saying agents can run by themselves. I mean people are arguing AGI is already here. What do you mean who is normalizing this?
- blitz_skull 10mo agoI fully believe there are misguided leaders advocating for "increasing velocity" or "productivity" or whatever, but the technical leaders should be pushing back. You can't make a ship go faster by removing the hull. And if you want to try... well you get what you get! But again, no one who is serious about their business and serious about building useful products is doing this.
- Verdex 10mo ago> But again, no one who is serious about their business and serious about building useful products is doing this. While this is potentially true for software companies, there are many companies for which software or even technology in general is not a core competency. They are very serious about their very useful products. They also have some, er, interesting ideas about what LLMs allow them to accomplish.
- YouAreWRONGtoo 10mo ago[dead]
- brabel 10mo ago
- st3fan 10mo agoThe first two cases indicate that you have some gaps in your change management process. Strict requirements for pulls and ci/cd checks.