4 ms·
There really is a category of these posts that are coming from some alternate dimension (or maybe we're in the alternate dimension and they're in the real one?)
by tptacek 2y ago
There really is a category of these posts that are coming from some alternate dimension (or maybe we're in the alternate dimension and they're in the real one?) where this isn't one of the most important things ever to happen to software development. I'm a person who didn't even use autocomplete (I use LSPs almost entirely for cross-referencing --- oh wait that's another thing I'm apparently never going to need to do again because of LLMs), a sincere tooling skeptic. I do not understand how people expect to write convincingly that tools that reliably turn slapdash prose into median-grade idiomatic working code "provide little value".
- quesera 2y ago> tools that reliably turn slapdash prose into median-grade idiomatic working code This may be the crux of it. Turning slapdash prose into median-grade code is not a problem I can imagine needing to solve. I think I'm better at describing code in code than I am in prose. I Want to Believe. And I certainly don't want to be "that guy", but my honest assessment of LLMs for coding so far is that they are a frustrating Junior, who maybe I should help out because mentoring might be part of my job, but from whom I should not expect any near-term technical contribution.
- tptacek 2y agoIt is most of the problem of delivering professional software.
- quesera 2y agoNot in my experience. The only slapdash prose in the cycle is in the immediate output of a product development discussion. And that is inevitably too sparse to inform, without the full context of the team, company, and industry.
- akerl_ 2y agoSorry, are you saying "the only place where there's slapdash prose is right before it would be super cool to have an alpha version of the code magically appear, that we can iterate on based on the full context of the team, company, and industry"?
- quesera 2y agoNo, not at all. Alpha code with zero context is an utter waste of attention. I must be confused about how y'all are developing software, because the path from "incompletely specified takeaways from a product design meeting", and "final product" does not pass through any intermediate steps where reduced contextual awareness is valuable. Writing code is not the hard part.
- tptacek 2y agoWhere's "zero context" coming from here?
- tptacek 2y agoI didn't say anything about "slapdash".
- quesera 2y agoUmm. Yeah, I think ya did. :)
- tptacek 2y agoNo.
- quesera 2y agoYou introduced the word into the thread. I quoted you. Unless you're operating at some notational level above the literal, yes I think you did.
- tptacek 2y agoSorry, I was referring the the prompt, not the code.
- quesera 2y agoI was referring to the prompt/prose as well. The median-quality code just doesn't seem like a valuable asset en route to final product, but I guess it's a matter of process at that point. Generative AI, as I've managed to use it, brings me to a place in the software lifecycle that I don't want to be. Median-quality code that lacks the context or polish needed to be usable. Or in some cases even parseable. I may be missing essential details though. Smart people are getting more out of AI than I am. I'd love to see a Youtube/Twitch/etc video of someone who knows what they're doing demoing the build of a typical TODO app or similar, from paragraphs to product.
- tptacek 2y agoMedian-quality code is extraordinarily valuable. It is most of the load-bearing code people actually ship. What's almost certainly happening here is that you and I have differing definitions of "median-quality" commercial code. I'm pretty sure that if we triangle-tested (say) a Go project from 'jerf and Gemini 2.5 Go output for the same (substantial; say, 2,000 lines) project --- not whatever Gemini's initial spew is, but a final product where Gemini is the author of 80+% of the lines --- you would not be able to pick the human code out from the LLM code.
- pcwalton 2y ago> I do not understand how people expect to write convincingly that tools that reliably turn slapdash prose into median-grade idiomatic working code "provide little value". Honestly, I'm curious why your experience is so different from mine. Approximately 50% of the time for me, LLMs hallucinate APIs, which is deeply frustrating and sometimes costs me more time than it would have taken to just look up the API. I still use them regularly, and the net value they've imparted has been overall greater than zero, but in general, my experience has been decidedly mixed. It might be simply that my code tends to be in specialized areas in which the LLM has little training data. Still, I get regular frustrating API hallucinations even in areas you'd think would be perfect use cases, like writing Blender plugins, where the documentation is poor (so the LLM has a relatively higher advantage over reading the documentation) and examples are plentiful. Edit: Specifically, the frustrating pattern is: (1) the LLM produces some code that contains hallucinated APIs; (2) in order to test (or even compile) that code, I need to write some extra supporting code to integrate it into my project; (3) I discover that the APIs were hallucinated because the code doesn't work; (4) now I not only have to rewrite the LLM's code, but I also have to rewrite all the supporting code I wrote, because it was based around a pattern that didn't work. Overall, this adds up to more time than if I had just written the code from scratch.
- senordevnyc 2y agoOne of the frustrating things about talking about this is that the discussion often sounds like we're all talking about the same thing when we talk about "AI". We're not. Not only does it matter what language you code in, but the model you use and the context you give it also matter tremendously. I'm a huge fan of AI-assisted coding, it's probably writing 80-90% of my code at this point, but I've had all the same experiences that you have, and still do sometimes. There's a steep learning curve to leveraging AIs effectively, and I think a lot of programmers stop before they get far enough along on that curve to see the magic. For example, right now I'm coding with Cursor and I'm alternating between Claude 3.7 max, Gemini 2.5 pro max, and o3. They all have their strengths and weaknesses, and all cost for usage above the monthly subscription. I'm spending like $10 per day on these models at the moment. I could just use the models included with the subscription, but they tend to hallucinate more, or take odd steps around debugging, etc. I've also got a bunch of documents and rules setup for Cursor to guide it in terms of what kinds of context to include for the model. And on top of that, there are things I'm learning about what works best in terms of how to phrase my requests, what to emphasize or tell the model NOT to do, etc. Currently I usually start by laying out as much detail about the problem as I can, pointing to relevant files or little snippets of other code, linking to docs, etc, and asking it to devise a plan for accomplishing the task, but not to write any code. We'll go back and forth on the plan, then I'll have it implement test coverage if it makes sense, then run the tests and iterate on the implementation until they're green. It's not perfect, I have to stop it and backup often, sometimes I have to dig into docs and get more details that I can hand off to shape the implementation better, etc. I've cursed in frustration at whatever model I'm using more than once. But overall, it helps me write better code, faster. I never could have built what I've built over the last year without AI. Never.