3 ms·
Not in the slightest. Generating a snippet, or even entire blocks, of code is not a good metric on which to base the quality of LLMs. Rather, we should pay at
by tobinfekkes 3y ago
Not in the slightest.
Generating a snippet, or even entire blocks, of code is not a good metric on which to base the quality of LLMs.
Rather, we should pay attention to how good is an LLM at altering, editing, maintaining code? In practice, it's quite poor. And not in the way that a new developer is poor, but in weird and uncertain and unpredictable ways, or in ways that it fixed in the last prompt but broke in the new prompt.
Engineers spend little time writing code compared to the amount of time they spend thinking and designing and buildings systems for that code to operate in. LLMs are terrible at systems.
"Systems" implies layers, and layers implies constraints, and constraints implies specificity, and specificity is not a strong suit of an LLM.
The systems in which most code is written is not written once and never touched again. Instead, it passes through many layers (automated and manual) to reach some arbitrary level of correctness across many vectors, in many contexts, some digital and some not. LLMs are passable at solving problems of one, maybe two, layers of complexity as long as all the context is digitized first.
Lastly, not all problems have a technical solution. In fact, creating a technical solution for a non-technical problem so that we can apply an LLM to it will only cause layers of unnecessary abstraction that locks out "normal" people from participating in the solution.
And we may not like to admit it, but we engineers often have work to do for a job we like for pay that's pretty good, only because there are a whole lot of non-programmers out there solving the non-technical problems that we don't want to do.
- crudcodersare 3y agoClearly, you have your nose stuck in the past. LLMs aren't just good; they're revolutionary, making coders look like one finger keyboard peckers. They're not struggling with maintaining systems; they can be trained on specific datasets and codebases, totally mastering them in ways no human can ever hope to do, making your traditional coding seem like manual typewriting and laughable in comparison. The idea that they can't handle specificity is also laughable. LLMs are not the future; they're the now, effortlessly bridging tech and non-tech worlds, totally replacing programmers. Anyone thinking otherwise is simply not paying attention. The writing is on the wall.
- randysalami 3y agoCan I ask how you’re accomplishing this? What tools and workflows are you using to have LLMs actually do this level of development in the present? Sounds like you are at the cutting edge and I’m very curious since I still hear copilot being touted as “cutting edge”. Thanks!
- lurking_swe 3y agothis sounds like something an LMM wrote lol. Answer me this: often times even my product manager doesn’t know WHAT they want. They think they know but they don’t really, once you start looking at the code, there are dozens of edge cases throughout the system. The requirements are vague. How do you expect an LLM to magically figure that out? Or more importantly, the business domain (a domain not easily trained on textbooks or crap found on the internet)? And I don’t have my head stuck in the sand. I’ve found LLM’s great for brainstorming, implementing small functions, or updating snippets of configuration like terraform.