4 ms·

>> > LLM’s never provide code that pass my sniff test > This is ego speaking. Consider this, 100% of AI training data is human-generated content. Generally s

>> > LLM’s never provide code that pass my sniff test > This is ego speaking. Consider this, 100% of AI training data is human-generated content. Generally speaking, we apply the 90/10 rule to human generated content: 90% of (books, movies, tv shows, software applications, products available on Amazon) is not very good. 10% shines. In software development, I would say it's more like 99 to 1 after working in the industry professionally for over 25 years. How do I divorce this from my personal ego? It's easy to apply objective criteria: - Is the intent of code easy to understand? - Are the "moving pieces" isolated, such that you can change the implementation of one with minimal risk of altering the others by mistake? - Is the solution in code a simple one relative to alternatives? The majority of human produced code does not pass the above sniff test. Most of my job, as a Principal on a platform team, is cleaning up other peoples' messes and training them how to make less of a mess in the future. If the majority of human generated content fails to follow basic engineering practices that are employed in other engineering disciplines (i.e: it never ceases to amaze me how much of an uphill battle it is just to get some SWEs just to break down their work into small, single responsibility, easily testable and reusable "modules") then we can't logically expect any better from LLMs because this is what they're being trained on. And we are VERY far off from LLMs that can weigh the merits of different approaches within the context of the overall business requirements and choose which one makes the most sense for the problem at hand, as opposed to just "what's the most common answer to this question?" LLMs today are a type of magic trick. You give it a whole bunch of 1s and 0s so that you can input some new 1s and 0s and it can use some fancy proability maths to predict "based on the previous 1s and 0s, what are the statistically most likely next 1s and 0s to follow from the input?" That is useful, and the result can be shockingly impressive depending on what you're trying to do. But the limitations are so limited that the prospect of replacing an entire high-skilled profession with that magic trick is kind of a joke.

m_ke 2y ago
Your customers don't care how your code smells, as long as it solves their problem and doesn't cost an arm and a leg. A ton of huge business full of Sr Principal Architect SCRUM masters are about to get disrupted by 80 line ChatGPT wrappers hacked together by a few kids in their dorm room.
- gspencley 2y ago
  > Your customers don't care how your code smells, as long as it solves their problem and doesn't cost an arm and a leg. Software is interesting because if you buy a refrigerator, even an inexpensive one, you have certain expectations as to its basic functions. If the compressor were to cut out periodically in unexpected ways, affecting your food safety, you would return it. But in software customers seem to be conditioned to just accept bugs and poor performance as a fact of life. You're correct that customers don't care about "code quality", because they don't understand code or how to evaluate it. But you're assuming that customers don't care about the quality of the product they are paying for, and you're divorcing that quality from the quality of the code as if the code doesn't represent THE implementation of the final product. The hardware matters too, but to assume that code quality doesn't directly affect product quality is to pretend that food quality is not directly impacted by its ingredients.
  - throwaway_43793 2y ago
    Code quality does not affect final product quality IMHO. I worked in companies with terrible code, that deployed on an over-engineered cloud provider using custom containers hacked with a nail and a screwdriver, but the product was excellent. Had bugs here and there, but worked and delivered what needs to be delivered. SWEs need to realize that code doesn't really matter. For 70 years we are debating the best architecture patterns and yet the biggest fear of every developer is working on legacy code, as it's an unmaintainable piece of ... written by humans.
    - gspencley 2y ago
      > Code quality does not affect final product quality IMHO. What we need, admittedly, is more research and study around this. I know of one study which supports my position, but I'm happy to admit that the data is sparse. https://arxiv.org/abs/2203.04374 https://arxiv.org/abs/2203.04374 From the abstract: "By analyzing activity in 30,737 files, we find that low quality code contains 15 times more defects than high quality code."
rybosworld 2y ago
> That is useful, and the result can be shockingly impressive depending on what you're trying to do. But the limitations are so limited that the prospect of replacing an entire high-skilled profession with that magic trick is kind of a joke. The possible outcome space is not binary (at least in the near term), i.e. either AI replace devs, or it doesn't. What I'm getting at is this: There's a pervasive attitude among some developers (generally older developers, in my experience) that LLM's are effectively useless. If we're being objective, that is quite plainly not true. These conversations tend to start out with something like: "Well _my_ work in particular is so complex that LLM's couldn't possibly assist." As the conversation grows, the tone gradually changes to admitting: "Yes there are some portions of a codebase where LLM's can be helpful, but they can't do _everything_ that an experienced dev does." It should not even be controversial to say, that AI will only improve at this task. That's what technology does, over the long run. Fundamentally, there's ego involved whenever someone says "LLM's have _never_ produced useable code." That statement, is provably false.
ben_w 2y ago
> Consider this, 100% of AI training data is human-generated content. Probably doesn't change the conclusion, but this is no longer the case. I've yet to try Phi-4, but the whole series has synthetic training data; I don't value Phi-3 despite what the benchmarks claim.

m_ke 2y ago

Your customers don't care how your code smells, as long as it solves their problem and doesn't cost an arm and a leg. A ton of huge business full of Sr Principal Architect SCRUM masters are about to get disrupted by 80 line ChatGPT wrappers hacked together by a few kids in their dorm room.

gspencley 2y ago
> Your customers don't care how your code smells, as long as it solves their problem and doesn't cost an arm and a leg. Software is interesting because if you buy a refrigerator, even an inexpensive one, you have certain expectations as to its basic functions. If the compressor were to cut out periodically in unexpected ways, affecting your food safety, you would return it. But in software customers seem to be conditioned to just accept bugs and poor performance as a fact of life. You're correct that customers don't care about "code quality", because they don't understand code or how to evaluate it. But you're assuming that customers don't care about the quality of the product they are paying for, and you're divorcing that quality from the quality of the code as if the code doesn't represent THE implementation of the final product. The hardware matters too, but to assume that code quality doesn't directly affect product quality is to pretend that food quality is not directly impacted by its ingredients.
- throwaway_43793 2y ago
  Code quality does not affect final product quality IMHO. I worked in companies with terrible code, that deployed on an over-engineered cloud provider using custom containers hacked with a nail and a screwdriver, but the product was excellent. Had bugs here and there, but worked and delivered what needs to be delivered. SWEs need to realize that code doesn't really matter. For 70 years we are debating the best architecture patterns and yet the biggest fear of every developer is working on legacy code, as it's an unmaintainable piece of ... written by humans.
  - gspencley 2y ago
    > Code quality does not affect final product quality IMHO. What we need, admittedly, is more research and study around this. I know of one study which supports my position, but I'm happy to admit that the data is sparse. https://arxiv.org/abs/2203.04374 https://arxiv.org/abs/2203.04374 From the abstract: "By analyzing activity in 30,737 files, we find that low quality code contains 15 times more defects than high quality code."

rybosworld 2y ago

> That is useful, and the result can be shockingly impressive depending on what you're trying to do. But the limitations are so limited that the prospect of replacing an entire high-skilled profession with that magic trick is kind of a joke. The possible outcome space is not binary (at least in the near term), i.e. either AI replace devs, or it doesn't. What I'm getting at is this: There's a pervasive attitude among some developers (generally older developers, in my experience) that LLM's are effectively useless. If we're being objective, that is quite plainly not true. These conversations tend to start out with something like: "Well _my_ work in particular is so complex that LLM's couldn't possibly assist." As the conversation grows, the tone gradually changes to admitting: "Yes there are some portions of a codebase where LLM's can be helpful, but they can't do _everything_ that an experienced dev does." It should not even be controversial to say, that AI will only improve at this task. That's what technology does, over the long run. Fundamentally, there's ego involved whenever someone says "LLM's have _never_ produced useable code." That statement, is provably false.

ben_w 2y ago

> Consider this, 100% of AI training data is human-generated content. Probably doesn't change the conclusion, but this is no longer the case. I've yet to try Phi-4, but the whole series has synthetic training data; I don't value Phi-3 despite what the benchmarks claim.