4 ms·

Nah. There are corners of the industry where people still write ASM by hand when necessary, but for the vast, vast majority it's neither necessary (because com

Nah. There are corners of the industry where people still write ASM by hand when necessary, but for the vast, vast majority it's neither necessary (because compilers are great) or worthwhile (because it's so time consuming). Most code is written in high-level, interpreted languages with no particular attention paid to its performance characteristics. Despite the frustration of those of us who know better, businesses and users seem to choose velocity over quality pretty consistently. LLM output is already good enough to produce working software that meets the stated requirements. The tooling used to work with them is improving rapidly. I think we're heading towards a world where actually inspecting and understanding the code is unusual (like looking at JVM/Python bytecode is today). Future liabilities? Not any more than we're currently producing, but produced faster.

sarchertech 8mo ago
Compilers take a formal language and translate it to another formal language. In most cases there is no ambiguity, it’s deterministic, and most importantly it’s not chaotic. That is changing one word in the source code doesn’t tend to produce a vastly different output, or changes to completely unrelated code. Because the LLM is working from informal language, it is by necessity making thousands of small (and not so small) decisions about how to translate the prompt into code. There are far more decisions here than can reasonably fixed in tests/specs. So any changes to the prompt/spec is likely to result in unintended changes to observable behavior that users will notice and be confused by. You’re right that programmers regularly churn out unoptimized code. But that’s very different than churning out a bubbling morass where ever little thing that isn’t bolted down is constantly changing. The ambiguity in translation from prompt to code means that the code is still the spec and needs to be understood. Combine that with prompt instability and we’ll be stuck understanding code for the foreseeable future.
- ytoawwhra92 8mo ago
  The problem you describe is real, but I think it can be addressed by improving tooling without any improvement in available LLM technology.
  - paulryanrogers 8mo ago
    How? Are you thinking of adversarial AI reviewers, runtime tests (also by AI), or something else? Guess I just don't see how you can take the human out of the loop and replace them with non-deterministic AIs and informal prompts / specs.
    - gilfaethwy 8mo ago
      Humans are also non-deterministic, though. Why does replacing one non-deterministic actor with another matter here? I'm not particularly swayed by arguments of consciousness, whether AI is currently capable of "thinking", etc. Those may matter right now... but how long will they continue to matter for the vast majority of use cases? Generally speaking, my feeling is that most code doesn't need to be carefully-crafted. We have error budgets for a reason, and AI is just shifting how we allocate them. It's only in certain roles where small mistakes can end your company - think hedge funds, aerospace, etc. - where there's safety in the non-determinism argument. And I say this as someone who is not in one of those roles. I don't think my job is safe for more than a couple of years at this point.
      - sarchertech 8mo ago
        It has nothing to do with whether small mistakes are allowable or not. It’s about customers needing a consistent product. The in-code tests and the expectations/assumptions about the product that your users have are wildly different. If you allow agents to make changes restricted only by those tests, they’re going to constantly make changes that break customer workflows and cause noticeable jank. Right now agents do this at a rate far higher than humans. This is empirically demonstrable by the fact that an agent requires tests to keep from spinning out of control when writing more than a few thousand lines and a human does not. A human is capable of writing tens of thousands as of lines with no tests, using only reason and judgement. An agent is not. They clearly lack the full capability of human reason, judgment, taste, and agency. My suspicion is that something close enough to AGI that it can essentially do all white dollar jobs is required to solve this.
      - the_af 8mo ago
alpaca128 8mo ago
"users seem to choose velocity over quality pretty consistently" When do they have a real choice, without vendor lock-in or other pressure? Windows 11 is 4 years old but until a few months ago barely managed to overtake Windows 10. Despite upgrades that were only "by choice" in the most user hostile sense imaginable (those dark patterns were so misleading I know multiple people who didn't notice that they "agreed" to it, and as it pop ups repeatedly it only takes a single wrong click to mess up). It doesn't look like people are very excited about the "velocity". In the gaming industry AAA titles being thrown on the market in an unfinished state tends to also not go over well with the users, but there they have more power to make a choice as the market is huge and games aren't necessary tools, and such games rarely recover after a failed launch.
deleted 8mo ago
[deleted]

sarchertech 8mo ago

Compilers take a formal language and translate it to another formal language. In most cases there is no ambiguity, it’s deterministic, and most importantly it’s not chaotic. That is changing one word in the source code doesn’t tend to produce a vastly different output, or changes to completely unrelated code. Because the LLM is working from informal language, it is by necessity making thousands of small (and not so small) decisions about how to translate the prompt into code. There are far more decisions here than can reasonably fixed in tests/specs. So any changes to the prompt/spec is likely to result in unintended changes to observable behavior that users will notice and be confused by. You’re right that programmers regularly churn out unoptimized code. But that’s very different than churning out a bubbling morass where ever little thing that isn’t bolted down is constantly changing. The ambiguity in translation from prompt to code means that the code is still the spec and needs to be understood. Combine that with prompt instability and we’ll be stuck understanding code for the foreseeable future.

ytoawwhra92 8mo ago
The problem you describe is real, but I think it can be addressed by improving tooling without any improvement in available LLM technology.
- paulryanrogers 8mo ago
  How? Are you thinking of adversarial AI reviewers, runtime tests (also by AI), or something else? Guess I just don't see how you can take the human out of the loop and replace them with non-deterministic AIs and informal prompts / specs.
  - gilfaethwy 8mo ago
    Humans are also non-deterministic, though. Why does replacing one non-deterministic actor with another matter here? I'm not particularly swayed by arguments of consciousness, whether AI is currently capable of "thinking", etc. Those may matter right now... but how long will they continue to matter for the vast majority of use cases? Generally speaking, my feeling is that most code doesn't need to be carefully-crafted. We have error budgets for a reason, and AI is just shifting how we allocate them. It's only in certain roles where small mistakes can end your company - think hedge funds, aerospace, etc. - where there's safety in the non-determinism argument. And I say this as someone who is not in one of those roles. I don't think my job is safe for more than a couple of years at this point.
    - sarchertech 8mo ago
      It has nothing to do with whether small mistakes are allowable or not. It’s about customers needing a consistent product. The in-code tests and the expectations/assumptions about the product that your users have are wildly different. If you allow agents to make changes restricted only by those tests, they’re going to constantly make changes that break customer workflows and cause noticeable jank. Right now agents do this at a rate far higher than humans. This is empirically demonstrable by the fact that an agent requires tests to keep from spinning out of control when writing more than a few thousand lines and a human does not. A human is capable of writing tens of thousands as of lines with no tests, using only reason and judgement. An agent is not. They clearly lack the full capability of human reason, judgment, taste, and agency. My suspicion is that something close enough to AGI that it can essentially do all white dollar jobs is required to solve this.
    - the_af 8mo ago

alpaca128 8mo ago

"users seem to choose velocity over quality pretty consistently" When do they have a real choice, without vendor lock-in or other pressure? Windows 11 is 4 years old but until a few months ago barely managed to overtake Windows 10. Despite upgrades that were only "by choice" in the most user hostile sense imaginable (those dark patterns were so misleading I know multiple people who didn't notice that they "agreed" to it, and as it pop ups repeatedly it only takes a single wrong click to mess up). It doesn't look like people are very excited about the "velocity". In the gaming industry AAA titles being thrown on the market in an unfinished state tends to also not go over well with the users, but there they have more power to make a choice as the market is huge and games aren't necessary tools, and such games rarely recover after a failed launch.

deleted 8mo ago

[deleted]