3 ms·

> software errors often cascade Whereas software defects in design and architecture subtly accumulate, until they leave the codebase in a state in which it bec

> software errors often cascade Whereas software defects in design and architecture subtly accumulate, until they leave the codebase in a state in which it becomes utterly unworkable. It is one of the chief reasons why good devs get paid what they do. Software discussions very often underrate software extensibility, or in other words, its structural and architectural scaleability. Even software correctness is trivial in comparison - you can't even keep writing correct code if you've made an unworkable tire-fire. This could be a massive mountain for AI to climb.

hawk_ 2y ago
I hear you. But I have wondered if there won't be a need to maintain certain like of software when you can just have it be rewritten for each iteration. Like some kind of schema evolution, yes but throwaway software at each iteration.
- aerhardt 2y ago
  Well in terms of processing speed the AI could iterate on different designs until it finds an extensible one, with some kind of reinforcement learning loop. Produce a certain design, get stuck, throw it away, try a new one. Just like humans learn to write good code really - except at an unfathomable speed of iteration. But it still all sounds ridiculously challenging. There is something there that isn't about predicting next tokens like LLMs do. It's about inferring very complex, highly abstract metastructures in the text.
- bamboozled 2y ago
  The challenge might be around the edges here, I guess you'd be able to instruct an agent to always code to a certain API spec, but no piece of software runs or does anything really useful in vacuum.
vl 2y ago
Fundamentally there is human with limited brain capacity that got trained to that. It’s just a question of time when there are equally capable, and then exceedingly capable models. There is nothing magical or special about human brain. The only question is how fast it is going to happen. Ie what percentage of jobs is going to be replaced next year and so on.
- aerhardt 2y ago
  > There is nothing magical or special about human brain. There is a lot about the human brain that even the world's top neuroscientists don't know. There's plenty of magic about it if we define magic as undiscovered knowledge. There's also no consensus among top AI researchers that current techniques like LLMs will get us anywhere close to AGI. Nothing I've seen on current models (not even o1-preview) suggests to me that AIs can reason about codebases of more than 5k LOC. A top 5% engineer can probably make sense of a codebase of a couple million LOC in time. Which models specifically have you seen that are looking like they will be able to surmount any time soon the challenges of software design and architecture I'm laying out in my previous comment?
  - snowe2010 2y ago
    Defining AGI as “can reason about 5MLOC” is ridiculous. When do the goal posts stop moving? When a computer can solve time travel? Babies have behavior all the time that is no more differentiable from what an LLM does on a normal basis (including terrible logic and hallucinations). The majority of people on the planet can barely reason about how any given politician will affect them, even when there’s a billion resources out there telling them exactly that. No reasonable human would ever define AGI as having anything to do with coding at all, since that’s not even “general intelligence”… it’s learned facts and logic.
    - weweweoo 2y ago
      Babies can at least manipulate the physical world. Large language model can never be defined as AGI until it can control a general purpose robot, similar to how human brain controls our body's motor functions.
      - snowe2010 2y ago
        You’re commenting that on an article about how Claude literally can do what you’re talking about.
    - throwup238 2y ago
nopinsight 2y ago
Current LLMs lack the ability to perform abstraction at the right level a problem requires. When this gets solved, we’d be quite a bit closer to AGI, which has implications far beyond job displacement. ARC-AGI Benchmark might serve as a canary in the coal mine. https://github.com/fchollet/ARC-AGI https://github.com/fchollet/ARC-AGI

hawk_ 2y ago

I hear you. But I have wondered if there won't be a need to maintain certain like of software when you can just have it be rewritten for each iteration. Like some kind of schema evolution, yes but throwaway software at each iteration.

aerhardt 2y ago
Well in terms of processing speed the AI could iterate on different designs until it finds an extensible one, with some kind of reinforcement learning loop. Produce a certain design, get stuck, throw it away, try a new one. Just like humans learn to write good code really - except at an unfathomable speed of iteration. But it still all sounds ridiculously challenging. There is something there that isn't about predicting next tokens like LLMs do. It's about inferring very complex, highly abstract metastructures in the text.
bamboozled 2y ago
The challenge might be around the edges here, I guess you'd be able to instruct an agent to always code to a certain API spec, but no piece of software runs or does anything really useful in vacuum.

vl 2y ago

Fundamentally there is human with limited brain capacity that got trained to that. It’s just a question of time when there are equally capable, and then exceedingly capable models. There is nothing magical or special about human brain. The only question is how fast it is going to happen. Ie what percentage of jobs is going to be replaced next year and so on.

aerhardt 2y ago
> There is nothing magical or special about human brain. There is a lot about the human brain that even the world's top neuroscientists don't know. There's plenty of magic about it if we define magic as undiscovered knowledge. There's also no consensus among top AI researchers that current techniques like LLMs will get us anywhere close to AGI. Nothing I've seen on current models (not even o1-preview) suggests to me that AIs can reason about codebases of more than 5k LOC. A top 5% engineer can probably make sense of a codebase of a couple million LOC in time. Which models specifically have you seen that are looking like they will be able to surmount any time soon the challenges of software design and architecture I'm laying out in my previous comment?
- snowe2010 2y ago
  Defining AGI as “can reason about 5MLOC” is ridiculous. When do the goal posts stop moving? When a computer can solve time travel? Babies have behavior all the time that is no more differentiable from what an LLM does on a normal basis (including terrible logic and hallucinations). The majority of people on the planet can barely reason about how any given politician will affect them, even when there’s a billion resources out there telling them exactly that. No reasonable human would ever define AGI as having anything to do with coding at all, since that’s not even “general intelligence”… it’s learned facts and logic.
  - weweweoo 2y ago
    Babies can at least manipulate the physical world. Large language model can never be defined as AGI until it can control a general purpose robot, similar to how human brain controls our body's motor functions.
    - snowe2010 2y ago
      You’re commenting that on an article about how Claude literally can do what you’re talking about.
  - throwup238 2y ago

nopinsight 2y ago

Current LLMs lack the ability to perform abstraction at the right level a problem requires. When this gets solved, we’d be quite a bit closer to AGI, which has implications far beyond job displacement. ARC-AGI Benchmark might serve as a canary in the coal mine. https://github.com/fchollet/ARC-AGI https://github.com/fchollet/ARC-AGI