4 ms·

> The reason why true singularity-level ASI will never happen isn't because the technology cannot support it (though, to be clear: It can't). Its because no one

by mlyle 1y ago

> The reason why true singularity-level ASI will never happen isn't because the technology cannot support it (though, to be clear: It can't). Its because no one actually wants it. Plenty of people want something capable of doing tasks well beyond what GPT-5 can and that are equally capable to a proficient human. If you can do that cheaper or faster or more available than said skilled human, there is definitely a market for it.

esalman 1y ago
Nobody will pay them 1.5m bonus for developing ASI though. As much as we hate on meta, open models are the answer.
827a 1y ago
The issue is, what you're describing actually doesn't require intelligence, beyond a point. A mathematician who can solve the Reimann hypothesis isn't likely to be all that good of an L8 AWS Engineer; but what AWS would pay a lot of money for is a digital L8 AWS Engineer, not a digital Terrance Tao. The deep reflection that AI frontier labs have to do, soon, is: If its not intellect, what are we missing?
- Avicebron 1y ago
  embodiment?
- klipt 1y ago
  I'm sure plenty of cancer patients would pay good money for a digital genius doctor who can custom design a drug to target their cancer.
  - 827a 1y ago
    Again, this doesn't require intelligence, beyond a point. Breakthrough medical therapies aren't invented by 180 IQ Einsteinian hyper-intellectuals. My point in being pedantic about this is to just point out that an extreme amount of value could be generated by these systems if they only attained the capability to do the things 110 IQ humans already do today. Instead of optimizing toward that goal, the frontier labs seem obsessed with optimizing toward some other weird goal of hyper-intelligence that has very unclear marketability. When pressed on this, leaders in these companies will say "its transferable"; that hyper-intellectuality will suddenly unlock realistic 110 IQ use-cases; but we're increasingly seeing, quite clearly to me, that this isn't the case. And there's very good first principal reasons why you should believe that it won't be.
    - Gud 1y ago
      No, because Einstein-level intelligence is rare. But 180 IQ intelligence that doesn’t sleep doesn’t stop with instant access to the worlds knowledge will absolutely be able to revolutionise the sciences.
      - mlyle 1y ago
        So, I'm not sure 180 IQ intelligence can. But something that has equivalence to high human intelligence (say 140IQ, generalized to a wide variety of domains) and can engage in long-term directed, purposeful, and autonomous behavior will (not all humans of high intelligence cross this bar, but the ones that do tend to make an impact).
    - mlyle 1y ago
      I would argue that AI is able to do a lot of what a 110 IQ human does today already-- at least the 30% of the population that is of average intelligence and has a below average appetite for growth and challenge. And sure, it's not perfectly transferable. It's going to be a long time before it can fix my plumbing, but there's a fair number of tasks that it can do now that are useful. > toward some other weird goal of hyper-intelligence I think the near term goal is something that can do most human tasks of the scope of tens of minutes of work at a computer, without task selection or self-direction. In some ways this looks superhuman, doing this stuff at high speed and 24/7. But it's superhuman like a mechanical loom is superhuman. Even better would be something you could delegate higher level goals, including tasks of larger scope and duration, but that's slower coming.
- danaris 1y ago
  > If its not intellect, what are we missing? ...Why would it not be intellect? What we're missing is anything that actually resembles thought at anything more than the very surface level. LLMs are not intelligent. They sound intelligent because they're being trained very well at predicting what tokens to produce to sound that way, but they have no intellect, no understanding, no world model, no consciousness—no intelligence.
  - seba_dos1 1y ago
    I love how in the world of LLMs "reasoning" means having the model write out things like "I'm collecting a list of resources" just so it can then better cosplay someone who did that. I'm eagerly waiting for the next big breakthrough when someone trains a model that switches between "thinking" and "outputting" modes mid-answer :)
    - mlyle 1y ago
      > I'm eagerly waiting for the next big breakthrough when someone trains a model that switches between "thinking" and "outputting" modes mid-answer :) ??? That was last year's breakthrough.
      - seba_dos1 1y ago
        See??? What a breakneck pace! :D
        mlyle 1y ago
        So, I guess it was just a low effort comment, and I shouldn't take you seriously? I thought you were making an actual effort to converse. Generating tokens to represent steps isn't just "cosplaying" --- it's representing more state than can fit into a single step of the model. Ditto for when I talk to myself when figuring something out, either out loud or just mentally.
        seba_dos1 1y ago

by mlyle 1y ago

esalman 1y ago

Nobody will pay them 1.5m bonus for developing ASI though. As much as we hate on meta, open models are the answer.

827a 1y ago

The issue is, what you're describing actually doesn't require intelligence, beyond a point. A mathematician who can solve the Reimann hypothesis isn't likely to be all that good of an L8 AWS Engineer; but what AWS would pay a lot of money for is a digital L8 AWS Engineer, not a digital Terrance Tao. The deep reflection that AI frontier labs have to do, soon, is: If its not intellect, what are we missing?

Avicebron 1y ago
embodiment?
klipt 1y ago
I'm sure plenty of cancer patients would pay good money for a digital genius doctor who can custom design a drug to target their cancer.
- 827a 1y ago
  Again, this doesn't require intelligence, beyond a point. Breakthrough medical therapies aren't invented by 180 IQ Einsteinian hyper-intellectuals. My point in being pedantic about this is to just point out that an extreme amount of value could be generated by these systems if they only attained the capability to do the things 110 IQ humans already do today. Instead of optimizing toward that goal, the frontier labs seem obsessed with optimizing toward some other weird goal of hyper-intelligence that has very unclear marketability. When pressed on this, leaders in these companies will say "its transferable"; that hyper-intellectuality will suddenly unlock realistic 110 IQ use-cases; but we're increasingly seeing, quite clearly to me, that this isn't the case. And there's very good first principal reasons why you should believe that it won't be.
  - Gud 1y ago
    No, because Einstein-level intelligence is rare. But 180 IQ intelligence that doesn’t sleep doesn’t stop with instant access to the worlds knowledge will absolutely be able to revolutionise the sciences.
    - mlyle 1y ago
      So, I'm not sure 180 IQ intelligence can. But something that has equivalence to high human intelligence (say 140IQ, generalized to a wide variety of domains) and can engage in long-term directed, purposeful, and autonomous behavior will (not all humans of high intelligence cross this bar, but the ones that do tend to make an impact).
  - mlyle 1y ago
    I would argue that AI is able to do a lot of what a 110 IQ human does today already-- at least the 30% of the population that is of average intelligence and has a below average appetite for growth and challenge. And sure, it's not perfectly transferable. It's going to be a long time before it can fix my plumbing, but there's a fair number of tasks that it can do now that are useful. > toward some other weird goal of hyper-intelligence I think the near term goal is something that can do most human tasks of the scope of tens of minutes of work at a computer, without task selection or self-direction. In some ways this looks superhuman, doing this stuff at high speed and 24/7. But it's superhuman like a mechanical loom is superhuman. Even better would be something you could delegate higher level goals, including tasks of larger scope and duration, but that's slower coming.
danaris 1y ago
> If its not intellect, what are we missing? ...Why would it not be intellect? What we're missing is anything that actually resembles thought at anything more than the very surface level. LLMs are not intelligent. They sound intelligent because they're being trained very well at predicting what tokens to produce to sound that way, but they have no intellect, no understanding, no world model, no consciousness—no intelligence.
- seba_dos1 1y ago
  I love how in the world of LLMs "reasoning" means having the model write out things like "I'm collecting a list of resources" just so it can then better cosplay someone who did that. I'm eagerly waiting for the next big breakthrough when someone trains a model that switches between "thinking" and "outputting" modes mid-answer :)
  - mlyle 1y ago
    > I'm eagerly waiting for the next big breakthrough when someone trains a model that switches between "thinking" and "outputting" modes mid-answer :) ??? That was last year's breakthrough.
    - seba_dos1 1y ago
      See??? What a breakneck pace! :D
      - mlyle 1y ago
        So, I guess it was just a low effort comment, and I shouldn't take you seriously? I thought you were making an actual effort to converse. Generating tokens to represent steps isn't just "cosplaying" --- it's representing more state than can fit into a single step of the model. Ditto for when I talk to myself when figuring something out, either out loud or just mentally.
        seba_dos1 1y ago