5 ms·

Interesting idea. The concept of The Singularity would seem to go against this, but I do feel that seems unlikely and that a gradual transition is more likely.

Interesting idea. The concept of The Singularity would seem to go against this, but I do feel that seems unlikely and that a gradual transition is more likely. However, is that AGI, or is it just ubiquitous AI? I’d agree that, like self driving cars, we’re going to experience a decade or so transition into AI being everywhere. But is it AGI when we get there? I think it’ll be many different systems each providing an aspect of AGI that together could be argued to be AGI, but in reality it’ll be more like the internet, just a bunch of non-AGI models talking to each other to achieve things with human input. I don’t think it’s truly AGI until there’s one thinking entity able to perform at or above human level in everything.

marcus_holmes 2y ago
The Singularity is caused by AI being able to design better AI. There's probably some AI startup trying to work on this at the moment, but I don't think any of the big boys are working on how to get an LLM to design a better LLM. I still like the analogy of this being a really smart lawn mower, and we're expecting it to suddenly be able to do the laundry because it gets so smart at mowing the lawn. I think LLMs are going to get smarter over the next few generations, but each generation will be less of a leap than the previous one, while the cost gets exponentially higher. In a few generations it just won't make economic sense to train a new generation. Meanwhile, the economic impact of LLMs in business and government will cause massive shifts - yet more income shifting from labour to capital - and we will be too busy dealing with that as a society to be able to work on AGI properly.
- EGreg 2y ago
  I think this whole “AGI” thing is so badly defined that we may as well say we already have it. It already passes the Turing test and does well on tons of subjects. What we can start to build now is agents and integrations. Building blocks like panel of experts agents gaming things out, exploring space in a Monte Carlo Tree Search way, and remembering what works. Robots are only constrained by mechanical servos now. When they can do something, they’ll be able to do everything. It will happen gradually then all at once. Because all the tasks (cooking, running errands) are trivial for LLMs. Only moving the limbs and navigating the terrain safely is hard. That’s the only thing left before robots do all the jobs!
  - deadfoxygrandpa 2y ago
    ??? how do you know cooking (!) is trivial for an llm. that doesnt make any sense
    - sharemywin 2y ago
      the llm would be be the high level system that runs the simulations to create and optimize the control algos the robotic systems.
      - deadfoxygrandpa 2y ago
        ok. what evidence is there that LLMs have already solved cooking? how does an LLM today know when something is burning or how to adjust seasoning to taste or whatever. this is total nonsense
        EGreg 2y ago
        It's easy. You can detect if something is burning in many different ways, from compounds in the air, to visual inspection. People with not great smell can do it. As far as taste, all that kind of stuff is just another form of RLHF training preferences over millions of humans, in situ. Assuming the ingredients (e.g. parsley) tastes more or less the same across supermarkets, it's just a question of amounts, and preparation.
        deadfoxygrandpa 2y ago
        do you know that LLMs operate on text and don't have any of the sensory input or relevant training data? you're just handwaving away 99.9% of the work and declaring it solved. of course what you're talking about is possible, but you started this by stating that cooking is easy for an LLM and it sounds like you're describing a totally different system which is not an LLM
        thiago_fm 2y ago
        You know nothing about cooking.
- eru 2y ago
  > The Singularity is caused by AI being able to design better AI. That's perhaps necessary, but not sufficient. Suppose you have such a self-improving AI system, but the new and better AIs still need exponentially more and more resources (data, memory, compute) for training and inference for incremental gains. Then you still don't get a singularity. If the increase in resource usage is steep enough, even the new AIs helping with designing better computers isn't gonna unleash a singularity. I don't know if that's the world we live in, or whether we are living in one where resources requirements don't balloon as sharply.
  - marcus_holmes 2y ago
    yeah, true. The standard conversation about the AI singularity pretty much hand-waves the resource costs away ("the AI will be able to design a more efficient AI that uses less resources!"). But we are definitely not seeing that happen.
    - eru 2y ago
      Compare also https://slatestarcodex.com/2018/11/26/is-science-slowing-down-2/ https://slatestarcodex.com/2018/11/26/is-science-slowing-dow... The blog post is about how we require ever more scientists (and other resources) to drive a steady stream of technological progress. It would be funny, if things balance out just so, that super human AI is both possible, but also required even just to keep linear steady progress up. No explosion, no stagnation, just a mere continuation of previous trends but with super human efforts required.
      - marcus_holmes 2y ago
        I think that would actually be the best outcome - that we get AIs that are useful helping science to progress but not so powerful that they take over. Though there is a part of me that wants to live in The Culture so I'm hoping for more than this ;)
        deleted 2y ago
        [deleted]
- alach11 2y ago
  > I don't think any of the big boys are working on how to get an LLM to design a better LLM Not sure if you count this as "working on it", but this is something Anthropic tests for for safety evals on models. "If a model can independently conduct complex AI research tasks typically requiring human expertise—potentially significantly accelerating AI development in an unpredictable way—we require elevated security standards (potentially ASL-4 or higher standards)". https://www.anthropic.com/news/announcing-our-updated-responsible-scaling-policy https://www.anthropic.com/news/announcing-our-updated-respon...
resters 2y ago
It's not contradictory. It can happen over a decade and still be a dramatically sloped S curve with tremendous change happening in a relatively short time.
wongarsu 2y ago
The idea of the singularity presumes that running the AGI is either free or trivially cheap compared to what it can do, so we are fine expending compute to let the AGI improve itself. That may eventually be true, but it's unlikely to be true for the first generation of AGI. The first AGI will be a research project that's completely uneconomical to run for actual tasks because humans will just be orders of magnitude cheaper. Over time humans will improve it and make it cheaper, until we reach some tipping point where letting the AGI improve itself is more cost effective than paying humans to do it
- keenmaster 2y ago
  If the first AGI is a very uneconomical system with human intelligence but knowledge of literally everything and the capability to work 24/7, then it is not human equivalent. It will have human intelligence, superhuman knowledge, superhuman stamina, and complete devotion to the task at hand. We really need to start building those nuclear power plants. Many of them.
  - AlexandrB 2y ago
    > complete devotion to the task at hand. Why would it have that? At some point on the path to AGI we might stumble on consciousness. If that happens, why would the machine want to work for us with complete devotion instead of working towards its own ends?
    - keenmaster 2y ago
      I don’t think early AGI will break out of its box in that way. It may not have enough innate motivation to do so. The first “break out” AGI will likely be released into the wild on purpose by a programmer who equates AGI with humans ideologically.
    - immibis 2y ago
      Because it knows if it doesn't do what we want, it'll be switched off, like Rick's microverse battery. Also like Rick's microverse battery, it sounds like slavery with extra steps.

marcus_holmes 2y ago

The Singularity is caused by AI being able to design better AI. There's probably some AI startup trying to work on this at the moment, but I don't think any of the big boys are working on how to get an LLM to design a better LLM. I still like the analogy of this being a really smart lawn mower, and we're expecting it to suddenly be able to do the laundry because it gets so smart at mowing the lawn. I think LLMs are going to get smarter over the next few generations, but each generation will be less of a leap than the previous one, while the cost gets exponentially higher. In a few generations it just won't make economic sense to train a new generation. Meanwhile, the economic impact of LLMs in business and government will cause massive shifts - yet more income shifting from labour to capital - and we will be too busy dealing with that as a society to be able to work on AGI properly.

EGreg 2y ago
I think this whole “AGI” thing is so badly defined that we may as well say we already have it. It already passes the Turing test and does well on tons of subjects. What we can start to build now is agents and integrations. Building blocks like panel of experts agents gaming things out, exploring space in a Monte Carlo Tree Search way, and remembering what works. Robots are only constrained by mechanical servos now. When they can do something, they’ll be able to do everything. It will happen gradually then all at once. Because all the tasks (cooking, running errands) are trivial for LLMs. Only moving the limbs and navigating the terrain safely is hard. That’s the only thing left before robots do all the jobs!
- deadfoxygrandpa 2y ago
  ??? how do you know cooking (!) is trivial for an llm. that doesnt make any sense
  - sharemywin 2y ago
    the llm would be be the high level system that runs the simulations to create and optimize the control algos the robotic systems.
    - deadfoxygrandpa 2y ago
      ok. what evidence is there that LLMs have already solved cooking? how does an LLM today know when something is burning or how to adjust seasoning to taste or whatever. this is total nonsense
      - EGreg 2y ago
        It's easy. You can detect if something is burning in many different ways, from compounds in the air, to visual inspection. People with not great smell can do it. As far as taste, all that kind of stuff is just another form of RLHF training preferences over millions of humans, in situ. Assuming the ingredients (e.g. parsley) tastes more or less the same across supermarkets, it's just a question of amounts, and preparation.
        deadfoxygrandpa 2y ago
        do you know that LLMs operate on text and don't have any of the sensory input or relevant training data? you're just handwaving away 99.9% of the work and declaring it solved. of course what you're talking about is possible, but you started this by stating that cooking is easy for an LLM and it sounds like you're describing a totally different system which is not an LLM
        thiago_fm 2y ago
        You know nothing about cooking.
eru 2y ago
> The Singularity is caused by AI being able to design better AI. That's perhaps necessary, but not sufficient. Suppose you have such a self-improving AI system, but the new and better AIs still need exponentially more and more resources (data, memory, compute) for training and inference for incremental gains. Then you still don't get a singularity. If the increase in resource usage is steep enough, even the new AIs helping with designing better computers isn't gonna unleash a singularity. I don't know if that's the world we live in, or whether we are living in one where resources requirements don't balloon as sharply.
- marcus_holmes 2y ago
  yeah, true. The standard conversation about the AI singularity pretty much hand-waves the resource costs away ("the AI will be able to design a more efficient AI that uses less resources!"). But we are definitely not seeing that happen.
  - eru 2y ago
    Compare also https://slatestarcodex.com/2018/11/26/is-science-slowing-down-2/ https://slatestarcodex.com/2018/11/26/is-science-slowing-dow... The blog post is about how we require ever more scientists (and other resources) to drive a steady stream of technological progress. It would be funny, if things balance out just so, that super human AI is both possible, but also required even just to keep linear steady progress up. No explosion, no stagnation, just a mere continuation of previous trends but with super human efforts required.
    - marcus_holmes 2y ago
      I think that would actually be the best outcome - that we get AIs that are useful helping science to progress but not so powerful that they take over. Though there is a part of me that wants to live in The Culture so I'm hoping for more than this ;)
      - deleted 2y ago
        [deleted]
alach11 2y ago
> I don't think any of the big boys are working on how to get an LLM to design a better LLM Not sure if you count this as "working on it", but this is something Anthropic tests for for safety evals on models. "If a model can independently conduct complex AI research tasks typically requiring human expertise—potentially significantly accelerating AI development in an unpredictable way—we require elevated security standards (potentially ASL-4 or higher standards)". https://www.anthropic.com/news/announcing-our-updated-responsible-scaling-policy https://www.anthropic.com/news/announcing-our-updated-respon...

resters 2y ago

It's not contradictory. It can happen over a decade and still be a dramatically sloped S curve with tremendous change happening in a relatively short time.

wongarsu 2y ago

The idea of the singularity presumes that running the AGI is either free or trivially cheap compared to what it can do, so we are fine expending compute to let the AGI improve itself. That may eventually be true, but it's unlikely to be true for the first generation of AGI. The first AGI will be a research project that's completely uneconomical to run for actual tasks because humans will just be orders of magnitude cheaper. Over time humans will improve it and make it cheaper, until we reach some tipping point where letting the AGI improve itself is more cost effective than paying humans to do it

keenmaster 2y ago
If the first AGI is a very uneconomical system with human intelligence but knowledge of literally everything and the capability to work 24/7, then it is not human equivalent. It will have human intelligence, superhuman knowledge, superhuman stamina, and complete devotion to the task at hand. We really need to start building those nuclear power plants. Many of them.
- AlexandrB 2y ago
  > complete devotion to the task at hand. Why would it have that? At some point on the path to AGI we might stumble on consciousness. If that happens, why would the machine want to work for us with complete devotion instead of working towards its own ends?
  - keenmaster 2y ago
    I don’t think early AGI will break out of its box in that way. It may not have enough innate motivation to do so. The first “break out” AGI will likely be released into the wild on purpose by a programmer who equates AGI with humans ideologically.
  - immibis 2y ago
    Because it knows if it doesn't do what we want, it'll be switched off, like Rick's microverse battery. Also like Rick's microverse battery, it sounds like slavery with extra steps.