4 ms·

The reason AGI would create a singularity is because of its ability to self learn. Presently we are still a long way from that. In my opinion we at least are a

The reason AGI would create a singularity is because of its ability to self learn. Presently we are still a long way from that. In my opinion we at least are as far away from AGI as 1970s mainframes were from LLMs. I really don’t expect to see AGI in my lifetime.

hollownobody 1y ago
This reminds me of: https://en.m.wikipedia.org/wiki/Flying_Machines_Which_Do_Not_Fly https://en.m.wikipedia.org/wiki/Flying_Machines_Which_Do_Not...
- hnlmorg 1y ago
  For every example where someone over predicted the time it would take for a breakthrough, there are at least 10 examples of people being too optimistic with their predictions. And with AGI, you also have the likes of Sam Altman making up bullshit claims just to pump up the investment into OpenAI. So I wouldn’t take much of their claims seriously either. LLMs are a fantastic invention. But they’re far closer to SMS text predict than they are to generalised intelligence. Though what you might see is OpenAI et al redefine the term “AGI” just so they can say they’ve hit that milestone, again purely for their own financial gain.
  - Davidzheng 1y ago
    in the history of AI usually people overestimate how long a capability is reached. There are very few counterexamples to this (GPT5 capability level might be one of them though)
  - hollownobody 1y ago
    Are there any predictions you'd want to make? Not about AGI, but about an intermediate goalpost you think we won't reach in the next 5 years
- zamadatix 1y ago
  This reminds me how, a few years after the first fission power plant, Teller, Bhaba, and other nuclear physicists of the 1950s were convinced fusion power plants were about as far away as the physicists of today still predict they are. I'm cautiously optimistic of each technology, but the point is it's easy to find bullshit predictions without actually gaining any insight into what will happen with a given technology.
adastra22 1y ago
That is already happening. These labs are writing next gen models using next gen models, with greater levels of autonomy. That doesn’t get the hard takeoff people talk about because those hypotheticals don’t consider sources of error, noise, and drift.
- hnlmorg 1y ago
  They’re using lossy models to feedback into the training and research of new lossy models. But none of it is AGI self learning. You need both the generalised part of AGI and the ability to self learn. One without the other wouldn’t cause a singularity.
  - adastra22 1y ago
    They are doing self-learning things. That’s what a lot of synthetic data is about. When managed by the AI, it is an AI picking what it want to train on in order to develop new capabilities. (Artificial General Intelligence says nothing about self-learning though. I presume you mean ASI?)
- theobreuerweil 1y ago
  The models may be writing the code but I would be surprised if they were contributing to the underlying science, which feels like the hard part
  - Davidzheng 1y ago
    it's hardly science it's mostly experimentation + ablations on new ideas. but yeah idk if they are asking llms to generate these ideas. probably not good enough as is. though it doesn't seem outo f reach to RL on generating ideas for AI research
    - yberreby 1y ago
      I'm curious what you think qualifies as science.
Muromec 1y ago
We have an ability to self learn right now, but we stil suck at basics
- jackomelon 1y ago
  meatspace constraints!
- hnlmorg 1y ago
  There’s a lot of other variables at play for humans. Like - the need to sleep for 1/3 of our life - the need to eat, causing more pauses in work - much slower (like several orders of magnitude slower) data input capabilities - lossy storage (aka forgetfulness) - emotions - other primal urges, like the need to procreate
- 8n4vidtmkvmk 1y ago
  Imagine never forgetting, and never getting bored or tired. I think we could achieve a lot more.
russellbeattie 1y ago
This is the key - right now each new model has had countless resources dedicated to training, then they are more or less set in stone until the next update. These big models don't dynamically update as days pass by - they don't learn. A personal assistant service may be able to mimic learning by creating a database of your data or preferences, but your usage isn't baked back into the big underlying model permanently. I don't agree with "in our lifetimes", but the difference between training and learning is the bright red line. Until there's a model which is able to continually update itself, it's not AGI. My guess is that this will require both more powerful hardware and a few more software innovations. But it'll happen.
layer8 1y ago
The ability to self-learn is necessary, but not necessarily sufficient. We don’t have much of an understanding of the intelligence landscape beyond human-level intelligence, or even besides it. There may be other constraints and showstoppers, for example related to computability.
runarberg 1y ago
I feel like technological singularity has been pretty solidly ruled as junk science, like cold fusion, Malthusian collapse, or Lynn’s IQ regression. Technologists have made numerous predictions and hypothetical scenarios, non of which have come to fruition, nor does it seem likely at any time in the future. I think we should be treating AGI like Cold Fusion, phrenology, or even alchemy. It is not science, but science fiction. It is not going to happen and no research into AGI will provide anything of value (except for the grifters pushing the pseudo-science).
mmcconnell1618 1y ago
Self-learning opens new training opportunities but not at the scale or speed of current training. The world only operates at 1x speed. Today's models have been trained on written and visual content created by billions of humans over thousands of years. You can only experience the world in one place in real time. Even if you networked a bunch of "experiencers" together to gather real time data from many places at the same time, you would need a way to learn and train on that data in real time that could incorporate all the simultaneous inputs. I don't see that capability happening anytime soon.
- 8n4vidtmkvmk 1y ago
  Why not? Once a computer can learn at 1x speed (say one camera and one mic with which to observe the world), if it can indeed "learn" as fast as a human would, it sounds like all we need to do is throw more hardware at it at that point. And even if we couldn't, it could at least learn around the clock with no sleep. We can give it some specific task to solve and it could work tirelessly for years to solve it. Spin up one of these specialist bots for each tough problem we want solved.. and it'd still be beneficial because they like 10xPhD people without egos to get in the way or children to feed. Point is, I think self-learning at any speed is huge and as soon as it's achieved, it'll explode quadratically even if the first few years are slow.
hathawsh 1y ago
There are areas where we seem to be much closer to AGI than most people realize. AGI for software development, in particular, seems incredibly close. For example, Claude Code has bewildering capabilities that feel like magic. Mix it with a team of other capable development-oriented AIs and you might be able to build AI software that builds better AI software, all by itself.
- smellf 1y ago
  The "G" in AGI stands for "general", so talking about "AGI for software development" makes no sense, and worse than that accepts the AI companies' goalpost-shifting at face value. We shouldn't do that.
  - hathawsh 1y ago
    Perhaps. Intelligent beings are always more skilled in some domains than others. I don't know why AGI would be an exception to that rule.
    - 8n4vidtmkvmk 1y ago
      For starters, I don't think an AI can self-learn but only one subject. If it can teach itself how to program, it can surely teach itself a lot more.
  - cman1444 1y ago
    But I feel like the point is that, in order to reach AGI, the most important area for AI to be good at first is software development. Because of the feedback loop that could allow.
    - hathawsh 1y ago
      My point exactly. Thanks.
- ulrikrasmussen 1y ago
  Claude Code is good, but it is far from being AGI. I use it every day, but it is still very much reliant on a human guiding it. I think it in particular shows when it comes to core abstractions - it really lacks the "mathematical taste" of a good designer, and it doesn't engage in long-term adversarial thinking about what might be wrong with a particular choice in the context of the application and future usage scenarios. I think this type of thinking is a critical part of human creativity, and I can't see the current incarnation of agentic coding tools get there. They currently are way too reliant on a human carefully crafting the context and being careful of not putting in too many contradictory instructions or overloading the model with irrelevant details. An AGI has to be able to work productively on its own for days or weeks without going off on a tangent or suffering Xerox-like amnesia because it has compacted its context window 100 times.
Davidzheng 1y ago
should be next year in math domain tbh

hollownobody 1y ago

This reminds me of: https://en.m.wikipedia.org/wiki/Flying_Machines_Which_Do_Not_Fly https://en.m.wikipedia.org/wiki/Flying_Machines_Which_Do_Not...

hnlmorg 1y ago
For every example where someone over predicted the time it would take for a breakthrough, there are at least 10 examples of people being too optimistic with their predictions. And with AGI, you also have the likes of Sam Altman making up bullshit claims just to pump up the investment into OpenAI. So I wouldn’t take much of their claims seriously either. LLMs are a fantastic invention. But they’re far closer to SMS text predict than they are to generalised intelligence. Though what you might see is OpenAI et al redefine the term “AGI” just so they can say they’ve hit that milestone, again purely for their own financial gain.
- Davidzheng 1y ago
  in the history of AI usually people overestimate how long a capability is reached. There are very few counterexamples to this (GPT5 capability level might be one of them though)
- hollownobody 1y ago
  Are there any predictions you'd want to make? Not about AGI, but about an intermediate goalpost you think we won't reach in the next 5 years
zamadatix 1y ago
This reminds me how, a few years after the first fission power plant, Teller, Bhaba, and other nuclear physicists of the 1950s were convinced fusion power plants were about as far away as the physicists of today still predict they are. I'm cautiously optimistic of each technology, but the point is it's easy to find bullshit predictions without actually gaining any insight into what will happen with a given technology.

adastra22 1y ago

That is already happening. These labs are writing next gen models using next gen models, with greater levels of autonomy. That doesn’t get the hard takeoff people talk about because those hypotheticals don’t consider sources of error, noise, and drift.

hnlmorg 1y ago
They’re using lossy models to feedback into the training and research of new lossy models. But none of it is AGI self learning. You need both the generalised part of AGI and the ability to self learn. One without the other wouldn’t cause a singularity.
- adastra22 1y ago
  They are doing self-learning things. That’s what a lot of synthetic data is about. When managed by the AI, it is an AI picking what it want to train on in order to develop new capabilities. (Artificial General Intelligence says nothing about self-learning though. I presume you mean ASI?)
theobreuerweil 1y ago
The models may be writing the code but I would be surprised if they were contributing to the underlying science, which feels like the hard part
- Davidzheng 1y ago
  it's hardly science it's mostly experimentation + ablations on new ideas. but yeah idk if they are asking llms to generate these ideas. probably not good enough as is. though it doesn't seem outo f reach to RL on generating ideas for AI research
  - yberreby 1y ago
    I'm curious what you think qualifies as science.

Muromec 1y ago

We have an ability to self learn right now, but we stil suck at basics

jackomelon 1y ago
meatspace constraints!
hnlmorg 1y ago
There’s a lot of other variables at play for humans. Like - the need to sleep for 1/3 of our life - the need to eat, causing more pauses in work - much slower (like several orders of magnitude slower) data input capabilities - lossy storage (aka forgetfulness) - emotions - other primal urges, like the need to procreate
8n4vidtmkvmk 1y ago
Imagine never forgetting, and never getting bored or tired. I think we could achieve a lot more.

russellbeattie 1y ago

This is the key - right now each new model has had countless resources dedicated to training, then they are more or less set in stone until the next update. These big models don't dynamically update as days pass by - they don't learn. A personal assistant service may be able to mimic learning by creating a database of your data or preferences, but your usage isn't baked back into the big underlying model permanently. I don't agree with "in our lifetimes", but the difference between training and learning is the bright red line. Until there's a model which is able to continually update itself, it's not AGI. My guess is that this will require both more powerful hardware and a few more software innovations. But it'll happen.

layer8 1y ago

The ability to self-learn is necessary, but not necessarily sufficient. We don’t have much of an understanding of the intelligence landscape beyond human-level intelligence, or even besides it. There may be other constraints and showstoppers, for example related to computability.

runarberg 1y ago

I feel like technological singularity has been pretty solidly ruled as junk science, like cold fusion, Malthusian collapse, or Lynn’s IQ regression. Technologists have made numerous predictions and hypothetical scenarios, non of which have come to fruition, nor does it seem likely at any time in the future. I think we should be treating AGI like Cold Fusion, phrenology, or even alchemy. It is not science, but science fiction. It is not going to happen and no research into AGI will provide anything of value (except for the grifters pushing the pseudo-science).

mmcconnell1618 1y ago

Self-learning opens new training opportunities but not at the scale or speed of current training. The world only operates at 1x speed. Today's models have been trained on written and visual content created by billions of humans over thousands of years. You can only experience the world in one place in real time. Even if you networked a bunch of "experiencers" together to gather real time data from many places at the same time, you would need a way to learn and train on that data in real time that could incorporate all the simultaneous inputs. I don't see that capability happening anytime soon.

8n4vidtmkvmk 1y ago
Why not? Once a computer can learn at 1x speed (say one camera and one mic with which to observe the world), if it can indeed "learn" as fast as a human would, it sounds like all we need to do is throw more hardware at it at that point. And even if we couldn't, it could at least learn around the clock with no sleep. We can give it some specific task to solve and it could work tirelessly for years to solve it. Spin up one of these specialist bots for each tough problem we want solved.. and it'd still be beneficial because they like 10xPhD people without egos to get in the way or children to feed. Point is, I think self-learning at any speed is huge and as soon as it's achieved, it'll explode quadratically even if the first few years are slow.

hathawsh 1y ago

There are areas where we seem to be much closer to AGI than most people realize. AGI for software development, in particular, seems incredibly close. For example, Claude Code has bewildering capabilities that feel like magic. Mix it with a team of other capable development-oriented AIs and you might be able to build AI software that builds better AI software, all by itself.

smellf 1y ago
The "G" in AGI stands for "general", so talking about "AGI for software development" makes no sense, and worse than that accepts the AI companies' goalpost-shifting at face value. We shouldn't do that.
- hathawsh 1y ago
  Perhaps. Intelligent beings are always more skilled in some domains than others. I don't know why AGI would be an exception to that rule.
  - 8n4vidtmkvmk 1y ago
    For starters, I don't think an AI can self-learn but only one subject. If it can teach itself how to program, it can surely teach itself a lot more.
- cman1444 1y ago
  But I feel like the point is that, in order to reach AGI, the most important area for AI to be good at first is software development. Because of the feedback loop that could allow.
  - hathawsh 1y ago
    My point exactly. Thanks.
ulrikrasmussen 1y ago
Claude Code is good, but it is far from being AGI. I use it every day, but it is still very much reliant on a human guiding it. I think it in particular shows when it comes to core abstractions - it really lacks the "mathematical taste" of a good designer, and it doesn't engage in long-term adversarial thinking about what might be wrong with a particular choice in the context of the application and future usage scenarios. I think this type of thinking is a critical part of human creativity, and I can't see the current incarnation of agentic coding tools get there. They currently are way too reliant on a human carefully crafting the context and being careful of not putting in too many contradictory instructions or overloading the model with irrelevant details. An AGI has to be able to work productively on its own for days or weeks without going off on a tangent or suffering Xerox-like amnesia because it has compacted its context window 100 times.

Davidzheng 1y ago

should be next year in math domain tbh