4 ms·

Whoever gets AGI first owns the future though, any GDP put into manufacturing not essential to that goal is a geopolitical opportunity cost

buellerbueller 1y ago
Whoever gets AGI first will just be another biobattery to power the matrix.
rhubarbtree 1y ago
Why is that the case? If a company gets to AGI a month later, why does that matter so much? We’re not talking super intelligence here, just human level intelligence. OpenAI was first to ChatGPT yet other companies are still in the game.
- atleastoptimal 1y ago
  My argument is based on 1. The first company to get AGI will likely have a multitude of high-leverage problems it would immediately put AGI to task on 2. One of those problems is simply improving itself. Another is securing that company's lead over its competitors (by basically helping every employee at that company do better at their job) 3. The company that reaches AGI for a language-style model will likely do so due to a mix of architectural tricks that can be applied to any general-purpose model, including chip design, tactical intelligence, persuasion, beating the stock market, etc
  - thundoe 1y ago
    Unless AGI includes a speed requirement, AGI is not sufficient to win the market. Take any genius in human history, the impact they had has been hugely limited by their lifespan, they didn’t solve every problem, and each discovery took them decades. The first AGIs will be the same, hyper slow for a while, giving competitors a chance to copy and stay in the race
    - atleastoptimal 1y ago
      If Einstein could replicate himself 1000x, perform all his thought experiments in parallel, not have to eat, sleep, or be limited by his human short-term memory, he would have likely have accomplished all he did far faster. AGI will at the very minimum have all the advantages of humans plus all the advantages of computers, which would be a huge automatic boon even without the fact that an AGI at human level would likely be able to scale even further.
  - mkl 1y ago
    All of that stuff takes time and resources. Self-improvement may not be easy, e.g. if they end up in a local maximum that doesn't extend, and it probably won't be cheap or fast (if it's anything like frontier LLMs it could be months of computation on enormous numbers of cutting-edge devices, costing hundreds of millions or billions, or it may not even be possible without inventing and mass-manufacturing better hardware). Another company achieving a slightly different form of AGI within a few years will probably be at least competitive, and if they have more resources or a better design they could overtake.
    - atleastoptimal 1y ago
      All the major companies are setting up to have all the hardware necessary to leverage an AGI-level system when it does emerge. The entire purpose of Starbase for OpenAI or the memphis supercomputer for X.ai is such that there would be very little delay between hitting the right marks on an AGI-capable model and deploying its capabilities en masse towards the highest-leverage aims. Certainly there will be bottlenecks, but the advantages will significantly accelerate progress.
  - ahoka 1y ago
    These companies already have access to the best meat-brains in the world and what tasks do they work on? Advertisement mostly?
    - atleastoptimal 1y ago
      Ads pay the bills, but the top tier researchers are all working on top-tier AI research.
  - jimbohn 1y ago
    The AGI argument assumes there is a 0 -> 1 moment where the model suddenly becomes "AGI" and starts performing miraculous tasks, accompanied by recursive self-improvement. So far, our experience shows that we are getting incremental improvements over time from different companies. These things are being commoditized, and we are still at the start of the curve when it comes to hardware, data centers, etc. Arguing for an all-in civilization/country bet on AGI given this premise, is either foolish or a sign that you are selling "AGI"
    - atleastoptimal 1y ago
      It won't be one miraculous moment, but once a model is capable of performing tasks on its own and verifying its outputs better than a domain-expert using that AI, a human no longer being a bottleneck will allow this model to be deployed on a far larger scale, which would appear much like a 0-1 moment on the outside, even if the improvement over a previous model is minor and only involves a slight tweak of something that increases reliability.
  - rhubarbtree 1y ago
    Wait, but why would an AGI be better at improving itself compared to an AI researcher doing it? AGI is not super intelligence.
    - atleastoptimal 1y ago
      Every frontier lab has maybe 100 top tier researchers, all of whom have limited short-term memory, need to sleep, eat, have biases, and can't rigorously parallelize themselves and run thousands of simultaneous experiments, while also modifying their own internal thought processes to a quantifiable degree. An AGI merely on the level of a mediocre AI researcher would be able to trivially leverage all these benefits, while also simply scaling its own compute (it is unlikely that the scaling gains would level out precisely at the level of a smart human)
      - rhubarbtree 1y ago
        > An AGI merely on the level of a mediocre AI researcher Perhaps it's semantics, but I'd say AGI is around the average human intelligence, not extreme outliers like mediocre AI researchers.
- YetAnotherNick 1y ago
  The argument is something like AGI or its owner wouldn't want other AGIs to exist. So it would destroy the capabilities of other AGI before it could evolve(by things like hacking, manipulation etc.).
  - buellerbueller 1y ago
    Oh, so the goal is to create an insane and predatory piece of software that is out of the control of its creators? Sounds wonderful.
agubelu 1y ago
Even assuming that's the case, everyone's acting like throwing more GPUs at the problem is somehow gonna get them to AGI
- atleastoptimal 1y ago
  Far more is being done than simply throwing more GPU's at the problem. GPT-5 required less compute to train than GPT-4.5. Data, RL, architectural improvements, etc. all contribute to the rate of improvement we're seeing now.
  - 4gotunameagain 1y ago
    The very idea that AGI will arise from LLMs is ridiculous at best. Computer science hubris at its finest.
    - atleastoptimal 1y ago
      Why is it ridiculous that an LLM or a system similar to or built off of an LLM could reach AGI?
      - saubeidl 1y ago
        Because LLMs are just stochastic parrots and don't do any thinking.
        atleastoptimal 1y ago
        Humans who repeatedly deny LLM capabilities despite the numerous milestones they've surpassed seem more like stochastic parrots. The same arguments are always brought up, often short pithy one-liners without much clarification. It seems silly that despite this argument first emerging when LLM's could barely write functional code, now that LLM's have reached gold-medal performance on the IMO, it is still being made with little interrogation into its potential faults, or clarification on the precise boundary of intelligence LLM's will never be able to cross.
alsetmusic 1y ago
We have an AI promoter here. AGI isn't the future of anything right now. It could be. But so could a lot of things, like vaccine research (we're making promising development on HIV and cancer). Try saying those people would own the future in the 1980s-1990s. Sure, that'd be an obvious outcome, but it wasn't on the horizon for the people in the field at the time (unless your family owned the company).
- atleastoptimal 1y ago
  Even if you could cure cancer or HIV with a vaccine it would have a relatively negligible impact compared to AGI. There are far more signals that AGI is going to be achieved by OpenAI, Anthropic, DeepMind or X.ai within the next 5-10 years than there were of any other hyped breakthrough in the past 100 years that ultimately never came to fruition. Doesn't mean it's guaranteed to happen, but to ignore the multitude of trends which show no signs of stopping, it seems naive in Anno Domani 2025 to discount it as a likely possibility.
  - dax_ 1y ago
    It's just as possible that they need to invest more and more for negligible improvements to model performance. These companies are burning through money at an astonishing rate. And as the internet deteriorates due to AI slop, finding good training material will become increasingly difficult. It's already happening that incorrect AI generated information is being cited as source for new AI answers.
    - atleastoptimal 1y ago
      They are burning through money, but their revenue is scaling at a similar rate. I'm sure most companies have understood the "AI outputs feeding AI's" incest issue for a while and have many methods to avoiding it. That's why so much has been put into synthetic data pipelines for years.
  - lm28469 1y ago
    > There are far more signals that AGI is going to be achieved by OpenAI, Anthropic, DeepMind or X.ai within the next 5-10 years So agi before autonomous tesla? "Just two more years guys I promise", how can people keep falling for these lol
saubeidl 1y ago
There is no proof "AGI" is a real thing. Any GDP put towards that goal is a huge gamble and the US is all-in, with potentially ruinous results.
lm28469 1y ago
It really shows how desperate some people are, sacrificing everything, present and future, in the quest of a digital god that might not even exist.
theevilsharpie 1y ago
I have seen no credible explanation on how current or proposed technology can possibly achieve AGI. If you want to hand-wave that away by stating that any company with technology capable of achieving AGI would guard it as the most valuable trade secret in history, then fine. Even if we assume that AGI-capable technology exists in secret somewhere, I've seen no credible explanation from any organization on how they plan to control an AGI and reliably convince it to produce useful work (rather than the AGI just turning into a real-life SHODAN). An uncontrollable AGI would be, at best, functionally useless. AGI is --- and for the foreseeable future, will continue to be --- science fiction.
- atleastoptimal 1y ago
  You seem to have two separate claims. The first that it would be difficult to achieve AGI with current or proposed technology, and the second being that it would be difficult to control AGI, thus making it too risky to use or deploy. The second is a significant open problem (the alignment problem) and I'd wager it is a very real risk which companies need to take more seriously. However, whether it would be feasible to control or direct an AGI towards reliably safe, useful outputs has no bearing on whether reaching AGI is possible via current methods. Current scaling gains and the rate of improvement (see METR's horizons on work an AI model can do reliably on its own) make it fairly plausible, at least more plausible than the plain denial that AGI is possible I see around here with very little evidence.
buellerbueller 1y ago
Thats a nice assertion, but do you have any facts?
- atleastoptimal 1y ago
  It is a reasonable common-sense claim that if a company which possesses a model that can perform all cognitive tasks as well as a human or better, that that model would be more powerful than any other technology available, barring significant limitations to its operation or deployment.
  - buellerbueller 1y ago
    >any GDP put into manufacturing not essential to that goal is a geopolitical opportunity cost is the assertion I am referring to; you do a lot of handwaving to assume that AGI is achievable.

buellerbueller 1y ago

Whoever gets AGI first will just be another biobattery to power the matrix.

rhubarbtree 1y ago

Why is that the case? If a company gets to AGI a month later, why does that matter so much? We’re not talking super intelligence here, just human level intelligence. OpenAI was first to ChatGPT yet other companies are still in the game.

atleastoptimal 1y ago
My argument is based on 1. The first company to get AGI will likely have a multitude of high-leverage problems it would immediately put AGI to task on 2. One of those problems is simply improving itself. Another is securing that company's lead over its competitors (by basically helping every employee at that company do better at their job) 3. The company that reaches AGI for a language-style model will likely do so due to a mix of architectural tricks that can be applied to any general-purpose model, including chip design, tactical intelligence, persuasion, beating the stock market, etc
- thundoe 1y ago
  Unless AGI includes a speed requirement, AGI is not sufficient to win the market. Take any genius in human history, the impact they had has been hugely limited by their lifespan, they didn’t solve every problem, and each discovery took them decades. The first AGIs will be the same, hyper slow for a while, giving competitors a chance to copy and stay in the race
  - atleastoptimal 1y ago
    If Einstein could replicate himself 1000x, perform all his thought experiments in parallel, not have to eat, sleep, or be limited by his human short-term memory, he would have likely have accomplished all he did far faster. AGI will at the very minimum have all the advantages of humans plus all the advantages of computers, which would be a huge automatic boon even without the fact that an AGI at human level would likely be able to scale even further.
- mkl 1y ago
  All of that stuff takes time and resources. Self-improvement may not be easy, e.g. if they end up in a local maximum that doesn't extend, and it probably won't be cheap or fast (if it's anything like frontier LLMs it could be months of computation on enormous numbers of cutting-edge devices, costing hundreds of millions or billions, or it may not even be possible without inventing and mass-manufacturing better hardware). Another company achieving a slightly different form of AGI within a few years will probably be at least competitive, and if they have more resources or a better design they could overtake.
  - atleastoptimal 1y ago
    All the major companies are setting up to have all the hardware necessary to leverage an AGI-level system when it does emerge. The entire purpose of Starbase for OpenAI or the memphis supercomputer for X.ai is such that there would be very little delay between hitting the right marks on an AGI-capable model and deploying its capabilities en masse towards the highest-leverage aims. Certainly there will be bottlenecks, but the advantages will significantly accelerate progress.
- ahoka 1y ago
  These companies already have access to the best meat-brains in the world and what tasks do they work on? Advertisement mostly?
  - atleastoptimal 1y ago
    Ads pay the bills, but the top tier researchers are all working on top-tier AI research.
- jimbohn 1y ago
  The AGI argument assumes there is a 0 -> 1 moment where the model suddenly becomes "AGI" and starts performing miraculous tasks, accompanied by recursive self-improvement. So far, our experience shows that we are getting incremental improvements over time from different companies. These things are being commoditized, and we are still at the start of the curve when it comes to hardware, data centers, etc. Arguing for an all-in civilization/country bet on AGI given this premise, is either foolish or a sign that you are selling "AGI"
  - atleastoptimal 1y ago
    It won't be one miraculous moment, but once a model is capable of performing tasks on its own and verifying its outputs better than a domain-expert using that AI, a human no longer being a bottleneck will allow this model to be deployed on a far larger scale, which would appear much like a 0-1 moment on the outside, even if the improvement over a previous model is minor and only involves a slight tweak of something that increases reliability.
- rhubarbtree 1y ago
  Wait, but why would an AGI be better at improving itself compared to an AI researcher doing it? AGI is not super intelligence.
  - atleastoptimal 1y ago
    Every frontier lab has maybe 100 top tier researchers, all of whom have limited short-term memory, need to sleep, eat, have biases, and can't rigorously parallelize themselves and run thousands of simultaneous experiments, while also modifying their own internal thought processes to a quantifiable degree. An AGI merely on the level of a mediocre AI researcher would be able to trivially leverage all these benefits, while also simply scaling its own compute (it is unlikely that the scaling gains would level out precisely at the level of a smart human)
    - rhubarbtree 1y ago
      > An AGI merely on the level of a mediocre AI researcher Perhaps it's semantics, but I'd say AGI is around the average human intelligence, not extreme outliers like mediocre AI researchers.
YetAnotherNick 1y ago
The argument is something like AGI or its owner wouldn't want other AGIs to exist. So it would destroy the capabilities of other AGI before it could evolve(by things like hacking, manipulation etc.).
- buellerbueller 1y ago
  Oh, so the goal is to create an insane and predatory piece of software that is out of the control of its creators? Sounds wonderful.

agubelu 1y ago

Even assuming that's the case, everyone's acting like throwing more GPUs at the problem is somehow gonna get them to AGI

atleastoptimal 1y ago
Far more is being done than simply throwing more GPU's at the problem. GPT-5 required less compute to train than GPT-4.5. Data, RL, architectural improvements, etc. all contribute to the rate of improvement we're seeing now.
- 4gotunameagain 1y ago
  The very idea that AGI will arise from LLMs is ridiculous at best. Computer science hubris at its finest.
  - atleastoptimal 1y ago
    Why is it ridiculous that an LLM or a system similar to or built off of an LLM could reach AGI?
    - saubeidl 1y ago
      Because LLMs are just stochastic parrots and don't do any thinking.
      - atleastoptimal 1y ago
        Humans who repeatedly deny LLM capabilities despite the numerous milestones they've surpassed seem more like stochastic parrots. The same arguments are always brought up, often short pithy one-liners without much clarification. It seems silly that despite this argument first emerging when LLM's could barely write functional code, now that LLM's have reached gold-medal performance on the IMO, it is still being made with little interrogation into its potential faults, or clarification on the precise boundary of intelligence LLM's will never be able to cross.

alsetmusic 1y ago

We have an AI promoter here. AGI isn't the future of anything right now. It could be. But so could a lot of things, like vaccine research (we're making promising development on HIV and cancer). Try saying those people would own the future in the 1980s-1990s. Sure, that'd be an obvious outcome, but it wasn't on the horizon for the people in the field at the time (unless your family owned the company).

atleastoptimal 1y ago
Even if you could cure cancer or HIV with a vaccine it would have a relatively negligible impact compared to AGI. There are far more signals that AGI is going to be achieved by OpenAI, Anthropic, DeepMind or X.ai within the next 5-10 years than there were of any other hyped breakthrough in the past 100 years that ultimately never came to fruition. Doesn't mean it's guaranteed to happen, but to ignore the multitude of trends which show no signs of stopping, it seems naive in Anno Domani 2025 to discount it as a likely possibility.
- dax_ 1y ago
  It's just as possible that they need to invest more and more for negligible improvements to model performance. These companies are burning through money at an astonishing rate. And as the internet deteriorates due to AI slop, finding good training material will become increasingly difficult. It's already happening that incorrect AI generated information is being cited as source for new AI answers.
  - atleastoptimal 1y ago
    They are burning through money, but their revenue is scaling at a similar rate. I'm sure most companies have understood the "AI outputs feeding AI's" incest issue for a while and have many methods to avoiding it. That's why so much has been put into synthetic data pipelines for years.
- lm28469 1y ago
  > There are far more signals that AGI is going to be achieved by OpenAI, Anthropic, DeepMind or X.ai within the next 5-10 years So agi before autonomous tesla? "Just two more years guys I promise", how can people keep falling for these lol

saubeidl 1y ago

There is no proof "AGI" is a real thing. Any GDP put towards that goal is a huge gamble and the US is all-in, with potentially ruinous results.

lm28469 1y ago

It really shows how desperate some people are, sacrificing everything, present and future, in the quest of a digital god that might not even exist.

theevilsharpie 1y ago

I have seen no credible explanation on how current or proposed technology can possibly achieve AGI. If you want to hand-wave that away by stating that any company with technology capable of achieving AGI would guard it as the most valuable trade secret in history, then fine. Even if we assume that AGI-capable technology exists in secret somewhere, I've seen no credible explanation from any organization on how they plan to control an AGI and reliably convince it to produce useful work (rather than the AGI just turning into a real-life SHODAN). An uncontrollable AGI would be, at best, functionally useless. AGI is --- and for the foreseeable future, will continue to be --- science fiction.

atleastoptimal 1y ago
You seem to have two separate claims. The first that it would be difficult to achieve AGI with current or proposed technology, and the second being that it would be difficult to control AGI, thus making it too risky to use or deploy. The second is a significant open problem (the alignment problem) and I'd wager it is a very real risk which companies need to take more seriously. However, whether it would be feasible to control or direct an AGI towards reliably safe, useful outputs has no bearing on whether reaching AGI is possible via current methods. Current scaling gains and the rate of improvement (see METR's horizons on work an AI model can do reliably on its own) make it fairly plausible, at least more plausible than the plain denial that AGI is possible I see around here with very little evidence.

buellerbueller 1y ago

Thats a nice assertion, but do you have any facts?

atleastoptimal 1y ago
It is a reasonable common-sense claim that if a company which possesses a model that can perform all cognitive tasks as well as a human or better, that that model would be more powerful than any other technology available, barring significant limitations to its operation or deployment.
- buellerbueller 1y ago
  >any GDP put into manufacturing not essential to that goal is a geopolitical opportunity cost is the assertion I am referring to; you do a lot of handwaving to assume that AGI is achievable.