6 ms·

I keep seeing this and I want to speak to it. When Dario and others say things like "this is happening and we should probably figure out what to do about it" w

by ctoth 4mo ago

I keep seeing this and I want to speak to it. When Dario and others say things like "this is happening and we should probably figure out what to do about it" what ends up happening is people hear "this is happening," see that the person warning them is the person doing the thing, and then short-circuit. "Why can't you just stop then?" Dario's point, and the point of the people actually trying to solve the problem, is that AI is not just Anthropic and OpenAI. It's the knowledge that you can put more compute in, and get more capability out. It is a technology now. It exists, in the world. Wishing will not make it go away. Being angry at it will not make it go away. Lying about how much water it uses will not make it go away. If Anthropic and OpenAI Shut down tomorrow, Accenture will not say "oh guess that llm thing won't work, let's go back to hiring humans!" It is a truth that you can multiply matrices and get something that is economically useful. We cannot un-know this. Physics allows it, so it will happen. So we should probably figure out what the heck to do about it. If your answer is something along the lines of "restrict it" then 1. let me know how that goes when other people don't, and 2. I really would rather prefer a world where we have the machines do the work the machines can do, not a world where we have human makework. If this means we need to figure out redistribution, let's talk about redistribution!

somesortofthing 4mo ago
A US-China AGI ban treaty could prevent superintelligence indefinitely. Data centers are hard to hide. Have fun buying GPUs when you're cut off from all global payments. America would have to make some unpleasant concessions but that seems like a solid trade for preventing a wide variety of nightmare futures.
- deleted 4mo ago
  [deleted]
- leonidasv 4mo ago
  Is very difficult (not to say impossible) to ban a ill-defined thing.
mbgerring 4mo ago
Physics allows this, and actually taking advantage of it requires billions of dollars of unprecedented infrastructure buildout that is already destabilizing the power grid. The only reason that infrastructure buildout is happening at all is the ideological capture of a small handful of obscenely wealthy people, who are fueling this buildout by spreading the extreme paranoia you’re echoing here. I do not understand why no one else can see the circularity of this reasoning. There is nothing inevitable about tying up all of this productive capital in the pursuit of AI. There are many, many other projects requiring similar capital and human effort, with much more obvious payoffs, such as decarbonizing the world’s energy systems. “It’s physically possible to provide abundant electricity without burning fossil fuels” is more provably true than any of the insane science fiction bullshit that undergirds the AI buildout, and yet, the entire clean energy industry is still having to build insane financial Rube Goldberg contraptions to make incremental progress. “Inevitability” is a lie, period. This entire thing is extremely historically contingent, and we could easily stop this train tomorrow.
- ctoth 4mo ago
  > This entire thing is extremely historically contingent, and we could easily stop this train tomorrow. So, the Baruch Plan? The Manhattan Project was $~2B in 1945 dollars, and a national-scale industrial mobilization. Now North Korea has the bomb. That's with nuclear material, which doesn't get easier and easier and easier to work with every year. Compare to the price to train GPT-2 in 2019 ($43,000), and in 2026 ($73) [0]. [0]: https://x.com/karpathy/status/2017703360393318587 https://x.com/karpathy/status/2017703360393318587
  - YeGoblynQueenne 4mo ago
    Since nobody uses GPT-2 any more it's more informative to compare the price to train GPT-2 in 2019 and the price to train GPT 5.5 in 2026. Unfortunately that cost is not disclosed but it's probably in the billions. The point being: the price to train frontier models isn't coming down, nor is it going to come down because for models to remain on the frontier they have to keep getting bigger and bigger (and trained on more and more data).
- 4mo ago
JumpCrisscross 4mo ago
> When Dario and others say things like "this is happening and we should probably figure out what to do about it" what ends up happening is people hear "this is happening," see that the person warning them is the person doing the thing, and then short-circuit. "Why can't you just stop then?" I still believe Dario asks these questions in good faith. Nobody believes that about e.g. Sam Altman or Elon Musk. They compared themselves to Oppenheimer because it helped them get attention. When it started an actual regulatory conversation, they were suddenly less worried.
aidenn0 4mo ago
I don't think we can stop it, but the people saying "this is happening and we should probably figure out what to do about it" are doing their best to accelerate it, which gives us less time to figure out what to do about it.
- fragmede 4mo ago
  The people saying this is happening are competing with each other for resources, so there's no way for one of them to hold back without losing out to the others. We see this with people dropping Claude subscriptions in favor of ChatGPT because codex 5.5 > Claude Opus 4.6/7/8. Anthropic is losing money by not releasing Mythos.
  - aidenn0 4mo ago
    You are proposing that going from N companies chasing AGI to N-1 companies chasing AGI will have zero effect on when AGI happens?
    - fragmede 4mo ago
      Aidenn0 said they're doing their best to accelerate it, and I'm saying "they're" isn't a single monolithic entity and that they're in competition with each other so they're incentivized to go as fast as possible, so it would be hard to hold them back.
      - palmotea 4mo ago
        > Aidenn0 said they're doing their best to accelerate it, and I'm saying "they're" isn't a single monolithic entity and that they're in competition with each other so they're incentivized to go as fast as possible, so it would be hard to hold them back. Maybe it would run afoul of antitrust regulations, but it's totally realistic for all those competitors to get together and say "hey, we could really fuck up society in our race to get rich with this tech, lets all slow down." And if these companies are run my mature people who don't subordinate every consideration to greed, they'd to it.
ksk23 4mo ago
Hot take, but concentration camps were invented in the past. No need to use them, though

by ctoth 4mo ago

somesortofthing 4mo ago

A US-China AGI ban treaty could prevent superintelligence indefinitely. Data centers are hard to hide. Have fun buying GPUs when you're cut off from all global payments. America would have to make some unpleasant concessions but that seems like a solid trade for preventing a wide variety of nightmare futures.

deleted 4mo ago
[deleted]
leonidasv 4mo ago
Is very difficult (not to say impossible) to ban a ill-defined thing.

mbgerring 4mo ago

Physics allows this, and actually taking advantage of it requires billions of dollars of unprecedented infrastructure buildout that is already destabilizing the power grid. The only reason that infrastructure buildout is happening at all is the ideological capture of a small handful of obscenely wealthy people, who are fueling this buildout by spreading the extreme paranoia you’re echoing here. I do not understand why no one else can see the circularity of this reasoning. There is nothing inevitable about tying up all of this productive capital in the pursuit of AI. There are many, many other projects requiring similar capital and human effort, with much more obvious payoffs, such as decarbonizing the world’s energy systems. “It’s physically possible to provide abundant electricity without burning fossil fuels” is more provably true than any of the insane science fiction bullshit that undergirds the AI buildout, and yet, the entire clean energy industry is still having to build insane financial Rube Goldberg contraptions to make incremental progress. “Inevitability” is a lie, period. This entire thing is extremely historically contingent, and we could easily stop this train tomorrow.

ctoth 4mo ago
> This entire thing is extremely historically contingent, and we could easily stop this train tomorrow. So, the Baruch Plan? The Manhattan Project was $~2B in 1945 dollars, and a national-scale industrial mobilization. Now North Korea has the bomb. That's with nuclear material, which doesn't get easier and easier and easier to work with every year. Compare to the price to train GPT-2 in 2019 ($43,000), and in 2026 ($73) [0]. [0]: https://x.com/karpathy/status/2017703360393318587 https://x.com/karpathy/status/2017703360393318587
- YeGoblynQueenne 4mo ago
  Since nobody uses GPT-2 any more it's more informative to compare the price to train GPT-2 in 2019 and the price to train GPT 5.5 in 2026. Unfortunately that cost is not disclosed but it's probably in the billions. The point being: the price to train frontier models isn't coming down, nor is it going to come down because for models to remain on the frontier they have to keep getting bigger and bigger (and trained on more and more data).
4mo ago

JumpCrisscross 4mo ago

> When Dario and others say things like "this is happening and we should probably figure out what to do about it" what ends up happening is people hear "this is happening," see that the person warning them is the person doing the thing, and then short-circuit. "Why can't you just stop then?" I still believe Dario asks these questions in good faith. Nobody believes that about e.g. Sam Altman or Elon Musk. They compared themselves to Oppenheimer because it helped them get attention. When it started an actual regulatory conversation, they were suddenly less worried.

aidenn0 4mo ago

I don't think we can stop it, but the people saying "this is happening and we should probably figure out what to do about it" are doing their best to accelerate it, which gives us less time to figure out what to do about it.

fragmede 4mo ago
The people saying this is happening are competing with each other for resources, so there's no way for one of them to hold back without losing out to the others. We see this with people dropping Claude subscriptions in favor of ChatGPT because codex 5.5 > Claude Opus 4.6/7/8. Anthropic is losing money by not releasing Mythos.
- aidenn0 4mo ago
  You are proposing that going from N companies chasing AGI to N-1 companies chasing AGI will have zero effect on when AGI happens?
  - fragmede 4mo ago
    Aidenn0 said they're doing their best to accelerate it, and I'm saying "they're" isn't a single monolithic entity and that they're in competition with each other so they're incentivized to go as fast as possible, so it would be hard to hold them back.
    - palmotea 4mo ago
      > Aidenn0 said they're doing their best to accelerate it, and I'm saying "they're" isn't a single monolithic entity and that they're in competition with each other so they're incentivized to go as fast as possible, so it would be hard to hold them back. Maybe it would run afoul of antitrust regulations, but it's totally realistic for all those competitors to get together and say "hey, we could really fuck up society in our race to get rich with this tech, lets all slow down." And if these companies are run my mature people who don't subordinate every consideration to greed, they'd to it.

ksk23 4mo ago

Hot take, but concentration camps were invented in the past. No need to use them, though