6 ms·

It will be very difficult, but if you think about nuclear weapons it doesn't sound so hopeless. First nuclear weapons were used in 1945, almost 80 years ago. T

by toth 3y ago

It will be very difficult, but if you think about nuclear weapons it doesn't sound so hopeless. First nuclear weapons were used in 1945, almost 80 years ago. Today, out of ~ 200 countries in the world only 9 have nukes. Of those, only 4 did it after Non-proliferation of Nuclear Weapons Treaty. South Africa had them and gave them up under international pressure. Ukraine and other former soviet republics voluntarily gave them up after the breakup of the USSR. So, yes. Unfortunately we still have nukes and there are way too many of them, but it's not like non-proliferation efforts achieved nothing. Iran, a large country with a lot of resources, has been trying to get them a for a while with no luck for instance. And for frontier AI research you don't just need matrix multiplication, you need a lot of it. There's less than a handful of semiconductors fabs globally that can make the necessary hardware. And if you can't get to TSMC for some reason, you can get to ASML, NVDIA, etc.

nazka 3y ago
Getting uranium, the hardware to have nukes, and all the research and production around it is hard without being detected. Starting to code AI on a Mac and a few good GPUs or even a cluster without being detected is crazy easy. Even if they start to treat the most powerful GPUs as small nukes and only a few can have them it’s super easy to steal, transport, store, use… Nukes not so much.
- toth 3y ago
  For frontier AI models you don't need a few GPUs or a small cluster. You need thousands of them, you need them all connected with high bandwidth and you need a lot of power. It's not something you can do in your garage or undetected. There is some risk that over time people will be able to do more with less, and eventually you can train an AGI with a few dozen A100s. If that happens I agree there's nothing you can do, but until then there is a chance.
  - pixl97 3y ago
    What is the difference between a weather model, a nuclear bomb simulation, and training a LLM? At the end of the day all these are just calculations, of which you need thousands of processors to do with high accuracy. Pretty much what you're suggesting is a global police force to ensure we're performing the 'correct calculations'. Having to form an authoritarian world computer police isn't much of a solution to the problem either. This just happens to work well in the nuclear world because it's hard to shield and the average person doesn't have a need for uranium.
    - toth 3y ago
      If we had to have global policing of every large computing cluster and everything that runs there, that does not feel to me like a huge overreach or something that could easily slide into totalitarianism. It would be better if we didn't need to do this, but I don't see a less intrusive way. Of course, if you don't think there is a real existential threat then it's pointless and bad to do this, it all depends on that premise.
      - pixl97 3y ago
        Intelligence is a real potential existential threat, just ask every large mammal as of about 10k years ago, save those in Africa. The question is, is an authoritarian nightmare that much better than getting tossed into the meat grinder?
        toth 3y ago
        > The question is, is an authoritarian nightmare that much better than getting tossed into the meat grinder? If the choice was between those two extremes, I'd take the authoritarian nightmare. Extinction is forever, dictatorships end.
VincentEvans 3y ago
> Ukraine and other former soviet republics voluntarily gave them up after the breakup of the USSR. And no country will ever make the same mistake again. Because the west promised to protect them from aggression. But when the time came to do just that - everyone involved dusted off their copy of Budapest memorandum and handed it over to the lawyers. Sorry for digression. Carry on.
- toth 3y ago
  For sure. If the Russian federation breaks up into a bunch of independent states we will have a huge problem. It was not very fair from civil war already, with the aborted coup. Could you imagine if the Wagner group became a nuclear power??
upupupandaway 3y ago
So... 44% of countries that have obtained nuclear weapons did so after the Nuclear Weapons Treaty. China alone added 60 warheads in 2022. It doesn't matter that it works 90% of the time - if a single entity is able to break out of the prohibition, it's game over.

by toth 3y ago

nazka 3y ago

Getting uranium, the hardware to have nukes, and all the research and production around it is hard without being detected. Starting to code AI on a Mac and a few good GPUs or even a cluster without being detected is crazy easy. Even if they start to treat the most powerful GPUs as small nukes and only a few can have them it’s super easy to steal, transport, store, use… Nukes not so much.

toth 3y ago
For frontier AI models you don't need a few GPUs or a small cluster. You need thousands of them, you need them all connected with high bandwidth and you need a lot of power. It's not something you can do in your garage or undetected. There is some risk that over time people will be able to do more with less, and eventually you can train an AGI with a few dozen A100s. If that happens I agree there's nothing you can do, but until then there is a chance.
- pixl97 3y ago
  What is the difference between a weather model, a nuclear bomb simulation, and training a LLM? At the end of the day all these are just calculations, of which you need thousands of processors to do with high accuracy. Pretty much what you're suggesting is a global police force to ensure we're performing the 'correct calculations'. Having to form an authoritarian world computer police isn't much of a solution to the problem either. This just happens to work well in the nuclear world because it's hard to shield and the average person doesn't have a need for uranium.
  - toth 3y ago
    If we had to have global policing of every large computing cluster and everything that runs there, that does not feel to me like a huge overreach or something that could easily slide into totalitarianism. It would be better if we didn't need to do this, but I don't see a less intrusive way. Of course, if you don't think there is a real existential threat then it's pointless and bad to do this, it all depends on that premise.
    - pixl97 3y ago
      Intelligence is a real potential existential threat, just ask every large mammal as of about 10k years ago, save those in Africa. The question is, is an authoritarian nightmare that much better than getting tossed into the meat grinder?
      - toth 3y ago
        > The question is, is an authoritarian nightmare that much better than getting tossed into the meat grinder? If the choice was between those two extremes, I'd take the authoritarian nightmare. Extinction is forever, dictatorships end.

VincentEvans 3y ago

> Ukraine and other former soviet republics voluntarily gave them up after the breakup of the USSR. And no country will ever make the same mistake again. Because the west promised to protect them from aggression. But when the time came to do just that - everyone involved dusted off their copy of Budapest memorandum and handed it over to the lawyers. Sorry for digression. Carry on.

toth 3y ago
For sure. If the Russian federation breaks up into a bunch of independent states we will have a huge problem. It was not very fair from civil war already, with the aborted coup. Could you imagine if the Wagner group became a nuclear power??

upupupandaway 3y ago

So... 44% of countries that have obtained nuclear weapons did so after the Nuclear Weapons Treaty. China alone added 60 warheads in 2022. It doesn't matter that it works 90% of the time - if a single entity is able to break out of the prohibition, it's game over.