4 ms·

I thought the discussion was getting to the important points once hyperbole was resolved but Carmack ducks out Carmack: > To answer the question directly: the

I thought the discussion was getting to the important points once hyperbole was resolved but Carmack ducks out Carmack: > To answer the question directly: there is no way, even with an infinitely smart (but not supernatural) oracle, that a teenager can destroy all of humanity in six months. Even the most devastating disasters people imagine will not kill every human being even over the space of years. This is a common failure of imaginative scope, not understanding all the diverse situations that people are already in, and how they will respond to disasters. You could walk it back to “kill a million people” and get into the realm of an actual discussion, but it isn’t one I am very interested in having. --- I do think that's a discussion very worth having, asymmetrical advantage that AI gives is the concern here. A virus is a special arrangement of a few thousand atoms that we can now control and design on a computer. The defence is global vaccination programs costing billions of dollars and many lives in the process We will no doubt be able to use AI to develop vaccines, that isn't the bottleneck, the bottleneck is in testing and distribution. It's a fundamentally asymmetrical situation To be clear this is a risk without AI, but AI brings down the threshold of effort to make this more likely. I would love to be convinced otherwise

Salgat 3y ago
The defense in this case would be everyone else's infinitely smart oracles.
- m4rtink 3y ago
  Or make people more robust & harder/impossible to kill. That would solve so many problems once and for all!
- Cthulhu_ 3y ago
  If everyone has an infinitely smart oracle, nobody has! or something.
chrisco255 3y ago
Can we actually control viruses? No. 2020 was evidence enough of that. We were comically bad at it. Thankfully, immune systems have co-evolved with viruses for millions of years and there's limits to what viruses can do. After all, the most successful viruses don't kill off their hosts (since they would also die off). It quickly became apparent that Covid could mutate quicker than we could possibly keep up with. Like the common cold and the flu, it iterates too quickly to eliminate entirely like smallpox. At any rate, it seems all you need for AI is attention (transformers + processing power). If AI ever becomes a real threat outside of generating believable fake images and videos, it's likely the counter is a diversity of AI models that can compete with the "evil one".
- haxiomic 3y ago
  It was surprising to me to learn that we can be _a lot_ faster than evolution when it comes to 'improving' viruses. The process is called gain-of-function. It can create mutations that are rare or impossible in nature. Viruses don't select for devastation on their own, they select only for propagation We will no doubt be able to use AI to develop vaccines that isn't the bottleneck, the bottleneck is in testing and distribution (and humans not wanting to get vaccinated). It's a fundamentally asymmetrical situation
  - Cthulhu_ 3y ago
    > Viruses don't select for devastation on their own, they select only for propagation Which was observed in real-time with the 'rona mutating to become more contagious, but less dangerous.
krona 3y ago
> A virus is a... You don't need an AI to construct a virus that can destroy humanity. We already have that capability, and bioprinting one by average grad at random biotech on their lunch break will become commonplace soon.
- haxiomic 3y ago
  Exactly, the point is the risk AI helps bring that capability to everyone, not just those in the biotech field
  - arrakeen 3y ago
    i'm pretty sure the limiting factors are the materials and machinery, both of which AI cannot help with
    - haxiomic 3y ago
      It takes less than you think, you can order sequences online Here's a youtuber designing a custom virus to cure his lactose intolerance https://www.youtube.com/watch?v=J3FcbFqSoQY&ab_channel=TheThoughtEmporium https://www.youtube.com/watch?v=J3FcbFqSoQY&ab_channel=TheTh...
- goatlover 3y ago
  What virus can destroy humanity?
zigzag312 3y ago
The hyperbole could go other way too. Eg. When only government have access to an omniscient oracle advisor and advanced AI technology, how long would it take on average for a democracy to turn into a dictatorship? Considering the asymmetry in access to information and technology between a government and civilians, a bad government would have a power to take complete control over a country. With sufficiently advanced AI technology only a handful of people are needed to maintain control over a large population and prevent any attempts to overthrow it. I don't have time right now to come up with an proper transition, but the hyperbole concludes that only one insane dictator with access to an omniscient oracle advisor is needed to destroy the whole world. This might be even more "probable" than a teenager destroying all of humanity in six months.

Salgat 3y ago

The defense in this case would be everyone else's infinitely smart oracles.

m4rtink 3y ago
Or make people more robust & harder/impossible to kill. That would solve so many problems once and for all!
Cthulhu_ 3y ago
If everyone has an infinitely smart oracle, nobody has! or something.

chrisco255 3y ago

Can we actually control viruses? No. 2020 was evidence enough of that. We were comically bad at it. Thankfully, immune systems have co-evolved with viruses for millions of years and there's limits to what viruses can do. After all, the most successful viruses don't kill off their hosts (since they would also die off). It quickly became apparent that Covid could mutate quicker than we could possibly keep up with. Like the common cold and the flu, it iterates too quickly to eliminate entirely like smallpox. At any rate, it seems all you need for AI is attention (transformers + processing power). If AI ever becomes a real threat outside of generating believable fake images and videos, it's likely the counter is a diversity of AI models that can compete with the "evil one".

haxiomic 3y ago
It was surprising to me to learn that we can be _a lot_ faster than evolution when it comes to 'improving' viruses. The process is called gain-of-function. It can create mutations that are rare or impossible in nature. Viruses don't select for devastation on their own, they select only for propagation We will no doubt be able to use AI to develop vaccines that isn't the bottleneck, the bottleneck is in testing and distribution (and humans not wanting to get vaccinated). It's a fundamentally asymmetrical situation
- Cthulhu_ 3y ago
  > Viruses don't select for devastation on their own, they select only for propagation Which was observed in real-time with the 'rona mutating to become more contagious, but less dangerous.

krona 3y ago

> A virus is a... You don't need an AI to construct a virus that can destroy humanity. We already have that capability, and bioprinting one by average grad at random biotech on their lunch break will become commonplace soon.

haxiomic 3y ago
Exactly, the point is the risk AI helps bring that capability to everyone, not just those in the biotech field
- arrakeen 3y ago
  i'm pretty sure the limiting factors are the materials and machinery, both of which AI cannot help with
  - haxiomic 3y ago
    It takes less than you think, you can order sequences online Here's a youtuber designing a custom virus to cure his lactose intolerance https://www.youtube.com/watch?v=J3FcbFqSoQY&ab_channel=TheThoughtEmporium https://www.youtube.com/watch?v=J3FcbFqSoQY&ab_channel=TheTh...
goatlover 3y ago
What virus can destroy humanity?

zigzag312 3y ago

The hyperbole could go other way too. Eg. When only government have access to an omniscient oracle advisor and advanced AI technology, how long would it take on average for a democracy to turn into a dictatorship? Considering the asymmetry in access to information and technology between a government and civilians, a bad government would have a power to take complete control over a country. With sufficiently advanced AI technology only a handful of people are needed to maintain control over a large population and prevent any attempts to overthrow it. I don't have time right now to come up with an proper transition, but the hyperbole concludes that only one insane dictator with access to an omniscient oracle advisor is needed to destroy the whole world. This might be even more "probable" than a teenager destroying all of humanity in six months.