3 ms·

I don't think it's impossible to come up with arguments against them. For one thing, extrapolating the current gradual rate of progress forwards means that we w

I don't think it's impossible to come up with arguments against them. For one thing, extrapolating the current gradual rate of progress forwards means that we will get to see several minor "intelligence spills" before the hypothetical big and last one, and by observing what went wrong, humanity will have the opportunity to come up with solutions. Since very smart human beings have in history done a lot of damage, but never ended the species or anything, we can conclude that the world has an "intelligence safety margin" that extends up to Alexander the Great or Napoleon. There have been some incredibly smart people in human history and none of them have ruined everything, a few countries at most.

Vecr 4y ago
I'm pretty sure they have stock arguments against what you said, even from people like Nick Bostrom who I think have a bit better handle on the situation. In that vein though, I think making the world more robust in general might be a way forward. Having multiple broad spectrum anti-virals for every class of virus ready to go and stockpiled, having flexible on-shore manufacturing infrastructure, with supply chains that are on the same continent or at least have massive component stockpiling on national levels, and having smart and vigilant people running these systems from the very top down to the janitors and security guards. Make a world where the machine stopping because a few agents, human or otherwise, doing something unexpected, much less likely. At least to the point EAs can start buying mosquito nets again.
arisAlexis 4y ago
something never happening before == can't happen is extremely dangerous attittude. I suggest you read Taleb and his duck example.
- Quarrelsome 4y ago
  perhaps its more about "worry about the issue when it _can_ happen" as opposed to planning for the expiry of the sun today.
ShredKazoo 4y ago
>I don't think it's impossible to come up with arguments against them. For one thing, extrapolating the current gradual rate of progress forwards means that we will get to see several minor "intelligence spills" before the hypothetical big and last one, and by observing what went wrong, humanity will have the opportunity to come up with solutions. This has been discussed under "warning shots" https://www.lesswrong.com/posts/idipkijjz5PoxAwju/warning-shots-probably-wouldn-t-change-the-picture-much https://www.lesswrong.com/posts/idipkijjz5PoxAwju/warning-sh...
- whatshisface 4y ago
  That blog post makes some giant unsupported assumptions, like that the reason for the failure of a few activists to stop gain-of-function research was that the government is fundamentally bad at addressing risks, and not that biologists might know more about the risk/benefit tradeoff than amateurs.
  - ShredKazoo 4y ago
    Was there even a serious policy discussion of banning gain-of-function research? Gain of function research is supposed to help with pandemics, right? Did it help with COVID?
    - whatshisface 4y ago
      I don't think I know enough to evaluate the case around gain of function research, all I know is that the blog post didn't make any effort to.
    - fidgewidge 4y ago
      GoF research was banned in the USA for long periods, that's why Fauci was outsourcing it to China and why DARPA refused to find that project. No, not only has GoF not helped with COVID and probably caused it, but that's also true of the entire field of virology.
mitthrowaway2 4y ago
Alexander the Great or Napoleon have been limited in how much damage they can do, because they're human too. They don't live for hundreds of years, they can't copy their brains to new hardware, and they can't modify their own minds to remove that pesky conscience when it slows them down. Perhaps most importantly, Napoleon's happiness, power, and continued survival depends on having a civilization of surviving humans working under his command, whereas that might only temporarily be true for AI. At best, history serves as a counterexample to the idea that if an AI goes bad the attendants would just unplug it, seeing how often it is that dictators don't get stabbed by their aides as soon as they start causing mass deaths, instead often receiving broad popular support as the world burns.

Vecr 4y ago

I'm pretty sure they have stock arguments against what you said, even from people like Nick Bostrom who I think have a bit better handle on the situation. In that vein though, I think making the world more robust in general might be a way forward. Having multiple broad spectrum anti-virals for every class of virus ready to go and stockpiled, having flexible on-shore manufacturing infrastructure, with supply chains that are on the same continent or at least have massive component stockpiling on national levels, and having smart and vigilant people running these systems from the very top down to the janitors and security guards. Make a world where the machine stopping because a few agents, human or otherwise, doing something unexpected, much less likely. At least to the point EAs can start buying mosquito nets again.

arisAlexis 4y ago

something never happening before == can't happen is extremely dangerous attittude. I suggest you read Taleb and his duck example.

Quarrelsome 4y ago
perhaps its more about "worry about the issue when it _can_ happen" as opposed to planning for the expiry of the sun today.

ShredKazoo 4y ago

>I don't think it's impossible to come up with arguments against them. For one thing, extrapolating the current gradual rate of progress forwards means that we will get to see several minor "intelligence spills" before the hypothetical big and last one, and by observing what went wrong, humanity will have the opportunity to come up with solutions. This has been discussed under "warning shots" https://www.lesswrong.com/posts/idipkijjz5PoxAwju/warning-shots-probably-wouldn-t-change-the-picture-much https://www.lesswrong.com/posts/idipkijjz5PoxAwju/warning-sh...

whatshisface 4y ago
That blog post makes some giant unsupported assumptions, like that the reason for the failure of a few activists to stop gain-of-function research was that the government is fundamentally bad at addressing risks, and not that biologists might know more about the risk/benefit tradeoff than amateurs.
- ShredKazoo 4y ago
  Was there even a serious policy discussion of banning gain-of-function research? Gain of function research is supposed to help with pandemics, right? Did it help with COVID?
  - whatshisface 4y ago
    I don't think I know enough to evaluate the case around gain of function research, all I know is that the blog post didn't make any effort to.
  - fidgewidge 4y ago
    GoF research was banned in the USA for long periods, that's why Fauci was outsourcing it to China and why DARPA refused to find that project. No, not only has GoF not helped with COVID and probably caused it, but that's also true of the entire field of virology.

mitthrowaway2 4y ago

Alexander the Great or Napoleon have been limited in how much damage they can do, because they're human too. They don't live for hundreds of years, they can't copy their brains to new hardware, and they can't modify their own minds to remove that pesky conscience when it slows them down. Perhaps most importantly, Napoleon's happiness, power, and continued survival depends on having a civilization of surviving humans working under his command, whereas that might only temporarily be true for AI. At best, history serves as a counterexample to the idea that if an AI goes bad the attendants would just unplug it, seeing how often it is that dictators don't get stabbed by their aides as soon as they start causing mass deaths, instead often receiving broad popular support as the world burns.