5 ms·

A virus can wipe out significant portions of the human population with so little complexity we don't even consider it life, and with no understanding of what it

A virus can wipe out significant portions of the human population with so little complexity we don't even consider it life, and with no understanding of what it's doing. There are systems on the planet in which a single circuit activation has the potential to turn everything into radioactive soup via a cascade of responses. The near term risk isn't super intelligence, it's the paper-clip optimizer pushing the wrong button.

avgcorrection 3y ago
> The near term risk isn't super intelligence, it's the paper-clip optimizer pushing the wrong button. This has already happened. The paper clip optimizer was capitalism. Just listen to some heatwave weather forecasts once in a while (as an example).
- staunton 3y ago
  There may be more than one thing. Capitalism as an information processing system is quite slow. Computers can be faster.
  - avgcorrection 3y ago
    “Already happened” merely means that it is not a hypothetical scenario any more.
- lukifer 3y ago
  To anyone who says we can just flip the off switch on AIs who go rogue: where exactly is the off switch to capitalism, or corporations, or global supply chains, or industrialized agriculture, or social media, or the State? AIs aren't a new phenomenon: we've been co-evolving with them for millennia, for both good and ill. The new more powerful iterations ("vast arrays of inscrutable floating-point matrices") are nested in those previous memetic life forms (corporations in particular [0]); and once something becomes infrastructure (and/or an income stream to some subset of humanity), it becomes non-trivial, if not impossible, to turn off. [0] https://omniorthogonal.blogspot.com/2013/02/hostile-ai-youre-soaking-in-it.html https://omniorthogonal.blogspot.com/2013/02/hostile-ai-youre...
c_crank 3y ago
Paper clip optimizers always need a secondary (and often dubious) physical element to pose an actual threat, like grey goo or nanomachines or strange matter.
- NoRelToEmber 3y ago
  Or a genetics lab hooked up to a computer...
  - sroussey 3y ago
    They all are. A quite a bit of AI there too.
- ben_w 3y ago
  The secondary element can be humans. Even if it only gets as far as running a Ponzi or pyramid scheme — and there are plenty of those where the scheme itself is a small block of text that convinces those reading it to (1) share it, and (2) give money to the person who first showed them this message — still breaks economics bad enough to matter, even if you don't give any credence at all to Yudkowsky etc. Or in a more literal sense of using humans, as each separate component of ordering the creation and delivery of a custom (non-computer) virus has been demonstrated, and the only good news is the bio-labs say no to printing the D/RNA for dangerous viruses and so do the LLMs that most noobs try to use.
  - c_crank 3y ago
    Pyramid schemes are a nuisance. Covid-19 should have been a wake up call that most bio labs ought to be closely re-examined or closed down, regardless of whether an AI happens to be abusing them. The bio-lab scenario is the only one I see as being an actually plausible concern, though.
- haswell 3y ago
  I think paper clip optimizers are already here in the form of advertising and engagement algorithms, and the physical element is us.

avgcorrection 3y ago

> The near term risk isn't super intelligence, it's the paper-clip optimizer pushing the wrong button. This has already happened. The paper clip optimizer was capitalism. Just listen to some heatwave weather forecasts once in a while (as an example).

staunton 3y ago
There may be more than one thing. Capitalism as an information processing system is quite slow. Computers can be faster.
- avgcorrection 3y ago
  “Already happened” merely means that it is not a hypothetical scenario any more.
lukifer 3y ago
To anyone who says we can just flip the off switch on AIs who go rogue: where exactly is the off switch to capitalism, or corporations, or global supply chains, or industrialized agriculture, or social media, or the State? AIs aren't a new phenomenon: we've been co-evolving with them for millennia, for both good and ill. The new more powerful iterations ("vast arrays of inscrutable floating-point matrices") are nested in those previous memetic life forms (corporations in particular [0]); and once something becomes infrastructure (and/or an income stream to some subset of humanity), it becomes non-trivial, if not impossible, to turn off. [0] https://omniorthogonal.blogspot.com/2013/02/hostile-ai-youre-soaking-in-it.html https://omniorthogonal.blogspot.com/2013/02/hostile-ai-youre...

c_crank 3y ago

Paper clip optimizers always need a secondary (and often dubious) physical element to pose an actual threat, like grey goo or nanomachines or strange matter.

NoRelToEmber 3y ago
Or a genetics lab hooked up to a computer...
- sroussey 3y ago
  They all are. A quite a bit of AI there too.
ben_w 3y ago
The secondary element can be humans. Even if it only gets as far as running a Ponzi or pyramid scheme — and there are plenty of those where the scheme itself is a small block of text that convinces those reading it to (1) share it, and (2) give money to the person who first showed them this message — still breaks economics bad enough to matter, even if you don't give any credence at all to Yudkowsky etc. Or in a more literal sense of using humans, as each separate component of ordering the creation and delivery of a custom (non-computer) virus has been demonstrated, and the only good news is the bio-labs say no to printing the D/RNA for dangerous viruses and so do the LLMs that most noobs try to use.
- c_crank 3y ago
  Pyramid schemes are a nuisance. Covid-19 should have been a wake up call that most bio labs ought to be closely re-examined or closed down, regardless of whether an AI happens to be abusing them. The bio-lab scenario is the only one I see as being an actually plausible concern, though.
haswell 3y ago
I think paper clip optimizers are already here in the form of advertising and engagement algorithms, and the physical element is us.