10 ms·

I wrote this in response to much of the hoopla around the 'threat' of A.I super-intelligence, and other unlikely disasters. The future of humanity as a species

by atroyn 11y ago

I wrote this in response to much of the hoopla around the 'threat' of A.I super-intelligence, and other unlikely disasters. The future of humanity as a species is worth thinking about, but we also ought to be considering the very real possibility of getting stuck in the awful conditions humans endured in the past, alongside more farfetched (and less likely) scenarios. I wish I had some unique idea about how to do that, but I suppose discussion is just as good.

javajosh 11y ago
I'm glad you're thinking about the long-term prospects of humanity. Your point is well-made that a technological regression, at this point, would be exceedingly difficult for future humans to reverse. If humans do not spread life beyond earth, then life will not survive long-term. It does feel like we must succeed in this first great technological expansion, or we might never succeed, and fall prey to Fermi's Demon.
- atroyn 11y ago
  I sort of left two additional open questions: - How long can the current era last? I.e. are we default-alive or default-dead? - Am I not thinking laterally enough about what pre-industrial civilizations could have done?
nickbauman 11y ago
Whether this particular scenario is plausible or not, the overarching concept behind it is path dependence. Path dependence implies that humankind may only have one chance to go through certain transformations. For instance, the Internet, begun in 1966, took nearly 30 years to become the behemoth force it is now. No single corporation could have pulled it off because of the timeline alone. Consider however that we've been doing this transformation thing since we began as a species. https://en.m.wikipedia.org/wiki/Path_dependence https://en.m.wikipedia.org/wiki/Path_dependence
mattmanser 11y ago
I think you haven't done even cursory research into A.I super-intelligence before dismissing it. Try reading Nick Bostrom's "Superintelligence: Paths, Dangers, Strategies"[1] and then I think you will change your mind. [1]http://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/0199678111 http://www.amazon.com/Superintelligence-Dangers-Strategies-N...
- atroyn 11y ago
  I don't dismiss it, I just believe it to be unlikely. I am in the middle of Bostrom's book right now.
VLM 11y ago
Consider that disasters scale fractally. You're much more likely to see a failed AI superintelligence destroy the one supplier of critical infrastructure product XYZ, than directly destroying the world. Its fractal self similar on a smaller scale of hubris, bad specification, bad planning, bad back out plans, poor estimation, bad competitive risk taking. The AI super intelligence that destroys our only global manufacturer of XYZ is very likely to be written in Excel in a spreadsheet convincing the execs that selling junk bonds and opening an ambitious new plant is a great idea, fast forward five years and humanity literally has no CPUs, or no flash memory, or no spark plug insulators. Its very unlikely to be black and white mad 1950s movie mad scientist writes bad LISP code that makes a humanoid-ish robot self aware making it crack open the smallpox vials intentionally. Its also worth considering that "we" are pretty good at destruction with existing dumb tools, it seems unlikely we'd need intelligent tools when simpler cheaper dumb tools cause equal chaos. It didn't take an AI to destroy HP, or Bell Labs, for example. Its statistically possible that the first AI super intelligence failure will instantly vaporize the planet in the same sense that its thermodynamically possible for all molecules in a box of air to randomly be found on one side of the box, however unlikely. In a similar way its very likely we'll have AI destroying individuals, companies, industries, markets, countries, long before it destroys the species.

by atroyn 11y ago

javajosh 11y ago

I'm glad you're thinking about the long-term prospects of humanity. Your point is well-made that a technological regression, at this point, would be exceedingly difficult for future humans to reverse. If humans do not spread life beyond earth, then life will not survive long-term. It does feel like we must succeed in this first great technological expansion, or we might never succeed, and fall prey to Fermi's Demon.

atroyn 11y ago
I sort of left two additional open questions: - How long can the current era last? I.e. are we default-alive or default-dead? - Am I not thinking laterally enough about what pre-industrial civilizations could have done?

nickbauman 11y ago

Whether this particular scenario is plausible or not, the overarching concept behind it is path dependence. Path dependence implies that humankind may only have one chance to go through certain transformations. For instance, the Internet, begun in 1966, took nearly 30 years to become the behemoth force it is now. No single corporation could have pulled it off because of the timeline alone. Consider however that we've been doing this transformation thing since we began as a species. https://en.m.wikipedia.org/wiki/Path_dependence https://en.m.wikipedia.org/wiki/Path_dependence

mattmanser 11y ago

I think you haven't done even cursory research into A.I super-intelligence before dismissing it. Try reading Nick Bostrom's "Superintelligence: Paths, Dangers, Strategies"[1] and then I think you will change your mind. [1]http://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/0199678111 http://www.amazon.com/Superintelligence-Dangers-Strategies-N...

atroyn 11y ago
I don't dismiss it, I just believe it to be unlikely. I am in the middle of Bostrom's book right now.

VLM 11y ago

Consider that disasters scale fractally. You're much more likely to see a failed AI superintelligence destroy the one supplier of critical infrastructure product XYZ, than directly destroying the world. Its fractal self similar on a smaller scale of hubris, bad specification, bad planning, bad back out plans, poor estimation, bad competitive risk taking. The AI super intelligence that destroys our only global manufacturer of XYZ is very likely to be written in Excel in a spreadsheet convincing the execs that selling junk bonds and opening an ambitious new plant is a great idea, fast forward five years and humanity literally has no CPUs, or no flash memory, or no spark plug insulators. Its very unlikely to be black and white mad 1950s movie mad scientist writes bad LISP code that makes a humanoid-ish robot self aware making it crack open the smallpox vials intentionally. Its also worth considering that "we" are pretty good at destruction with existing dumb tools, it seems unlikely we'd need intelligent tools when simpler cheaper dumb tools cause equal chaos. It didn't take an AI to destroy HP, or Bell Labs, for example. Its statistically possible that the first AI super intelligence failure will instantly vaporize the planet in the same sense that its thermodynamically possible for all molecules in a box of air to randomly be found on one side of the box, however unlikely. In a similar way its very likely we'll have AI destroying individuals, companies, industries, markets, countries, long before it destroys the species.