3 ms·

For those interested in another very different analysis of the evidence for and against superintelligence being a worthwhile concern, here's Holden Karnofsky, e

For those interested in another very different analysis of the evidence for and against superintelligence being a worthwhile concern, here's Holden Karnofsky, executive director of GiveWell and the related Open Philanthropy Project: - Up until 2016, Holden held the opinion that it was not a worthwhile concern. He describes why he changed his mind here: http://www.openphilanthropy.org/blog/three-key-issues-ive-changed-my-mind-about#Changing_my_mind_about_potential_risks_from_advanced_artificial_intelligence http://www.openphilanthropy.org/blog/three-key-issues-ive-ch... - And he outlines his current arguments in favor of treating it as a significant concern here: http://www.openphilanthropy.org/blog/potential-risks-advanced-artificial-intelligence-philanthropic-opportunity http://www.openphilanthropy.org/blog/potential-risks-advance... In that lengthy write-up, he addresses both the shorter-term risks from things like misuse of non-superintelligent AI, along with longer-term risks from superintelligent AI.

mcguire 10y ago
What does he suggest doing about it? That's always my question for this sort of topic. It seems to me that the only reasonable behavior for someone accepting SAI as a legitimate concern is to advocate defunding all AI researchers, as well as criminal punishments for AI research.
- blakeweb 10y ago
  There's a section on Tractability--he can speak to his opinions better than I can: http://www.openphilanthropy.org/blog/potential-risks-advanced-artificial-intelligence-philanthropic-opportunity#Tractability http://www.openphilanthropy.org/blog/potential-risks-advance... TL;DR: He talks about several avenues of technical and strategy research that seem plausibly very useful and are not currently being pursued by more than a handful of people in the world. Many of these currently-engaged people are precisely the folks the author of this post disparages for being weird or insular. One of the avenues of technical research he mentions is "transparency, understandability, and robustness against or at least detection of large changes in input distribution" for AI/ML systems. In other words, technical research to produce methods capable of reducing the likelihood of advanced systems behaving in severely bad, unexpected ways.

mcguire 10y ago

What does he suggest doing about it? That's always my question for this sort of topic. It seems to me that the only reasonable behavior for someone accepting SAI as a legitimate concern is to advocate defunding all AI researchers, as well as criminal punishments for AI research.

blakeweb 10y ago
There's a section on Tractability--he can speak to his opinions better than I can: http://www.openphilanthropy.org/blog/potential-risks-advanced-artificial-intelligence-philanthropic-opportunity#Tractability http://www.openphilanthropy.org/blog/potential-risks-advance... TL;DR: He talks about several avenues of technical and strategy research that seem plausibly very useful and are not currently being pursued by more than a handful of people in the world. Many of these currently-engaged people are precisely the folks the author of this post disparages for being weird or insular. One of the avenues of technical research he mentions is "transparency, understandability, and robustness against or at least detection of large changes in input distribution" for AI/ML systems. In other words, technical research to produce methods capable of reducing the likelihood of advanced systems behaving in severely bad, unexpected ways.