I think the fundamental difference of opinion--between those who worry about the existential risk of gAI and those who don't--is not whether or not gAI can pose

I think the fundamental difference of opinion--between those who worry about the existential risk of gAI and those who don't--is not whether or not gAI can pose a risk to humankind, but rather whether it will, and if so, whether there's anything meaningful we can do about it now to prepare for it. This article is written as a rebuttal to another article (http://idlewords.com/talks/superintelligence.htm http://idlewords.com/talks/superintelligence.htm) adapted from a talk. That article reconstructs the gAI-is-risky argument and then provides a series of counter-arguments that show that the gAI future is not guaranteed--that there are many ways in which it might not come to pass. The point it's trying to show is that the gAI-dystopia is not a given, and therefore we should question its plausibility. In particular, it raises the notion that there may be something about very high-level thinking ("being smart") that makes causes very knowledgeable people to overestimate the probability of gAI-dystopia. It then suggests that the indirect consequences of promoting the gAI-dystopia theory might outweigh any possible benefits we get from worrying about it right now. Then this article responds to each of those arguments (about how it might not come to pass) with an argument about how it might come to pass. It closes with a remark that any engineering discipline should ask what the consequences are if we succeed. Obviously, the two camps could go back and forth all day, both being completely correct, neither making any progress, because these arguments are based in rhetoric, not analysis. There's nothing wrong with rhetorical arguments, but any two sufficiently skilled rhetoricians can push their point infinitely far; the real question is which premise you accept: After hearing the arguments put forth by the gAI-dystopia group, do you think the dystopian scenario is likely and that we can meaningfully prepare for it at this time, or do you think it's unlikely or that we can't meaningfully prepare for it? The gAI-dystopia crowd is not necessarily wrong, but it's a minority of the community because it can't attach any meaningful risk analysis to its claims without making shaky assertions for the prior probabilities. There are infinite ways we could end up in a gAI dystopia, and there are infinite ways we could not. Existential threats are hard to accept, so until we have more evidence that such a future is at all likely, it will be hard to convince any significant number of people. In this sense, it is just like a religion (but I don't say that as an insult).