Although I agree with many points (maybe too many!) this author raises and can always be counted in for a bit of 'Bostrom bashing', I find the argument for the

Although I agree with many points (maybe too many!) this author raises and can always be counted in for a bit of 'Bostrom bashing', I find the argument for the emergence superintelligence (aka AI 'singularity') overall fairly convincing. The biggest hole in it is the assumption thatthe implementation of an intelligence at level X - say, roughly the human level - can be scaled up to a much higher level. We know from complexity theory that this is not necessarily true. The opposite is more likely for a complex algorithm like that. However, there is a pressing issue that the superintelligence thought experiment also raises. A colleague of mine from AI called it the value alignment problem: How do we make sure that an AI's values are sufficiently aligned with human values? The problem with this are the human values. As it turns out at a closer look, we cannot even agree on what structure these have and there is substantial disagreement among 'experts' about what human values are or what they should be. For example, as surprising as this may sound to some of you, there is substantial disagreement among philosophers of value whether 'better than' is transitive or not. There are good arguments for and good arguments against the transitivity of overall 'better than', and that's just the tip of the iceberg. To give another example, systems of law have been studied extensively and one might try to formalize them in input/output logics or normative systems, but any closer look at real systems of laws quickly reveals that their specification is incomplete and that there are many inconsistencies in them. These inconsistencies could be modelled, of course, but the big question is whether they are features or deficiencies. Again, you will find all kinds of positions among legal scholars. In a nutshell, we don't even know how to adequately formalize the form of human values from a normative perspective, and even if we could, we would substantially disagree about their content. However, just describing human values cannot possibly solve the value alignment problem in a satisfying way, because humans have waged wars, committed genocide, mass killings, etc. Therefore, without a reasonable theory of normative values and their structure upon which we can somehow agree, the value alignment problem is underdetermined and cannot even be tackled. The underlying problem is that human values aren't really aligned either, of course. As a consequence, we already now have completely different sets of values being applied and incorporated into AI software, depending on who develops it. This will become a serious ethical problem in the future that requires political solutions sooner than later. That's why the controversy about the hypothetical superintelligence is actually quite beneficial, even if the argument itself is no less shaky than Bostrom's Simulation Argument.