Look at where we are now with ML, and ask what we would need to do to "accidentally" create HGI. That title's wording is clickbait, not an argument. Mathematic

Look at where we are now with ML, and ask what we would need to do to "accidentally" create HGI. That title's wording is clickbait, not an argument. Mathematicians know exactly why neural networks work – because they contain a model of the data, and they try to develop ways to build (train) those models so that they do it better. The model may not have human concepts associated with it, but that doesn't mean it's somehow "out of control". In order to achieve HGI, one would have to train models of quite a few domains (the same way schools create models in people's minds), as well as tons of ethical and practical heuristics to weigh an infinite number of reasons for or against doing something.