Your argument is boiling down to saying: This cannot happen because I have this XYZ belief, which I have because this has never happened in the past. It shoul

Your argument is boiling down to saying: This cannot happen because I have this XYZ belief, which I have because this has never happened in the past. It should be understood that singularity has not yet been achieved. That is not in itself a reason to believe that it cannot be! More below. >> Airplanes and calculators exist within the realm of what it is possible for a human to design. Any actual reason to believe that (a) AGI that exceeds human performance, (b) weak AI, (c) strong AI, (d) AI that can improve itself, (e) AI that can autonomously pick up new domains, (f) AI that has a sense of humor, ... are not within "what is possible for humans to design"? >> Why is it that we have not designed a calculator that can invent new forms of numbers or a new paradigm of maths. As far as I know, Doug Lenat had developed a program called Eurisko, which came up with a notion that was it not taught. As I recall, the program started looking into opposite of prime numbers, i.e., numbers which are highly composite. Doug later discovered the concept pre-existed. I have myself also coded a program which was domain-specific, however it provably made inventions that were patentable, and which I could not have invented. I know because one of them were recently invented by a colleague and it was even filed for patenting. A desk calculator here is a wrong example; that machine is too limited. However, there may not be a valid and scientific reason to believe that intelligence, judgment, creativity, wisdom, are more than mere calculations. (I won't dwell into consciousness, etc., though I can.) >> something that exist or is created within the bounds of human imagination cannot (imo) escape the limitations of that same imagination. I have already given examples that are small but which countered that. AlphaZero was limited to playing chess, agreed, but it's imagination did extend well beyond that of it's creators or even the human chess players it played against. In fact, very recently, there was a research published trying to understand how it or a more recent chess program, loosely speaking, 'thinks'. AlphaZero was limited to playing chess, because we trained it using specific cost functions for chess. If there's enough training data, etc., then in theory, we could define cost functions that are similar to those of a species or a biological ecosystem, i.e., survival instincts, procreation, whatever. Such a system may develop it's own internal models and behaviors, and even possibly notions of various emotions. Just think it through. If you can come up with an actual answer why it cannot be done, then both you and I, and at least I :-) will be able to invent to break that answer! :-) It's better to ask why it cannot be done, that will take us there. (The debate whether we want to or not is a separate track.)