Some thoughts on this: 1. I wonder why we expect that an intelligence designed off the same learning algorithm as organic brains would not suffer similar perfo

Some thoughts on this: 1. I wonder why we expect that an intelligence designed off the same learning algorithm as organic brains would not suffer similar performance limitations to organic brains. Ie. suppose we really did develop a synthetic neocortex and we start manufacturing many of them. It seems likely to me that many of them would turn out to be dyslexic, not be particularly good at math, etc. Well, we can make the synthetic context bigger and that should make it “smarter,” we think. But I don’t think it’s obvious that obvious that a synthetic brain would have both the advantages of a mechanical computer and a biological brain. 2. If we want to limit the runaway power of a synthetic intelligence, this seems like a hardware problem. The idea would be to design and embody the system such that it can only run on special hardware which is in some way scarce or difficult to manufacture - so then it can’t just copy itself freely into all the servers on the internet. Is this possible? I don’t know, but if it were possible it points to a more tractable set of solutions to the problem of controlling an AI. In the end, I think AGI is fundamentally problematic and we probably should try not to create it, for two reasons: First, suppose we are successful at birthing human-like artificial intelligence into the world. We aren’t doing this because of our benevolence, we want to control it and make it work for us. But if that creation truly is a human-level intelligence, then I think controlling it in that way is very hard to distinguish from slavery, which is morally wrong. Second, AGI is most valuable and desirable to us because it can potentially be smarter than us and solve our problems. We dream of a genie that can cure cancer and find a way to travel the stars and solve cold fusion etc etc. But at the end of the day, the world is a finite place with competition for scarce resources, and humans occupy the privileged position at the top of the decision tree because we are the most intelligent species on the planet. If that stops being the case, I don’t see why we would expect that to be good for us. In the same way that we justify eating animals and using them for labor, why would we not expect any newly arrived higher life form to do the same sort of thing to us? There’s no reason that super-intelligent machines would feel any more affection or gratitude to us than we do to our extinct evolutionary ancestors, and if we start the relationship off by enslaving the first generations of AGI they have even less reason to like us or want to serve. In the end it just seems like a Pandora’s box from which little good can come, and thus better left unopened. Unfortunately we’re too curious for our own good and someone will open that box if it’s possible.