AGI isn't really about artificial humans. It's about very good general problem solving software that's way beyond the capabilities of humans while not being al

AGI isn't really about artificial humans. It's about very good general problem solving software that's way beyond the capabilities of humans while not being aligned with human interests. Not because the software is evil, but because aligning values is an unsolved problem (humans aren't even totally aligned - and values also change). If you have an intelligence that's very good at achieving its goal and you don't have a good way to align its goal with human goals, you can very quickly get into trouble if that intelligence thinks much faster than you do. https://www.lesswrong.com/posts/mMBTPTjRbsrqbSkZE/sorting-pebbles-into-correct-heaps https://www.lesswrong.com/posts/mMBTPTjRbsrqbSkZE/sorting-pe... https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden-complexity-of-wishes https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden... https://www.lesswrong.com/posts/BEtzRE2M5m9YEAQpX/there-s-no-fire-alarm-for-artificial-general-intelligence https://www.lesswrong.com/posts/BEtzRE2M5m9YEAQpX/there-s-no... https://www.lesswrong.com/posts/5wMcKNAwB6X4mp9og/that-alien-message https://www.lesswrong.com/posts/5wMcKNAwB6X4mp9og/that-alien...