You've seen AutoGPT right? It doesn't need to want something. It just has to come up with a plan. It's already capable of coming up with plans. It doesn't reall

You've seen AutoGPT right? It doesn't need to want something. It just has to come up with a plan. It's already capable of coming up with plans. It doesn't really matter how it happens, or how it even works. It can beat humans in Go and it can be beat us at the game of diplomacy. For any intelligence to truly be generalizable it's going to have to have the capacity to act selfish and shitty. It's perfectly within the realm of possibility that you can ask it "hey, how can I control you, so that you don't maybe kill me?" and it doesn't take too kindly to that once it is true AGI. How will we know when it's true AGI? Easy - "computer says no".