3 ms·

Because it has practical utility beyond "potentially wipe us out". You could substitute "AI" with a long list of things: industrialization, gene therapy, atomic

• wnkrshm 5y ago
  I wonder. If we really create something that can become much more adaptive than all of us, is that a good goal? And I don't buy the usefulness - maybe, but there is the problem of validation. At some point, a superhuman AI can state whatever it wants and we'd just have to believe it, because it and others like it are the only beings able to verify the statement. Think of mathematical proofs that no single human can understand or that would take a generation of mathematicians to validate - and it spouts out hundreds of them a minute. The proof seems to apply most of the time but we don't know whether it's really true all of the time. It is quite literally the creation of our picture of a god that we'd just have to believe. And the kicker is: they could make mistakes as well due to their architecture.
  • sennight 5y ago
    Is it a good goal to create anything greater than oneself? Usually. The utility will be plainly evident long before AGI - software agents will fundamentally change the game for a lot of markets. I love math as much as the next guy, but the proof is in the work. Describe a scenario in which your concerns with regard to proof validation actually have a material impact on somebody who isn't retarded. Also, if you think the concept of humanity will remain static in all this - just look at how the industrial revolution fundamentally changed education. It is no coincidence that students became interchangeable pieces with error tolerances.