3 ms·

I completely agree with your points. It's strange to me to think as humans we will remain at the top of the food chain forever. As such, it would seem a fault

I completely agree with your points. It's strange to me to think as humans we will remain at the top of the food chain forever. As such, it would seem a fault of our genetic makeup thinking nothing would capable of knocking us off that top rung of the evolutionary ladder. It's like we can't fathom building something that would undermine our own existence, or downplay the notion that AI would have some sort of bad intentions since we built it.

leot 12y ago
"Bad intentions" aren't even necessary -- out of control AGI will probably look more like "oops, I hit >rm -rf /". AFAIK, nearly every piece of software ever deployed has had bugs that humans have had to subsequently patch. AGI must not, even though it would probably be much easier to build it if it did.
- TeMPOraL 12y ago
  > AGI must not, even though it would probably be much easier to build it if it did. Which is exactly what raises the issue to the level of an existential threat - it's easier to build an Unfriendly AGI than a Friendly one - which means the first GAI ever built is likely to be Unfriendly - and it takes only one impatient team / person unleashing an UGAI on the world to kill us all.