3 ms·
If you're suggesting the emergence of an AI that can exponentially self-improve, coopt the resources to overwhelem human confrol, and magically build out the ph
by swatcoder 3y ago
If you're suggesting the emergence of an AI that can exponentially self-improve, coopt the resources to overwhelem human confrol, and magically build out the physical infrastructure sufficient to drive its globe-spanning humanity-squelching computational needs... that's a foom scenario.
If something that some people call AGI emerges in the next five years, (a) it's designation is going to be under tremendous dispute, (b) the infrastructure to enable a nightmare scenario won't be anywhere near reality.
Escaping that practical reality requires a complete ahistorical, ascientific faith in "yeah, but what if it's that capable?!?! The point is we can't know!!" that more strongly recalls history's secretive, sometimes dangerous, perennially incorrect apocolypse cults and alchemical societies than it does the diligent and reasoned scientists we might usually think of handing hundreds of billions of dollars.
- theptip 3y agoFoom specifically refers to a very fast takeoff to ASI, usually hours/days or at the outside weeks. It’s a term of art, well-defined. https://www.lesswrong.com/posts/LF3DDZ67knxuyadbm/contra-yudkowsky-on-doom-from-foom-2 https://www.lesswrong.com/posts/LF3DDZ67knxuyadbm/contra-yud... It does not refer to a fast takeoff where an AGI self-improves to ASI over multiple years. It’s also not referring to the AGI threshold. If foom happens it’s probably unambiguous; there is now an entity that is far more intelligent than humans. I think the foom scenarios are fairly unrealistic, for thermodynamic reasons. But I think it’s perfectly plausible that an ASI could persuade the company that built it to keep it secret while it acquired more resources and wealth over the course of years.