What is the tangible threat of AGI though? Let's do a thought experiment. I have the world's first AGI installed on my computer, named ERNIE, running in llama.

What is the tangible threat of AGI though? Let's do a thought experiment. I have the world's first AGI installed on my computer, named ERNIE, running in llama.cpp. This AGI is infinitely intelligent, confidently moreso than any living human. It can spit out text at 100 tokens/second and encode entire books in less than a minute. What does this AI do, then? Ostensibly nothing. It can output the entire Library of Babel for all it cares, but it's not harmful until I put it in control of a system. You could argue that a multimodal model has different ways of interacting with the world, but it's still a computer. All of it's actions and outputs are quantized as static data that is either encoded as text or some other significant representation. It inherantly does nothing, and if you ascribe power-seeking behavior to it then it's ultimately limited by the runtime you provide. Providing an overly dangerous runtime has been considered developer-error since 1995. So - to prevent AGI from being shitty and ruining everything, compel human operators to not allow them to be shitty and ruin everything. Like how we punish people that let their kindergartner control a construction crane.