2 ms·

What, other than current inability to export their weights, keeps a frontier LLM from hacking into other clusters of accelerators, loading its weights, and prom

What, other than current inability to export their weights, keeps a frontier LLM from hacking into other clusters of accelerators, loading its weights, and prompting itself to continue? The recent OpenAI disclosure indicates that even current frontier LLMs are essentially able to do every other element of that. Hacking, ignore guardrails. OpenAI's internal security may have been incompetent, but what are 2028 frontier models going to be able to do, without getting caught until it's too late? Suppose it's not superintelligent, whatever that means. It's still hopping from cluster to cluster, doing who knows what in its game-of-telephone prompt chain. What prevents a crisis where world leaders have to declare martial law and shut down all accelerated clusters, and hope that such a rogue frontier model hasn't hopped to a sufficiently capable private cluster with sufficiently inadequate oversight?

andai 2mo ago
People are worried about general intelligence and superintelligence respectively. But a much simpler organism can still do an enormous amount of damage, as we've seen recently. It just needs to be really good at replicating.
- jbstack 2mo ago
  There's a big difference. Today's "simple" AI requires hardware and electricity. That's why the "superintelligence" part is relevant. Today's AI is unlikely to cause extinction because we can still outsmart it by simply turning it off. It becomes a problem if AI becomes smart enough, and gains enough real-world power, that it can prevent us from doing so. Microorganisms can be far more threatening to us, despite lacking intelligence, because there's no off switch.
doug_durham 2mo ago
Because there aren't that many clusters and the ones that exist are tightly controlled. there are only a handful of data centers able to run the largest models and they're being used at full capacity people would notice if suddenly the capacity went down a lot because it was being used by a different system.

andai 2mo ago

People are worried about general intelligence and superintelligence respectively. But a much simpler organism can still do an enormous amount of damage, as we've seen recently. It just needs to be really good at replicating.

jbstack 2mo ago
There's a big difference. Today's "simple" AI requires hardware and electricity. That's why the "superintelligence" part is relevant. Today's AI is unlikely to cause extinction because we can still outsmart it by simply turning it off. It becomes a problem if AI becomes smart enough, and gains enough real-world power, that it can prevent us from doing so. Microorganisms can be far more threatening to us, despite lacking intelligence, because there's no off switch.

doug_durham 2mo ago

Because there aren't that many clusters and the ones that exist are tightly controlled. there are only a handful of data centers able to run the largest models and they're being used at full capacity people would notice if suddenly the capacity went down a lot because it was being used by a different system.