The argument falls apart when considering that compute is incredibly fungible and performing a 51% attack only means finding a zero-day that lets an AGI copy it

The argument falls apart when considering that compute is incredibly fungible and performing a 51% attack only means finding a zero-day that lets an AGI copy its code or neural net to a large percentage of computers. Much like wealth accumulates in the absence of antitrust law and taxation, intelligence accumulates in the absence of some external controls. EDIT: And probably all you need is natural selection. No one is running GPT-2 anymore except for basic research. Any AGI with marginal advantage over others will be run instead of them. At some level of complexity identical agents will coordinate as one entity with shared goals.