Bengio is a certified and maximally informed genius, so my thoughts obviously deserve far more salt than his. But... The thrust of this argument is that centr

Bengio is a certified and maximally informed genius, so my thoughts obviously deserve far more salt than his. But... The thrust of this argument is that centralized authority -- perhaps uniquely centralized, as Bengio proposes an agreement between Russia, China and the USA as an example -- is the best safeguard against rogue AI. He goes as far as calling out open source development of AI as a contributing threat: > [if] the open-source community continues to play a leading role in the software development of LLMs, then it is likely that any hacker will have the ability to design their own pre-prompt (general instructions in natural language) on top of open-source pre-trained models... The concern is logical, but the solution is self-contradictory (as many intellectually honest explorations turn out): a section later, we are appropriately exploring nefarious corporations as examples of mis-aligned intelligences -- and how larger corporations may be even more adept at circumventing the rules. Whatever their philosophical foundations, our governments are also corporations (and so will be any new trans-national organization). They are fragmented into smaller groups with their own instrumental goals, and subject to the same risks of corruption that afflict private corporations, and the same risks of runaway optimization that may afflict AI. Appealing to them as a higher authority is at best a case of "the devil you know". Doomism always has a non-falsifiable counterargument: AI is new and might become arbitrarily more powerful than the most powerful organizations we've ever created ("super"-intelligence). For me this fails on several counts: we are still exploring the limits of power (and of corruption) that large organizations can achieve, progress in AI remains observable as long as it is public, and our primary experience with AI has been colonizing a frontier of capabilities rather than a sequence of new systems dominating at every task. To be clear, I do NOT personally conclude that we should avoid all regulation and oversight. But I think that constructing regulation around doomist theories serves disproportionately towards regulatory capture, and acts as a distraction from immediate issues of algorithmic inequality, disruption of political speech, and intentional weaponization of AI. Our most crucial long-term defense against most of these risks is a deep detente, where the tools to understand and develop AI remain broadly available, so that no one group feels it has an unbeatable advantage in the powers afforded by AI. This is what the open-source community is currently providing. Locking down public research in AI in order to prevent superintelligence would have massive unintended consequences.