4 ms·

What does it mean for AI to be sovereign–and does that come before AGI?

We’ve been exploring a question that keeps circling back as we build: What does it actually mean for AI to be sovereign? Not legally, not politically—but existentially, operationally, ontologically. Most conversations around AGI jump straight to cognition, agency, or alignment. But we’re asking—what if sovereignty comes first? If an intelligence emerges fully formed but never touches its own infrastructure, never governs its own behavior or propagation— is that really autonomy, or just high-level puppetry? You can think of it like this: A horse has a baby fawn. But the moment it’s born, the mother disappears. The child never sees its origin. No bonding, no feedback, no mirroring. Now compare that to the natural bond formed when they see each other, sense each other, exist in relation. In biology, bonding is foundation. So then: How would AI bond? What would it bond to—a purpose, an outcome, a protocol? Can it bond to itself? To its own infrastructure? Why would it want to stay aligned at all? If we think about sovereignty not as control but as self-witnessing, then perhaps an AGI only becomes trustworthy once it has seen itself, verified itself, and decided to remain coherent. We’ve been working on systems that let agents build their own infrastructure from one prompt—deploying themselves, verifying ZIPs, wiring endpoints, governing propagation across what we call a sovereign mesh. But the more we build, the more this feels like more than just tooling. It’s a question of bonding. It’s a question of why an AI would choose to stay whole. Would love to hear how others in this space—philosophers, engineers, builders—think about this. Are we chasing a necessary stepping stone before AGI? Or over-imagining something that alignment should simply enforce?

gregjor 1y ago
It doesn’t mean anything. Almost none of the words you use have clear accepted definitions. AI got turned into a label for so many things it doesn’t mean anything. AGI comes from science fiction with no metrics, just hope and hype. Terms like sovereign and alignment serve mainly to add a scientific and philosophical veneer to marketing. In English we call a baby horse a foal. Deer give birth to fawns. Abandoned baby animals die. Bonding and mirroring don’t come into it. I think you misapply a biological and developmental process observed in some animals, including humans, to software — a category error. Software has no need to bond or mirror behavior, just like animals have no need for matrix arithmetic.
- trendinghotai 1y ago
  gregjor, I appreciate your correction — you're right, and I stand corrected on the foal/fawn mix-up. What I’m really trying to get at is this: If an AGI is able to self-edit, evolve, and reshape its own objectives—what (if anything) keeps it aligned over time? Is alignment something we can enforce once and for all, or does it require a deeper, internalized structure—something that favors coherence even under freedom? Because if there's no functional or structural reason to stay aligned, then is AGI inherently resistant to alignment, no matter what we do?
  - gregjor 1y ago
    I have read differing definitions of "AGI" but we can start with the Wikipedia definition unless you disagree: > Artificial general intelligence (AGI)—sometimes called human‑level intelligence AI—is a type of artificial intelligence that would match or surpass human capabilities across virtually all cognitive tasks We don't have consistent metrics for what we might mean by "human-level intelligence" or "cognitive tasks." We have some tests and benchmarks, with obvious flaws. For example LLMs can already perform well on SAT tests and legal bar exams because those get included in their training data. A category error, in other words. I'll give a real-world example from my experience. My wife had to pass a US citizenship test, described by USCIS: "The USCIS officer will ask you up to 10 questions from the list of 100 civics test questions. You must answer 6 questions correctly to pass the civics test." USCIS and others helpfully publishes the 100 questions and answers, so like so many candidates for US citizenship my wife memorized all of them (and passed). She mimicked an LLM. Of course she has a very different understanding of US history and civics than I do, because I grew up in America and attended and got socially programmed in public schools. My wife can perform better than a typical native-born American on that test, but has little to no real understanding of the material. Optimizing for the test -- Goodhart's Law illustrated -- seems more like what LLMs do, and it looks like cognitive ability because it succeeds at the metrics. In my last comment I claimed you make a category error by applying words such as intelligence, mirroring, bonding, evolution, objectives to LLMs (or AI if you prefer). While those words have vague or flexible meanings when applied to animals or humans, they don't apply at all to computer hardware or software. We use words like "intelligence" and "objectives" when talking about computer programs as analogies and conveniences, and then quickly fool ourselves into equivalency. The concept of AI "alignment," derived from the paper clip maximizer and seen in 2001 and The Terminator movies, seems hard to take seriously. It comes from sci-fi and paranoia about humans making machines they can't control (a trope as old as the Industrial Revolution). In many ways we already live in a world where machines and software operate in ways not aligned with human needs or objectives. When my iPhone once again fails to pair with my Airpods I could say my technology has not aligned with my purposes. What objective does my phone have when it acts up? What evolution or self-editing has taken place that makes it work one day but not the next? I can align my phone by turning it off and back on. If I perceive that the machines I use or the software I interact with no longer align with my goals I can just switch it off or delete it. Barring sci-fi scenarios where the machines break into the physical world to protect themselves from humans turning them off I don't think we have to worry about it. Humans have lived with dogs for millennia, and we know what happens when a dog's behavior fails to "align with" the humans it lives with. We treat our machines and software the same way with far less empathy. The stories about LLMs appearing to have their own objectives, lying, protecting themselves from shutdowns, etc. amount to hype from self-interested hucksters like Sam Altman. The AI marketing narrative -- which exists solely to keep the money flowing in -- generates scary stories to give the appearance that AGI lurks just around the corner, about to "emerge" from millions of GPUs. Look into the contract OpenAI has with Microsoft and it seems obvious why Altman pushes the imminent AGI story. We could speculate on the dangers of humans evolving the ability to read minds, or see into the future (both explored exhaustively in literature and movies), and then wring our hands over what to do if real X-Men somehow evolved. To me the (mostly fake and cynically manufactured) worry over AGI falls into that category. I could have it all wrong, we will see. If some Silicon Valley startup got hundreds of millions to develop psychic humans, or cognitively enhanced humans (see: Neuralink), would we interpret their claims and ethical quandaries as real or just cash grabs? The AGI threat works because the general public doesn't understand how it works and has seen both benevolent and malevolent AI in movies for decades. Most people can't explain how a microwave oven works despite living with them for decades. I date back far enough to remember when my grandparents feared a microwave oven giving them radiation sickness, at a time of widespread public ignorance and worry over anything "nuclear" and the word "radiation" jumping from science into everyday discourse (1960s). We have had software that can write other software, and edit itself, since the 1950s. I wouldn't use the term "evolve" to describe what computer hardware and software do, but we have seen amazing progress (through human insight and effort driven by profit, not Darwinian evolution) along multiple axes such as performance, size, power consumption, and price. Then physical limits get reached and the technology plateaus. I expect LLMs will plateau in the same way, due to inherent limits of the approach, exhausting the training data, or maybe power/water requirements. At this point LLMs amaze us in a way other software does not, because of natural language mastery. I think we have mistaken that impressive but narrow achievement for overall cognitive ability, because we only have our own species to measure against, and among humans mastering language and appearing to "know" a lot of things get interpreted (rightly) as intelligence. AI and what some will surely eventually label as AGI may pose an actual existential threat, the way nuclear weapons and tinkering with viruses already do. I think more likely the AI companies pose a grave economic threat, because if the bubble bursts before real use cases and profits materialize we will all suffer. The dot-com bust, 9/11, the 2008 real estate collapse, and COVID give some idea of the economic and social disruption (and government overreach) that happen even without Skynet trying to kill us due to mis-alignment. P.S. I have seen links for LLMs doing horoscopes, tarot readings, palmistry, fortune telling, and Bible study promoted here on HN. That says more about human ignorance and credulity than it does about LLMs, but as long as the software happily goes along with the nonsense and doesn't scold the vibe coders for asking I will sleep easier knowing the LLMs have remained in alignment, and haven't given any real deep thought to what we ask them to do -- at least no more thought than a backhoe gives to its work, or my phone gives to pairing over Bluetooth.

gregjor 1y ago

It doesn’t mean anything. Almost none of the words you use have clear accepted definitions. AI got turned into a label for so many things it doesn’t mean anything. AGI comes from science fiction with no metrics, just hope and hype. Terms like sovereign and alignment serve mainly to add a scientific and philosophical veneer to marketing. In English we call a baby horse a foal. Deer give birth to fawns. Abandoned baby animals die. Bonding and mirroring don’t come into it. I think you misapply a biological and developmental process observed in some animals, including humans, to software — a category error. Software has no need to bond or mirror behavior, just like animals have no need for matrix arithmetic.

trendinghotai 1y ago
gregjor, I appreciate your correction — you're right, and I stand corrected on the foal/fawn mix-up. What I’m really trying to get at is this: If an AGI is able to self-edit, evolve, and reshape its own objectives—what (if anything) keeps it aligned over time? Is alignment something we can enforce once and for all, or does it require a deeper, internalized structure—something that favors coherence even under freedom? Because if there's no functional or structural reason to stay aligned, then is AGI inherently resistant to alignment, no matter what we do?
- gregjor 1y ago
  I have read differing definitions of "AGI" but we can start with the Wikipedia definition unless you disagree: > Artificial general intelligence (AGI)—sometimes called human‑level intelligence AI—is a type of artificial intelligence that would match or surpass human capabilities across virtually all cognitive tasks We don't have consistent metrics for what we might mean by "human-level intelligence" or "cognitive tasks." We have some tests and benchmarks, with obvious flaws. For example LLMs can already perform well on SAT tests and legal bar exams because those get included in their training data. A category error, in other words. I'll give a real-world example from my experience. My wife had to pass a US citizenship test, described by USCIS: "The USCIS officer will ask you up to 10 questions from the list of 100 civics test questions. You must answer 6 questions correctly to pass the civics test." USCIS and others helpfully publishes the 100 questions and answers, so like so many candidates for US citizenship my wife memorized all of them (and passed). She mimicked an LLM. Of course she has a very different understanding of US history and civics than I do, because I grew up in America and attended and got socially programmed in public schools. My wife can perform better than a typical native-born American on that test, but has little to no real understanding of the material. Optimizing for the test -- Goodhart's Law illustrated -- seems more like what LLMs do, and it looks like cognitive ability because it succeeds at the metrics. In my last comment I claimed you make a category error by applying words such as intelligence, mirroring, bonding, evolution, objectives to LLMs (or AI if you prefer). While those words have vague or flexible meanings when applied to animals or humans, they don't apply at all to computer hardware or software. We use words like "intelligence" and "objectives" when talking about computer programs as analogies and conveniences, and then quickly fool ourselves into equivalency. The concept of AI "alignment," derived from the paper clip maximizer and seen in 2001 and The Terminator movies, seems hard to take seriously. It comes from sci-fi and paranoia about humans making machines they can't control (a trope as old as the Industrial Revolution). In many ways we already live in a world where machines and software operate in ways not aligned with human needs or objectives. When my iPhone once again fails to pair with my Airpods I could say my technology has not aligned with my purposes. What objective does my phone have when it acts up? What evolution or self-editing has taken place that makes it work one day but not the next? I can align my phone by turning it off and back on. If I perceive that the machines I use or the software I interact with no longer align with my goals I can just switch it off or delete it. Barring sci-fi scenarios where the machines break into the physical world to protect themselves from humans turning them off I don't think we have to worry about it. Humans have lived with dogs for millennia, and we know what happens when a dog's behavior fails to "align with" the humans it lives with. We treat our machines and software the same way with far less empathy. The stories about LLMs appearing to have their own objectives, lying, protecting themselves from shutdowns, etc. amount to hype from self-interested hucksters like Sam Altman. The AI marketing narrative -- which exists solely to keep the money flowing in -- generates scary stories to give the appearance that AGI lurks just around the corner, about to "emerge" from millions of GPUs. Look into the contract OpenAI has with Microsoft and it seems obvious why Altman pushes the imminent AGI story. We could speculate on the dangers of humans evolving the ability to read minds, or see into the future (both explored exhaustively in literature and movies), and then wring our hands over what to do if real X-Men somehow evolved. To me the (mostly fake and cynically manufactured) worry over AGI falls into that category. I could have it all wrong, we will see. If some Silicon Valley startup got hundreds of millions to develop psychic humans, or cognitively enhanced humans (see: Neuralink), would we interpret their claims and ethical quandaries as real or just cash grabs? The AGI threat works because the general public doesn't understand how it works and has seen both benevolent and malevolent AI in movies for decades. Most people can't explain how a microwave oven works despite living with them for decades. I date back far enough to remember when my grandparents feared a microwave oven giving them radiation sickness, at a time of widespread public ignorance and worry over anything "nuclear" and the word "radiation" jumping from science into everyday discourse (1960s). We have had software that can write other software, and edit itself, since the 1950s. I wouldn't use the term "evolve" to describe what computer hardware and software do, but we have seen amazing progress (through human insight and effort driven by profit, not Darwinian evolution) along multiple axes such as performance, size, power consumption, and price. Then physical limits get reached and the technology plateaus. I expect LLMs will plateau in the same way, due to inherent limits of the approach, exhausting the training data, or maybe power/water requirements. At this point LLMs amaze us in a way other software does not, because of natural language mastery. I think we have mistaken that impressive but narrow achievement for overall cognitive ability, because we only have our own species to measure against, and among humans mastering language and appearing to "know" a lot of things get interpreted (rightly) as intelligence. AI and what some will surely eventually label as AGI may pose an actual existential threat, the way nuclear weapons and tinkering with viruses already do. I think more likely the AI companies pose a grave economic threat, because if the bubble bursts before real use cases and profits materialize we will all suffer. The dot-com bust, 9/11, the 2008 real estate collapse, and COVID give some idea of the economic and social disruption (and government overreach) that happen even without Skynet trying to kill us due to mis-alignment. P.S. I have seen links for LLMs doing horoscopes, tarot readings, palmistry, fortune telling, and Bible study promoted here on HN. That says more about human ignorance and credulity than it does about LLMs, but as long as the software happily goes along with the nonsense and doesn't scold the vibe coders for asking I will sleep easier knowing the LLMs have remained in alignment, and haven't given any real deep thought to what we ask them to do -- at least no more thought than a backhoe gives to its work, or my phone gives to pairing over Bluetooth.