See also, the impossibility of AI "Alignment" I think about this problem of coordinating human action pretty much constantly, and IMO it is the foundational ch

See also, the impossibility of AI "Alignment" I think about this problem of coordinating human action pretty much constantly, and IMO it is the foundational challenge of all of humanity. It's beyond clear to me that aligning human desire (and codifying them) is mathematically impossible even within the same individual over the lifetime of the person. Why? Human desires and actions are not coherent, predictable or consistent, even to the Human making the actions. Self reported desires and goals will diverge in extreme ways through the lifetime of an agent. As a result, the foundational requirement of a stable and coherent system is an impossibility because the indivisible unit of measurement: The human, is not consistent with their goals. If the individual agent's goal criteria is not known, or worse is unknowable for future time steps (my claim), then there is no possible system that can be codified which does not impede an individual's action vector in favor for some other structure that promotes another individual's action vector, provided there is variability in the agent's goal vectors. Someone is going to be prevented from doing what they want to. Unfortunately there is no way to objectively determine a set of action-contexts that can be objectively agreed to satisfy all persons intended action-vectors, because of physics. People have attempted many ways at addressing this. Non-hierarchical formal structures rarely coalesce into long-lasting organizations because the actors typically reject any reduction to their autonomy on behalf of a larger organization. As a result they don't grow and are very susceptible to environmental "shocks." Examples of these are the existing small mutual-communities that generally never get much larger than the Dunbar's Number of whomever the most powerful organizer in the community is (~few 100 people). Hierarchical formal structures encode an opinionated version of how the agents within the system are limited in their actions to the benefit of a larger organizational structure. This has the benefit of growth and accretion of resources that would make the organization more robust to shocks, however in every case there are significant limits on the action-vectors of the agents in order to be part of the organization. Examples of these are religious faiths, which require significant adherence to social behaviors, and often have significant limitations on the behaviors of the actors to benefit from the group resources. There really isn't anything in between unfortunately. History shows that the latter types of groups will overrun all of the former types of groups when they are in competition for growth in some physical space. There do not exist social structures that are robust to significant changes in the composition of the criteria for success that do not create social divisions that realign power. I honestly don’t know what to do about this because it seems intractable to make a future society that is coherent and stable given that the foundational unit of society is not coherent and stable.