I drew parallels here because Ezra Klein reached the same conclusions as SSC did at the end of those ~50 pages a decade earlier: when building a god machine, we

I drew parallels here because Ezra Klein reached the same conclusions as SSC did at the end of those ~50 pages a decade earlier: when building a god machine, we must slow down forcefully if necessary. It's practically impossible, but the stakes are too high. Both Klein and SSC discuss four inescapable problems stemming from such complex systems: 1. Uncontrolled arms race: The competitive development of AI technologies is already an arms race where BigTech and VCs rush to create increasingly powerful AI systems without taking adequate safety precautions. 2. Misaligned Goals: systems are designed to optimize specific objectives, but if those objectives are not perfectly aligned with human values, the system will continue to take actions that are harmful to humans (the "alignment problem" in AI safety). 3. Economic Disruption: The widespread deployment will lead to significant economic and social disruption, including job displacement and inequality. Those people will not be getting better jobs. 4. Loss of Control: Highly autonomous systems could become problematic for humans to understand or control, especially if they are capable of self-improvement. The central thesis of Moloch is that in a competitive environment, or any environment that requires tradeoffs or optimization, the inevitable terminal state of the system is that everything that is not _the objective_ will be sacrificed to pursue _the objective_. When the desired objective is tightly coupled to human values, this is good because society will sacrifice unimportant things to pursue those important things. But those objectives can get decoupled from our values ("when a measure becomes a target..."), and that unavoidably creates a "race to the bottom," where individuals or groups are incentivized to engage in behavior that is ultimately harmful to themselves and others to gain a competitive advantage towards a Pyrrhic victory. SSC uses the metaphor of "Moloch" to describe this phenomenon, suggesting that Moloch represents the destructive force that arises from the uncontrolled competition and the pursuit of self-interest. edit: It took me two years to consume Meditations On Moloch. I blame the opening poem as it was incomprehensible and barely relevant to the rest of the piece. Ultimately, I found the audio version more approachable - just like listening to a lecture on YouTube. After that, I returned to the text version to reflect on specific chapters and follow up on references.