It's selection pressure. Both OpenAI (back in 2015) and Anthropic (much later, in 2021) were founded by people who could foresee the concept of AI x-risks and

It's selection pressure. Both OpenAI (back in 2015) and Anthropic (much later, in 2021) were founded by people who could foresee the concept of AI x-risks and wanted to work on preventing it. For OpenAI's founding, the idea was that it's much safer if AGI is achieved by a nonprofit explicitly dedicated to humanity than if it's done by a profit-driven company. For Anthropic's, it was that OpenAI seems to be going insane and it'd be better if a more safety-conscious company was competing. I'd say in hindsight, the former motivation was reasonable and the latter one was... probably bad, but not obviously so - it doesn't seem even in hindsight that it was inevitable that Anthropic's existence would only drive competition and cause them both to race to AGI. That said, even at founding time, there was immediately a lot of selection - quite a few people who cared about AI safety simply wouldn't agree to work at a capabilities company, even given an elaborate argument why this is a good idea, so those were selected against. And then, of course, many years passed, OpenAI was stolen by Sam Altman and turned into a for-profit, multiple researchers left OpenAI realising that they're only helping build misaligned AI faster, multiple researchers left Anthropic realising that they, too, are only helping build misaligned AI faster, and here we are in 2026. There's lots of AI researchers these days, so you can select for conformance very heavily and still have a fully-staffed company. The people who are still at these companies are outliers in various ways. They either manage to dismiss the importance AI risks (for example, by adopting some sort of belief in the vein of "there's no use worrying about AI wiping out humanity - it'd just be a successor species, like we were to apes, which is good"*), or they still think that leaving will not make things go better (Dario Amodei is pretty clearly in this camp, and has always been), or they weight the risk of extinction against the potential benefit of an post-singularity utopia and consider it a good bet (not realizing that the alternative isn't giving up on a post-singularity utopia, but getting it a few decades later and without the risk), or they just manage not to think of the contradictions, which humans are of course very good at. * If that sounds like a strawman, see minute 15 and on of this Richard Sutton presentation: https://twitter.com/RichardSSutton/status/1898082481251008929 https://twitter.com/RichardSSutton/status/189808248125100892...