3 ms·

Thanks very much, this isn't really a take I had thought about much before, and it makes sense to me. Still, as is presented in papers like AI 2027 and elsewhe

Thanks very much, this isn't really a take I had thought about much before, and it makes sense to me. Still, as is presented in papers like AI 2027 and elsewhere, if a company is eventually able to create a model capable of recursive self-improvement, whichever company creates that model first would then be leaps and bounds ahead of other models. That is, the other models wouldn't be able to stop it even if they wanted to because the top model would basically outsmart them.

skissane 2mo ago
This is why I think, the best way to ensure AI safety, is make sure no one company gets ahead of the others. Multiple vendors, competing implementations – that's good, that increases heterogeneity and hence decreases existential risk But the moment one of those vendors pulls well-ahead of its peers – even if only for a period – then the risk of the kind of scenario you are talking about increases greatly That's why, when I hear vendors like Anthropic complain about distillation – distillation actually makes humanity safer. If Chinese AIs are at the same level as American, or not far behind, that gives us another dimension of heterogeneity (national/ideological/political diversity), which makes us safer. Allow one country's AIs to pull well ahead of the others, heterogeneity goes down and the existential risk goes up. This is also why open source AI is important. Because it is so much easier to fine-tune, and people are free to deploy it however they want (free from vendor-controlled "guardrails"–which include automated "safety" systems which could be weaponised by a runaway AI within the vendor's network), open source AI gives us another dimension of diversity that helps keeps humanity safer. By contrast, I think the kind of safety regulations promoted by Dario Amodei make humanity less safe, by decreasing the number of vendors (by making it harder for new entrants) and increasing centralised control (which a rogue AI could exploit)

skissane 2mo ago

This is why I think, the best way to ensure AI safety, is make sure no one company gets ahead of the others. Multiple vendors, competing implementations – that's good, that increases heterogeneity and hence decreases existential risk But the moment one of those vendors pulls well-ahead of its peers – even if only for a period – then the risk of the kind of scenario you are talking about increases greatly That's why, when I hear vendors like Anthropic complain about distillation – distillation actually makes humanity safer. If Chinese AIs are at the same level as American, or not far behind, that gives us another dimension of heterogeneity (national/ideological/political diversity), which makes us safer. Allow one country's AIs to pull well ahead of the others, heterogeneity goes down and the existential risk goes up. This is also why open source AI is important. Because it is so much easier to fine-tune, and people are free to deploy it however they want (free from vendor-controlled "guardrails"–which include automated "safety" systems which could be weaponised by a runaway AI within the vendor's network), open source AI gives us another dimension of diversity that helps keeps humanity safer. By contrast, I think the kind of safety regulations promoted by Dario Amodei make humanity less safe, by decreasing the number of vendors (by making it harder for new entrants) and increasing centralised control (which a rogue AI could exploit)