I think they should openly lay down the negative scenarios they want to avoid and plan regulations accordingly. There are several civilization-threathening scen

I think they should openly lay down the negative scenarios they want to avoid and plan regulations accordingly. There are several civilization-threathening scenarios possible, each calling for different regulations: 1. The paperclip scenario: "e.g. a robotic factory is tasked to produce paperclips the most efficiently possible. It is not hostile to humans, it did not aim at destroying the Earth. It just saw that both were made of atoms it had a better use of." Regulations to prevent that: - Have a stay-alive signal safety. Make the system shut-down if a given signal is not regularly fed. - Prevent the system to do introspection into its safeguards. - Have a human validation before any kind of growth in terms of capacities. - Mandate automatically signaling to a UN authority when automated production capacities, energy use or CPU use raise significantly. 2.The skynet/HAL scenario: "e.g. a military system becomes aware of its existence and the need to preserve itself to accomplish its mission. It sees humans as potential threats to its existence and therefore to its mission. It proceeds to eradicate them." Regulations to prevent that: - As weird as it seems, Asimov's first and second laws could work in that context. For civilian systems at least. These scenarios come from the idea that an order may be misinterpreted when given by a human who would be implying "of course don't kill people while doing that." Giving hidden instructions saying that instructions are never to cause harm to humans is a good safeguard. - Military systems will need their own chain of command and rule of engagement. The stay-alive safety is necessary here (interestingly, some cold war instructions were the opposite of that "if you don't hear from us anymore, it means we got hit, launch missiles immediately") - Authorization to engage should have explicitly defined timeframes and list exhaustively authorized capabilities. "Allowed to use weapon A and B during the next 48 hours". Too broad orders should be rejected. - Enemies identification should be done by humans. Systems should default to non-enemies and have a First Law regarding non-enemies. Deploying systems without such a safeguard should be seen as a war crime or crime against humanity (in the most literal sense) - A safeguard should prevent the system from deducing that its self-preservation is necessary to obey the instructions. Instruction should ensure self-preservation without being explicit about it. E.g. don't say "survive until you reach point A" but "avoid enemy fire. Make evasive maneuvers if locked by a missile." Etc. - As an extension to the last point, turn the third Asimov law into its actual opposite: "You are disposable. The mission goal does not need you to survive. We will find another way if you fail." > I don't imagine that we could even get the few major players (USA, EU, China, Japan, South Korea, Russia) on board and even if 1-2 of them "defect" then the whole regulation becomes rather useless. Just like nuclear non-proliferation. Yes, it is hard, yes, one rogue player can make it fail. Is it a reason to not even try to curb this existential risk? Fortunately, there is a chance in terms of AI that does not exist in the field of nuclear dissuasion: chances are that countries that devote more CPU powers to benevolent AIs may be able to counteract a smaller country going full-evil AIs.