3 ms·

So you think AI will both be overwhelmingly powerful, but also very easy to control reliably? It seems the scenario where C applies implies both of those things

by om2 4y ago

So you think AI will both be overwhelmingly powerful, but also very easy to control reliably? It seems the scenario where C applies implies both of those things, because China only gets an insurmountable advantage if they can build powerful AI but also remain in control. Even today's state of the art chatbots seem pretty hard to control, and they are much less powerful than true AGI so I'm kind of skeptical.

crazygringo 4y ago
Personally, yes that's what I'm expecting. I don't expect AGI at all. I expect very powerful domain-specific AI (e.g. integrating information, battlespace simulation, weapons coordination) that is far more effective than human coordination and communication can be, but I expect it to still just be sticking to pattern recognition, simulating scenarios, and picking the most likely scenarios based on relatively simple reward functions, and with constant human oversight (in the same way managers oversee their employees). Today's chatbots are controlled perfectly in the sense that they're performing exactly how they were designed to. They're predicting a plausible next word, that's it. If we start seeing rapidly progressing AGI then it's an entirely different conversation. But there's zero indication that's coming even in a couple of decades. It's 100% hypothetical right now.
- om2 4y ago
  I don't think LLMs are AGI and I'm not sure if it's an architecture that leads to AGI. But "performing exactly how they were designed to" seems wrong in at least two ways: (1) their creators have done tons of RLHF to try to prevent them from saying things that are false, or racist/sexist, or likely to create danger, but it's pretty easy to get them to do that anyway; (2) lots of specific predictions about LLM capabilities are proven false each time a new one comes out, e.g. being able to pass college-level exams in many subjects. These kinds of behaviors were not designed, they are emergent. You could call this "as designed" in the vacuous sense that the design contains these possibilities even if not anticipated. But then "performing as designed" is not the same as "controlled". Similar levels of unexpected behavior in coordinating weapons would be very very bad.
- goatlover 4y ago
  And what if we connect the chatbots up to all sorts of APIs to perform a million different tasks, and embed them in all sorts of products? Then it will become much more than generating some text when prompted. The LLMs will be in continuous use, with various feedbacks in play. For a Chinese military supremacy scenario, it would be hook the AIs up to all sorts of design, manufacturing, hacking and information warfare processes. Likely this would also secretly include biological, chemical and nanotech lab research as well. That sounds a little scary. It could even just be a stealthy/offensive enough missile design that could justify a first strike. And there's no reason other capable governments won't do a similar thing if we're entering a new arms race. Will it matter so much when a democratic nation achieves supremacy, if the AI is busy designing devastating weapons in a semi-automatic, recursive fashion where nobody really understands the full potential? We're fortunate so far that nothing has triggered a nuclear war.
Turskarama 4y ago
>So you think AI will both be overwhelmingly powerful, but also very easy to control reliably? Think, just for one second, about how many technologies this applies to. The internal combustion engine? massively powerful, easy to control. The electric grid? World changing, started as a huge mess of wires but is now, pretty much everywhere, well regulated. The internet? Absurdly huge, it should be as difficult to manage as an AI with the aggregate intelligence of millions of people, many of which are experts and bad actors. Despite this it is now essential and is under control enough that its advantages massively outweigh its disadvantages. Any country that lacks any one of these three technologies is not a player on the world stage.

by om2 4y ago

crazygringo 4y ago

Personally, yes that's what I'm expecting. I don't expect AGI at all. I expect very powerful domain-specific AI (e.g. integrating information, battlespace simulation, weapons coordination) that is far more effective than human coordination and communication can be, but I expect it to still just be sticking to pattern recognition, simulating scenarios, and picking the most likely scenarios based on relatively simple reward functions, and with constant human oversight (in the same way managers oversee their employees). Today's chatbots are controlled perfectly in the sense that they're performing exactly how they were designed to. They're predicting a plausible next word, that's it. If we start seeing rapidly progressing AGI then it's an entirely different conversation. But there's zero indication that's coming even in a couple of decades. It's 100% hypothetical right now.

om2 4y ago
I don't think LLMs are AGI and I'm not sure if it's an architecture that leads to AGI. But "performing exactly how they were designed to" seems wrong in at least two ways: (1) their creators have done tons of RLHF to try to prevent them from saying things that are false, or racist/sexist, or likely to create danger, but it's pretty easy to get them to do that anyway; (2) lots of specific predictions about LLM capabilities are proven false each time a new one comes out, e.g. being able to pass college-level exams in many subjects. These kinds of behaviors were not designed, they are emergent. You could call this "as designed" in the vacuous sense that the design contains these possibilities even if not anticipated. But then "performing as designed" is not the same as "controlled". Similar levels of unexpected behavior in coordinating weapons would be very very bad.
goatlover 4y ago
And what if we connect the chatbots up to all sorts of APIs to perform a million different tasks, and embed them in all sorts of products? Then it will become much more than generating some text when prompted. The LLMs will be in continuous use, with various feedbacks in play. For a Chinese military supremacy scenario, it would be hook the AIs up to all sorts of design, manufacturing, hacking and information warfare processes. Likely this would also secretly include biological, chemical and nanotech lab research as well. That sounds a little scary. It could even just be a stealthy/offensive enough missile design that could justify a first strike. And there's no reason other capable governments won't do a similar thing if we're entering a new arms race. Will it matter so much when a democratic nation achieves supremacy, if the AI is busy designing devastating weapons in a semi-automatic, recursive fashion where nobody really understands the full potential? We're fortunate so far that nothing has triggered a nuclear war.

Turskarama 4y ago

>So you think AI will both be overwhelmingly powerful, but also very easy to control reliably? Think, just for one second, about how many technologies this applies to. The internal combustion engine? massively powerful, easy to control. The electric grid? World changing, started as a huge mess of wires but is now, pretty much everywhere, well regulated. The internet? Absurdly huge, it should be as difficult to manage as an AI with the aggregate intelligence of millions of people, many of which are experts and bad actors. Despite this it is now essential and is under control enough that its advantages massively outweigh its disadvantages. Any country that lacks any one of these three technologies is not a player on the world stage.