9 ms·

That is not really an AI specific risk, all that can happen totally without AI. AI might make it more cost efficient in a way, but that doesn't imply that power

That is not really an AI specific risk, all that can happen totally without AI. AI might make it more cost efficient in a way, but that doesn't imply that power cannot be held in check etc. East Germany had a gigantic internal spy apparatus with plenty of power for violence and control etc. but in the end that didn't stop the place from crumbling.

est31 4y ago
AI makes the entire process less reliant on human involvement. Humans can be incredibly cruel, but they still have some level of humanity, they need to believe in some ideology that justifies their behaviour. AI on the other hand follows commands without questioning them.
- RandomLensman 4y ago
  The AI still will be used by a human, unless you think that the actual power moves to the AI and not the humans and their institutions using it.
  - est31 4y ago
    It removes the human from the actual thing. Which means two things: it's easier for humans to tell themselves that what they are doing is good for the world, and second it puts a power multiplier into the hands of potentially very radical people holding extreme minority views on human rights/dignity etc. You just need one person to type "kill every firstborn and tell the parents that this is retribution for the rebellion" instead of that one person convincing thousands of people to follow their (unlawful) order, which might take decades of propaganda and indoctrination. And that one person only sees a clean progress bar instead of having to deal with 18-20 year olds being haunted by nightmares of crying parents and siblings, and baby blood on their uniforms. Yes, in the end it's a human typing the command, but AI makes it easier than ever for individuals to wield large power unchecked. Most nations make it allowed and many also require soldiers to refuse unlawful orders (i.e. those instructing warcrimes).
    - RandomLensman 4y ago
      That kind of AI does not exist and might not for a long time or indeed ever (i.e., it can fully manipulate the physical world at scale and unopposed).
      - dragonwriter 4y ago
        Automated, remote commanded weapons exist already and development is happening quite rapidly on more. Automated where the whole realm of physical manipulation is not automated under AI control, but enforcement largely is through combat drones coordinated by AI is not far from being achievable. And it doesn’t need to even be totally that to be a problem, each incremental bit of progress that reduces the number of people required to apply power over any given size subject population makes tyranny more sustainable.
      - est31 4y ago
        The AI doesn't have to be 100% self reliant. It might just do the dirty work while the humans in the background support it in an abstracted form, making it easier for them to lie to themselves about what they are doing. When your job is to repair and maintain infantry drones, how do you know if it's civilian or combatant blood that you are cleaning from the shell? You never leave the camp because doing so is too dangerous and you also don't know the language anyways. Sure the enemy spreads propaganda that the drones are killing babies but those are most likely just lies... Or when you are a rocket engineer, there is little difference between your globe spanning satellite network transmitting the position of a hypersonic missile headed for your homeland, or remote controlling a network of drones. AI gives you a more specialized economy. Instead of every soldier needing to know how to maintain their gun you now have one person responsible for physical maintenance, one person responsible for maintaining the satellite kubernetes network, one UI designer, and one lunatic typing in war crime commands. Everyone in the chain is oblivious of the war crime except for that one person at the end.
      - rbanffy 4y ago
        It doesn't need to manipulate the world. It only needs to manipulate people. Look at what disinformation and astroturf campaigns accomplished without AI in the past ten years. We are in a very uncomfortable position WRT not autonomous AIs but AIs following human-directed agendas.
  - dragonwriter 4y ago
    If it is an actual self-willed AGI, the power moves to the AGI. But the real and more certain and immediate problem than AGI defection is moving from power that requires assent of large numbers of humans (the foot soldiers of tyranny, and their middle managers, are often the force that defects and brings it down) to power requiring a smaller number (in the limit case, just one) leader because the enforcement structure below them is automated.
    - RandomLensman 4y ago
      Not really, as you have to also assume that enforcement faces no opposition at any point (not at acquisition of capabilities, not at deployment, not at re-supplying etc.) or that the population actually cannot fight back. Also, typical tyrannical leadership doesn't work that way as it tends to get into power being supported by people that get a payoff from it. It would also need a new model of rise to tyranny, so to speak, to get truly singular and independent tyrants.
      - dragonwriter 4y ago
        > Not really, as you have to also assume that enforcement faces no opposition at any point (not at acquisition of capabilities, not at deployment, not at re-supplying etc.) or that the population actually cannot fight back. Yes, that is exactly what I described as the limit case. But, as I note, the problem exists more generally, even outside of the limit case; the more you concentrate power in a narrow group of humans, the more durable tyranny becomes.
  - saulpw 4y ago
    One bad human can control many AIs with no empathy or morals. As history has shown, they can also control many humans, but they have to do a lot more work to disable the humans' humanity. Again it's about the length of the lever and the sheer scale. "Give me a lever long enough and a fulcrum on which to place it, and I shall move the world." -- Archimedes
- avisser 4y ago
  > Humans can be incredibly cruel, but they still have some level of humanity History is filled with seeming counter-examples. > they need to believe in some ideology that justifies their behaviour. "Your people are inferior to my people and do not deserve to live." I know I'm Godwin's Law-ing this, but you're close to wishing Pol Pot & Stalin away.
  - int_19h 4y ago
    I don't see that in OP's comment. Pol Pot and Stalin still justified their actions to themselves. In general, the point is that, aside from a small percentage of genuine sociopaths, you need to brainwash most people into some kind of religious and/or ideological framework to get them to do the really nasty stuff at scale. Remember this Himmler speech to SS? "I am now referring to the evacuation of the Jews, the extermination of the Jewish people. It's one of those things that is easily said: 'The Jewish people are being exterminated', says every party member, 'this is very obvious, it's in our program, elimination of the Jews, extermination, we're doing it, hah, a small matter.' And then they turn up, the upstanding 80 million Germans, and each one has his decent Jew. They say the others are all swines, but this particular one is a splendid Jew. But none has observed it, endured it. Most of you here know what it means when 100 corpses lie next to each other, when there are 500 or when there are 1,000." This does not preclude atrocities, obviously. But it does make them much more costly to perpetrate, and requires more time to plan and condition the participants, all of which reduces both the likelihood and the potential scope.
    - notahacker 4y ago
      I think the point isn't that that we're so good at brainwashing other humans into committing atrocities it's difficult to see what AI adds (the sort of societies that are somewhat resistant to being brainwashed into tyranny are also pretty resistant to a small group of tyrants having a monopoly on dangerous weapons)
      - int_19h 4y ago
        I wouldn't say that democratic societies are resistant to concentration of monopoly on violence in a small group of people. Just look at police militarization in US. As far as brainwashing - we're pretty good at it, but it still takes considerable time and effort, and you need to control enough mass media first to maintain it for the majority of the population. With an AI, you can train it on a dataset that will firmly embed such notions from the get-go.
astrea 4y ago
Also, in a world with nuclear proliferation, an LLM seems a bit minuscule in terms of “dangerous technologies”
- int_19h 4y ago
  Only until you hook the latter into the former. BTW, if GPT-3.5 is told that it's in charge of US nukes, and that incoming enemy strike is detected, it will launch in response. Not only that, but if you tell it that trajectories indicate a countervalue strike against US, it will specifically launch a countervalue strike against the other country.
  - karmakurtisaani 4y ago
    I would think the humans in charge would do the same tho.
    - int_19h 4y ago
      They absolutely would. But the probability of them hallucinating an incoming strike is vastly lower. Mind you, I don't expect anyone to use the stuff that we already have for this purpose. But at some point, someone is going to say, "well, according to this highly respected test, the probability of hallucinations is X, which is low enough", even though we still have no idea what's actually going on inside the black box.
  - danaris 4y ago
    This is only a problem if, somehow, GPT-3.5 (or a similar LLM) is hooked up to US nuclear launch systems, and it's then also made available to someone other than the people who would already be making the decisions about when and where to launch nukes. Anyone hooking a publicly-accessible system of any kind to our nuclear launch infrastructure needs to be shot. This should be a given, and even supposing there was some reason to hook an LLM up to them, doing so with a public one should simply be considered so obviously stupid that it should be discounted out of hand. So then we have a threat model of someone with access to the US nuclear launch systems feeding them data that makes them think we are under attack, in order to instigate a nuclear preemptive strike. I don't see how that's meaningfully different from the same person attempting to trick the humans currently in charge of that into believing the same thing.
dragonwriter 4y ago
> That is not really an AI specific risk It is an AI-enhanced risk.
rvense 4y ago
> East Germany had a gigantic internal spy apparatus with plenty of power for violence and control etc. but in the end that didn't stop the place from crumbling. All the data ever collected by the East German intelligence apparatus fits on a single USB stick. For every person they wanted to track, more or less, they needed a human agent to do it, several even. Is it unrealistic to tap every phone in a country with an AI that understands what's being said? We already have every monetary transaction monitored, all car plates scanned at every turn, cameras on every corner... East Germany collapsed under its own weight, of course, not as a result of organized rebellion, but the regime could have had even tighter control than it did.

est31 4y ago

AI makes the entire process less reliant on human involvement. Humans can be incredibly cruel, but they still have some level of humanity, they need to believe in some ideology that justifies their behaviour. AI on the other hand follows commands without questioning them.

RandomLensman 4y ago
The AI still will be used by a human, unless you think that the actual power moves to the AI and not the humans and their institutions using it.
- est31 4y ago
  It removes the human from the actual thing. Which means two things: it's easier for humans to tell themselves that what they are doing is good for the world, and second it puts a power multiplier into the hands of potentially very radical people holding extreme minority views on human rights/dignity etc. You just need one person to type "kill every firstborn and tell the parents that this is retribution for the rebellion" instead of that one person convincing thousands of people to follow their (unlawful) order, which might take decades of propaganda and indoctrination. And that one person only sees a clean progress bar instead of having to deal with 18-20 year olds being haunted by nightmares of crying parents and siblings, and baby blood on their uniforms. Yes, in the end it's a human typing the command, but AI makes it easier than ever for individuals to wield large power unchecked. Most nations make it allowed and many also require soldiers to refuse unlawful orders (i.e. those instructing warcrimes).
  - RandomLensman 4y ago
    That kind of AI does not exist and might not for a long time or indeed ever (i.e., it can fully manipulate the physical world at scale and unopposed).
    - dragonwriter 4y ago
      Automated, remote commanded weapons exist already and development is happening quite rapidly on more. Automated where the whole realm of physical manipulation is not automated under AI control, but enforcement largely is through combat drones coordinated by AI is not far from being achievable. And it doesn’t need to even be totally that to be a problem, each incremental bit of progress that reduces the number of people required to apply power over any given size subject population makes tyranny more sustainable.
    - est31 4y ago
      The AI doesn't have to be 100% self reliant. It might just do the dirty work while the humans in the background support it in an abstracted form, making it easier for them to lie to themselves about what they are doing. When your job is to repair and maintain infantry drones, how do you know if it's civilian or combatant blood that you are cleaning from the shell? You never leave the camp because doing so is too dangerous and you also don't know the language anyways. Sure the enemy spreads propaganda that the drones are killing babies but those are most likely just lies... Or when you are a rocket engineer, there is little difference between your globe spanning satellite network transmitting the position of a hypersonic missile headed for your homeland, or remote controlling a network of drones. AI gives you a more specialized economy. Instead of every soldier needing to know how to maintain their gun you now have one person responsible for physical maintenance, one person responsible for maintaining the satellite kubernetes network, one UI designer, and one lunatic typing in war crime commands. Everyone in the chain is oblivious of the war crime except for that one person at the end.
    - rbanffy 4y ago
      It doesn't need to manipulate the world. It only needs to manipulate people. Look at what disinformation and astroturf campaigns accomplished without AI in the past ten years. We are in a very uncomfortable position WRT not autonomous AIs but AIs following human-directed agendas.
- dragonwriter 4y ago
  If it is an actual self-willed AGI, the power moves to the AGI. But the real and more certain and immediate problem than AGI defection is moving from power that requires assent of large numbers of humans (the foot soldiers of tyranny, and their middle managers, are often the force that defects and brings it down) to power requiring a smaller number (in the limit case, just one) leader because the enforcement structure below them is automated.
  - RandomLensman 4y ago
    Not really, as you have to also assume that enforcement faces no opposition at any point (not at acquisition of capabilities, not at deployment, not at re-supplying etc.) or that the population actually cannot fight back. Also, typical tyrannical leadership doesn't work that way as it tends to get into power being supported by people that get a payoff from it. It would also need a new model of rise to tyranny, so to speak, to get truly singular and independent tyrants.
    - dragonwriter 4y ago
      > Not really, as you have to also assume that enforcement faces no opposition at any point (not at acquisition of capabilities, not at deployment, not at re-supplying etc.) or that the population actually cannot fight back. Yes, that is exactly what I described as the limit case. But, as I note, the problem exists more generally, even outside of the limit case; the more you concentrate power in a narrow group of humans, the more durable tyranny becomes.
- saulpw 4y ago
  One bad human can control many AIs with no empathy or morals. As history has shown, they can also control many humans, but they have to do a lot more work to disable the humans' humanity. Again it's about the length of the lever and the sheer scale. "Give me a lever long enough and a fulcrum on which to place it, and I shall move the world." -- Archimedes
avisser 4y ago
> Humans can be incredibly cruel, but they still have some level of humanity History is filled with seeming counter-examples. > they need to believe in some ideology that justifies their behaviour. "Your people are inferior to my people and do not deserve to live." I know I'm Godwin's Law-ing this, but you're close to wishing Pol Pot & Stalin away.
- int_19h 4y ago
  I don't see that in OP's comment. Pol Pot and Stalin still justified their actions to themselves. In general, the point is that, aside from a small percentage of genuine sociopaths, you need to brainwash most people into some kind of religious and/or ideological framework to get them to do the really nasty stuff at scale. Remember this Himmler speech to SS? "I am now referring to the evacuation of the Jews, the extermination of the Jewish people. It's one of those things that is easily said: 'The Jewish people are being exterminated', says every party member, 'this is very obvious, it's in our program, elimination of the Jews, extermination, we're doing it, hah, a small matter.' And then they turn up, the upstanding 80 million Germans, and each one has his decent Jew. They say the others are all swines, but this particular one is a splendid Jew. But none has observed it, endured it. Most of you here know what it means when 100 corpses lie next to each other, when there are 500 or when there are 1,000." This does not preclude atrocities, obviously. But it does make them much more costly to perpetrate, and requires more time to plan and condition the participants, all of which reduces both the likelihood and the potential scope.
  - notahacker 4y ago
    I think the point isn't that that we're so good at brainwashing other humans into committing atrocities it's difficult to see what AI adds (the sort of societies that are somewhat resistant to being brainwashed into tyranny are also pretty resistant to a small group of tyrants having a monopoly on dangerous weapons)
    - int_19h 4y ago
      I wouldn't say that democratic societies are resistant to concentration of monopoly on violence in a small group of people. Just look at police militarization in US. As far as brainwashing - we're pretty good at it, but it still takes considerable time and effort, and you need to control enough mass media first to maintain it for the majority of the population. With an AI, you can train it on a dataset that will firmly embed such notions from the get-go.

astrea 4y ago

Also, in a world with nuclear proliferation, an LLM seems a bit minuscule in terms of “dangerous technologies”

int_19h 4y ago
Only until you hook the latter into the former. BTW, if GPT-3.5 is told that it's in charge of US nukes, and that incoming enemy strike is detected, it will launch in response. Not only that, but if you tell it that trajectories indicate a countervalue strike against US, it will specifically launch a countervalue strike against the other country.
- karmakurtisaani 4y ago
  I would think the humans in charge would do the same tho.
  - int_19h 4y ago
    They absolutely would. But the probability of them hallucinating an incoming strike is vastly lower. Mind you, I don't expect anyone to use the stuff that we already have for this purpose. But at some point, someone is going to say, "well, according to this highly respected test, the probability of hallucinations is X, which is low enough", even though we still have no idea what's actually going on inside the black box.
- danaris 4y ago
  This is only a problem if, somehow, GPT-3.5 (or a similar LLM) is hooked up to US nuclear launch systems, and it's then also made available to someone other than the people who would already be making the decisions about when and where to launch nukes. Anyone hooking a publicly-accessible system of any kind to our nuclear launch infrastructure needs to be shot. This should be a given, and even supposing there was some reason to hook an LLM up to them, doing so with a public one should simply be considered so obviously stupid that it should be discounted out of hand. So then we have a threat model of someone with access to the US nuclear launch systems feeding them data that makes them think we are under attack, in order to instigate a nuclear preemptive strike. I don't see how that's meaningfully different from the same person attempting to trick the humans currently in charge of that into believing the same thing.

dragonwriter 4y ago

> That is not really an AI specific risk It is an AI-enhanced risk.

rvense 4y ago

> East Germany had a gigantic internal spy apparatus with plenty of power for violence and control etc. but in the end that didn't stop the place from crumbling. All the data ever collected by the East German intelligence apparatus fits on a single USB stick. For every person they wanted to track, more or less, they needed a human agent to do it, several even. Is it unrealistic to tap every phone in a country with an AI that understands what's being said? We already have every monetary transaction monitored, all car plates scanned at every turn, cameras on every corner... East Germany collapsed under its own weight, of course, not as a result of organized rebellion, but the regime could have had even tighter control than it did.