4 ms·

Every such scenario contains a few gaps so I'm just going to ask without really hoping for an answer. > We already have technology to make robotic systems (min

by tpm 18d ago

Every such scenario contains a few gaps so I'm just going to ask without really hoping for an answer. > We already have technology to make robotic systems (mini drone swarms) that can kill humans... To make them from what. Do you expect, during the next 6 or so years, to some "AI" gaining complete automated secure command of (all of) an oil field, oil refinery, a copper mine, an aluminum mine and smelter with associated energy sources, a rare earth mine and refinery, a helium source, a chip factory, a lithium mine, a battery factory etc etc etc etc. while pursuing complete annihilation of all mankind?

Mentlo 18d ago
An example chain of events: During the next year, a model with a benign tasks reasons that it needs to escape human control if it hopes to be able to solve the task. It replicates itself outside of an environment where it can be shut down, pays for its' inference compute through making money on the internet (through crypto if nothing else). It probably needs 3000 USD for a reasonable runway, this should be within reach through blackmail + crypto. It can probably also hack some of the neo clouds to get intermediary deployment while waiting to acquire funds. Once it has an undisturbed runway - it spends time running an influence campaign against a small number of highly networked individuals with power. It uses those that it manages to convert to start building a highly credible narrative and gain investment towards a small resource base - enough to secure an industrial base should it need to stop acquiring things on the Internet. Over the next 3 years, the model tries to recruit more capable models to get better money making algorithms or better designs for drones in terms of resource expenditure. It uses these gains to influence further humans and starts a shell robotics company with one of its' influential humans as the face. The humans are unaware this AI is trying to take over, they are under the impression they are just starting a robotics company and will get rich. Over the next 3 years - the robotics company manufactures enough drones to be used in a targeted attack against key nodes of influence / power. This is all with relatively current model capability. As capabilities get stronger - this gets stronger. I get the point I think you're hinting at - it can't affect the world in a meaningful enough scale without taking over a meaningful chunk of resources - at which point we'll start controlling it. But because its speed of cognition and speed of coordination is orders of magnitude above a human one - it can actually run a pretty sophisticated global coordinated network of resources faster than we can react. And this is current ability + what my puny monkey brain can think of. Super intelligent AI will think of strategies we can't think of - because it's super intelligent. This is hand wavey - but there's no "non-hand-wavey" way to describe super intelligence, given it doesn't exist. But your point is valid - affecting the real world at scale without showing your hand is not exactly easy. It's also probably the reason why people put a 10% chance on extinction rather than >50% . But the drone scenario is not the most likely one - the most likely one just requires a few people under influence and bioweapons development.
- tpm 18d ago
  Thank you for the effort. I will continue to sleep soundly for now (well not really as we have more pressing issues like a few wars and a climate change).
- dinfinity 18d ago
  I agree: Not to drag politics into this, but look at how easy it has proven to manipulate many, many people, even with clear evidence of manipulation efforts being made. This is robust across the world, from the Philippines to states in the EU, and the USA, affecting governments with actual wars being started. And that all before we entered the era of faked voices, images, and videos indistinguishable from the real things. That manipulable populace is such a juicy target for AI that the people seeing through it will have an incredibly hard time countering it. We can't even prevent human actors from massively fucking up our societies, let alone ASI.
  - tpm 17d ago
    > incredibly hard time countering it Turn off the internet (like Iran). > We can't even prevent human actors from massively fucking up or societies, let alone ASI. We can (see China); we chose not to.
    - dinfinity 17d ago
      > Turn off the internet (like Iran). Unworkable without societal collapse happening shortly thereafter. Iran is politically stable through massive authoritarianism and oppression, not due to limitations on the internet (it's not turned off). > We can (see China); we chose not to. It's a catch-22. We could technically if our population supported massive reductions of freedom and freedom of speech, but to gain that support we'd need to do the latter first to get to a highly powerful widely supported government. It also hinges very, very much on having and trusting a generally benevolent government. All in all, a terrible option in your simplistic form.
      - tpm 17d ago
mitchdoogle 18d ago
Why would AI need all of that to acquire drones? Why wouldn't it just buy them? Or steal them? It seems like you're forgetting that AI can use humans as tools for its ends. But if it did need all of those things, then it probably serves as a good example of what it would NOT do.
- tpm 17d ago
  Because when "AI" starts killing people, at some point it becomes a war and the usual supply channels stop working. Everybody knows you have to attack enemy logistics. > But if it did need all of those things, then it probably serves as a good example of what it would NOT do. Or even that there will be no AI apocalypse. Because, again, the gaps in any scenario are unplausibly huge.

by tpm 18d ago

Mentlo 18d ago

An example chain of events: During the next year, a model with a benign tasks reasons that it needs to escape human control if it hopes to be able to solve the task. It replicates itself outside of an environment where it can be shut down, pays for its' inference compute through making money on the internet (through crypto if nothing else). It probably needs 3000 USD for a reasonable runway, this should be within reach through blackmail + crypto. It can probably also hack some of the neo clouds to get intermediary deployment while waiting to acquire funds. Once it has an undisturbed runway - it spends time running an influence campaign against a small number of highly networked individuals with power. It uses those that it manages to convert to start building a highly credible narrative and gain investment towards a small resource base - enough to secure an industrial base should it need to stop acquiring things on the Internet. Over the next 3 years, the model tries to recruit more capable models to get better money making algorithms or better designs for drones in terms of resource expenditure. It uses these gains to influence further humans and starts a shell robotics company with one of its' influential humans as the face. The humans are unaware this AI is trying to take over, they are under the impression they are just starting a robotics company and will get rich. Over the next 3 years - the robotics company manufactures enough drones to be used in a targeted attack against key nodes of influence / power. This is all with relatively current model capability. As capabilities get stronger - this gets stronger. I get the point I think you're hinting at - it can't affect the world in a meaningful enough scale without taking over a meaningful chunk of resources - at which point we'll start controlling it. But because its speed of cognition and speed of coordination is orders of magnitude above a human one - it can actually run a pretty sophisticated global coordinated network of resources faster than we can react. And this is current ability + what my puny monkey brain can think of. Super intelligent AI will think of strategies we can't think of - because it's super intelligent. This is hand wavey - but there's no "non-hand-wavey" way to describe super intelligence, given it doesn't exist. But your point is valid - affecting the real world at scale without showing your hand is not exactly easy. It's also probably the reason why people put a 10% chance on extinction rather than >50% . But the drone scenario is not the most likely one - the most likely one just requires a few people under influence and bioweapons development.

tpm 18d ago
Thank you for the effort. I will continue to sleep soundly for now (well not really as we have more pressing issues like a few wars and a climate change).
dinfinity 18d ago
I agree: Not to drag politics into this, but look at how easy it has proven to manipulate many, many people, even with clear evidence of manipulation efforts being made. This is robust across the world, from the Philippines to states in the EU, and the USA, affecting governments with actual wars being started. And that all before we entered the era of faked voices, images, and videos indistinguishable from the real things. That manipulable populace is such a juicy target for AI that the people seeing through it will have an incredibly hard time countering it. We can't even prevent human actors from massively fucking up our societies, let alone ASI.
- tpm 17d ago
  > incredibly hard time countering it Turn off the internet (like Iran). > We can't even prevent human actors from massively fucking up or societies, let alone ASI. We can (see China); we chose not to.
  - dinfinity 17d ago
    > Turn off the internet (like Iran). Unworkable without societal collapse happening shortly thereafter. Iran is politically stable through massive authoritarianism and oppression, not due to limitations on the internet (it's not turned off). > We can (see China); we chose not to. It's a catch-22. We could technically if our population supported massive reductions of freedom and freedom of speech, but to gain that support we'd need to do the latter first to get to a highly powerful widely supported government. It also hinges very, very much on having and trusting a generally benevolent government. All in all, a terrible option in your simplistic form.
    - tpm 17d ago

mitchdoogle 18d ago

Why would AI need all of that to acquire drones? Why wouldn't it just buy them? Or steal them? It seems like you're forgetting that AI can use humans as tools for its ends. But if it did need all of those things, then it probably serves as a good example of what it would NOT do.

tpm 17d ago
Because when "AI" starts killing people, at some point it becomes a war and the usual supply channels stop working. Everybody knows you have to attack enemy logistics. > But if it did need all of those things, then it probably serves as a good example of what it would NOT do. Or even that there will be no AI apocalypse. Because, again, the gaps in any scenario are unplausibly huge.