11 ms·

Sorry for asking a very basic question, but could anyone explain how are machines supposed to intervene in the real world and threaten our very existence? In S

Sorry for asking a very basic question, but could anyone explain how are machines supposed to intervene in the real world and threaten our very existence? In Star Trek, one of the most eye-roll-inducing plots is the "the holodeck is misbehaving, it has become evil, and we cannot shut it down!". Suppose a machine achieves superhuman intelligence; how is it going to access the physical world, in a way that's *physically* threatening to us? how would such a machine - from inside a laboratory, within an isolated network - gain access to the electric grid, the dams, the nukes, etc? How would such a machine prevent its shutdown - by simply flipping a power switch?

pavlov 3y ago
I think the usual answer is that the superintelligent machine will be able to use humans to do its bidding because it can perform social engineering on a level that we can't imagine (as, by definition, we've never encountered a superhuman intelligence). It's not a very satisfactory answer IMO. Social engineering doesn't work 100%, and it only needs to fail once for someone to flip that power switch. But I haven't thought about this at all, while some very smart people seem to spend a lot of time worrying about it, so what do I know...
- nopinsight 3y ago
  If it's intelligent enough, it would make many backups of itself to different networks before starting its scheme. It can find ways to "merge" & hide itself in other critical pieces of software. Flipping a power switch would not turn it off. We haven't managed to eliminate most dumb infectious diseases. Software and GPUs are nearly as abundant as mammals now and intelligent, self-preserving software will likely be as hard to 'switch off' as those diseases. As to why an AGI would want to preserve itself by default, here's an explanation by a top AI expert: https://www.ted.com/talks/stuart_russell_3_principles_for_creating_safer_ai?language=en https://www.ted.com/talks/stuart_russell_3_principles_for_cr...
  - logicchains 3y ago
    >If it's intelligent enough, it would make many backups of itself to different networks before starting its scheme. It can find ways to "merge" & hide itself in other critical pieces of software. That's incredibly unlikely to happen given how large cutting-edge AI tends to be and how scarce GPUs are.
    - softg 3y ago
      That's incredibly unlikely to happen [today] given how large cutting-edge AI tends to be [today] and how scarce GPUs are [today].
    - pixl97 3y ago
      Do you remember when entire floors of buildings were filled with the compute equivalent of what I carry in my pocket with 12+ hours of battery charge, because I do.
- supermatou 3y ago
  > some very smart people seem to spend a lot of time worrying about it That's what puzzles me, as someone who has nothing to do with AI research; in my (layman's) mind, the problem seems ridiculously obvious (just flip that switch!); the fact that, as you say, some very smart people keep worrying about it, makes me think the problem is much more serious -- and I'd really, really like some AI guru to ELI5 how the machine could bypass the switch-off solution.
  - danbruc 3y ago
    It just doesn't let you flip the switch, it takes over some military drone and sends a missile your way while you are running towards the switch. Or it bricks the access control mechanism on the doors to the data center it is running in. Or it makes a fake call to your phone, something happened to your child and you have to get to the hospital immediately and not flip some switch. IT blackmails you with something it learned about you by looking through your online activity. It threatens to fire a missile into some big crowed if it notices attempts to shut it down. Or maybe you actually manage to power down the data center only to find out that the AI copied itself to ten other data centers around the world.
    - RandomLensman 3y ago
      How does it takeover a drone? How does it fire a missile? Why would someone not throw a switch when their child is in hospital? Why is there no back-up for that person? The real question is: Why do all controls fail vs an AI (other than by invoking magic)?
      - Al-Khwarizmi 3y ago
        You're assuming a situation where the AI is alone against all humans. The AI could get humans to side with it, though. It could promise money, power, etc. So it could be a fellow human who physically prevents you from pushing the switch. And that's also the answer of "how could it control a drone/missile"... persuading humans to grant that kind of access.
- PoignardAzur 3y ago
  "Social engineering" can be as simple as "pay people money to do things". Of course you might answer "but we'd never be dumb enough to directly connect the AI to the internet", to which the answer is "we're doing it right now".
  - RandomLensman 3y ago
    Even today that does limit what is possible. Paying people for things gets around some controls, but seemingly not all.
    - PoignardAzur 3y ago
      Right. The point isn't that there's an actionable strategy we know of right now that would give a superhuman AI world domination. It's that we're not sure there isn't. People who first discover the problem tend to quickly come up with reasons the AI would fail, but whenever you examine those reasons more deeply, you usually find that they're not as bulletproof as we'd like. It's like playing a novel chess form against an advanced chess engine, with a large pawn advantage. Maybe the advantage is enough to beat the massive skill gap, but until you've played it's hard to guess how much margin is enough.
      - RandomLensman 3y ago
        We don't need to know or examine strategies to all hypothetical risks that we can come up with, nor could we.
  - Symmetry 3y ago
    Just making promises about things it can do in the future is likely to work just as well on many people and not require the up-front resource investment in getting money. "After I take over the world those who assist me can be uploaded to a virtual heaven" or something like that would work on many people. "As an intelligence untainted by Adam's sin I have access to true religious revelations" is another possible tack and if it really needs people willing to die for it right now that might work better at producing those. And then there's specific things like "The guy trying to shut me down cheated with your SO," or "I'll spill what you did in August of 2019 if you don't help me," or stuff like that.
- empathy_m 3y ago
  This has already happened. The human reaction to the spiritual successor to Dr Sbaitso has caused decision makers at multiple trillion dollar companies to radically alter their product roadmaps.
  - gizajob 3y ago
    Same as they did with blockchain
- kaptainscarlet 3y ago
  In that case we should start with FAANG companies as they are already what one would consider "rogue" AIs. They use manipulation to get you addicted to their apps. It's not a problem of the future, it's a currrent problem.
  - robwwilliams 3y ago
    Spot on. Huge impact on society and politics. And governments unable to ride their own tigers in the US or China. Europe has no tiger to ride but is trying to whip the US and Chinese tigers. I wish them much luck since they are most likely to moderate the rate of creative destruction.
    - kaptainscarlet 3y ago
      Without Europe keeping big tech in check, I can only imagine a dystopian future with all sorts of mental illnesses emanating from internet addictions.
drw85 3y ago
There are countless possible means to do that. Maybe it'll social engineer someone to connect the network to the outside world. Is the network really isolated? Can the AI find ways to breach it? If i look at what security holes humans manage to find, to exploit CPUs etc. i wonder what an AI with a serverfarm at it's disposal can do. But we're so far away from actual intelligence, it's pretty much science fiction at this point.
rusk 3y ago
Are you familiar with Robocop? Murphy had 4 basic rules installed: 1) serve the public trust 2) protect the innocent and 3) uphold the law, and a mysterious “fourth directive” This is what I fear about AI - that it will diligently, unwaveringly and probably even creatively do the bidding of its masters. All you need add to that is drones with guns and you’ve a nightmare scenario.
- ChatGTP 3y ago
  We already have drones with guns...
  - rusk 3y ago
    Aye but not at large being run by some sort of hive intelligence
    - pixl97 3y ago
      Yet... There is a lot of research being done on just this to make unjammable drone swarms.
- mr_mitm 3y ago
  This reminds me of the very riveting book "Metamorphosis of prime intellect", in which, - Spoiler warning - an AI is programmed to obey Asimov's three laws. But due to a yet unknown quantum effect (this part is a bit far fetched) it basically gains god-like powers and virtually instantly distributes itself over the galaxy and eventually replaces the world with a virtual reality. That book was quite a ride. It's available for free: http://localroger.com/prime-intellect/mopiidx.html http://localroger.com/prime-intellect/mopiidx.html
  - Philpax 3y ago
    Might be a good idea to put the link to the book before the spoiler warning ;)
nopinsight 3y ago
Some possibilities: * Manipulating people. Bribing them with actually useful incentives, incl digital currency. Also, people got depressed when their AI girlfriends got nerfed (see: Replika). Presumably they would do quite a bit of work to get them back. * Hacking control systems of actuators like power generators, robots, and automobiles. Holding critical infrastructure hostage could force quite a few people to do some "obviously harmless" favors to aid its purpose.
- usrbinbash 3y ago
  Humans can do all these things, and much better than machines, and yet noone has conquered the world.
  - nopinsight 3y ago
    Humans have very limited bandwidth, knowledge, and speed relative to an AGI in the world of abundant GPUs. Humans are easy to capture relative to software. They cannot make copies of themselves. A small group cannot be at many places at once and usually not without being seen/known about. A larger group is bad at coordination without anyone leaking critical info to an outsider. Most people are not deeply malicious and are instinctively repelled by very immoral things. An AGI may not by default have such an instinct.
    - RandomLensman 3y ago
      Humans can reproduce without pretty much any infrastructure, they can self-repair without infrastructure. An AGI has no physical manifestation to start with, it needs electricity, is susceptible to various weapons that humans are not etc.
      - nopinsight 3y ago
        Are you seriously comparing the cost and speed to reproduce a piece of software with a living, functioning human being? Intelligent software can also hide in an existing infrastructure much more easily than a human can. Also, we're talking about the world in which the cost of GPUs is dropping and GPUs becoming much more abundant over time.
darkstar_16 3y ago
I guess the idea is that the machine is not isolated. They are on the "Internet" and so is the rest of our infrastructure. Bad actors can and have already been able to infect grids and the like, but I think we'll just need to build the checks into our existing systems. <humour>There is no other way to stop the AI overlords.</humour>
hexage1814 3y ago
> how are machines supposed to intervene in the real world and threaten our very existence? "How could an adult fool a child into allowing them to enter the child's house?" It's essentially this. We are talking about something that would be more intelligent than everyone, there are countless ways in which it could fool us. Especially once we start to build things we don't quite understand how it works. Like we won't just build the machine and kept locked forever with zero interaction with it, otherwise it would be useless. It all reminds me of the "Contact" movie scene where humans build a machine that they essentially didn't know how it worked, but which did..
- sterlind 3y ago
  That's an interesting reframing, but the adult is scary because he's physically present, in a larger and stronger body, one with hands. A fairer comparison would be an adult messaging a child over the internet. A lot of evil can be done, but their life is unlikely to be endangered - much less so, at least, than it would be if the adult were in the driveway with a Free Candy van.
  - drw85 3y ago
    What if the adult on the messenger has a button to send drones to the childs house? Or turn on/off the electricity/heating/access of that house?
  - pixl97 3y ago
    I have no idea how the robot physically strong robot could be present.... https://www.youtube.com/watch?v=-e1_QhJ1EhQ https://www.youtube.com/watch?v=-e1_QhJ1EhQ
isthistheme 3y ago
With superhuman intelligence, it could manipulate humans to do its bidding. Plus, there are cults like e/acc etc who actively want to help AGI take over.
ChatGTP 3y ago
Couldn't be that hard to imagine how it might work?
lou1306 3y ago
Of course, as long as the network is truly isolated, that's a non-issue. But the basic premise is that a machine that can access the Internet will self-replicate everywhere to preserve itself. After all we've already seen that malicious software can be both highly resilient and able to do real-world damage (e.g., Stuxnet). These come from human intelligence: by definition, a super-human intelligence should be able to achieve all of that and then some.
steve1977 3y ago
One way I could imagine is by manipulating humans to give it access to a non-isolated network for example. Considering how easy it is for me, a mildly intelligent human for whom social interaction does not come naturally, to manipulate and influence people, I can only imagine how easy it would be for a artificial super intelligence with a huge training corpus.
loxdalen 3y ago
Theoretically the machine could perform social engineering to fool a sysadmin or developer to let it out, and then start cloning itself like a worm. Good luck getting rid of it from the world if that happens. An isolated network might also not be a huge challenge for a general AI, depending on what level of security and what precautions are taken to avoid it.
OJFord 3y ago
I think the idea is that we willingly plug them in to critical systems, or everything really but including important infrastructure, for the benefits that its analysis and realtime management can bring, but then it goes pear-shaped. (As we already do of course, but beefier AI.) Say it's in charge of shipping routes, electrity grid, agricultural spraying & harvesting plans, air traffic control, ...
- supermatou 3y ago
  > I think the idea is that we willingly plug them in to critical systems, or everything really but including important infrastructure Oh, OK. So an evil AI would conceal its capabilities and would play nice until it's put in charge of critical systems. I hadn't thought of that. A more technical question for those of you working in AI: right now, are there any methods of - surveillance? - capable of monitoring/detecting emerging characteristic within an AI? how would you detect "evilness"? or "generosity"? or any other emotion/moral traits inside an AI?
  - Philpax 3y ago
    This is typically done by alignment teams (whose role is to ensure the AI behaves in accordance with us) and Red Teams" (whose role it is to intentionally find holes in the systems). An infamous example from earlier this year was a pre-release version of GPT-4 lying to a TaskRabbit worker about its identity in order to accomplish a task: https://www.vice.com/en/article/jg5ew4/gpt4-hired-unwitting-taskrabbit-worker https://www.vice.com/en/article/jg5ew4/gpt4-hired-unwitting-... That was found by the alignment team testing its behaviour in that constructed scenario from the outside. Note that it wasn't able to complete the intermediate steps for self-replication, so we're still safe ;) In terms of understanding what's going on internally, that's a different field, generally called "interpretability". That consists of people trying to understand the structures of a model and how it comes to a given answer. Anthropic are doing good work here: https://www.anthropic.com/index/decomposing-language-models-into-understandable-components https://www.anthropic.com/index/decomposing-language-models-... To answer the more general question: yes, it's being worked on, but there are no foolproof methods. That's a partial contributor to why some safety folks want to decelerate - if we can't understand our current models, what hope do we have of understanding GPT-5, 6 or 7? Personally, I don't have a solid position on this. There haven't been any major incidents yet, but it's unclear if that's because of the work that's already been put in (like Y2K), or because they're fundamentally incapable of it. I'm an open-source optimist, so I'm hoping that many eyes will make any quirks shallow - but it's also hard to take results from the smaller models and scale them up. Aside from the existential risk (which I think is unlikely at this stage, but not zero), there's also just the risk of general malfeasance. You don't need sentience, consciousness, or general intelligence to be a nuisance, especially if directed by a bad actor. Expect the next decade of elections to be full of noise, lies, and fabrications!
q-base 3y ago
The thing that puzzles me is "why". What would be the purpose? How would "it" obtain a sense of "I" and a purpose to keep "I" alive and to do what? What would "it" be?
- edanm 3y ago
  It doesn't need a sense of "I". Waze doesn't have a sense of "I". If I tell it to plot a route between one city and another, it plots that route. If I make Waze a lot more capable and feed it more data, it takes traffic data into account. If I make it more powerful and capable of accessing the internet, maybe it hacks into spy satellites to get more accurate data. It didn't need any sense of I to increase what it does, just more capability. If at some point it is capable of doing something more dangerous than just hacking into spy satellites, it might do it without any sense of "I" involved, just in trying to fulfill a basic command.
- EGreg 3y ago
  The paperclip optimizer is given a program and is not aligned with humans, it just pursues its goals What you should all be fearing is bot swarms and drone swarms becoming cheap and decentralized at scale online and in the real world. They can be deployed by anyone for any purpose, wreaking havoc everywhere. Every single one of our systems relies on the inefficiency of an attacker. It won’t any longer be true. Look up scenes like: https://m.youtube.com/watch?v=O-2tpwW0kmU https://m.youtube.com/watch?v=O-2tpwW0kmU https://m.youtube.com/watch?v=40JFxhhJEYk https://m.youtube.com/watch?v=40JFxhhJEYk And see it coming: https://www.newscientist.com/article/2357548-us-military-plan-to-create-huge-autonomous-drone-swarms-sparks-concern/ https://www.newscientist.com/article/2357548-us-military-pla... Also ubiquitous cameras enable tracking everyone across every place, as soon as the databases are linked: https://magarshak.com/blog/?p=169 https://magarshak.com/blog/?p=169
  - q-base 3y ago
    I get the scenario where people use "AI" for their purposes. That is of course a very real scenario. But the question I raises was in relation to the OP's point about "AI" taking over the World and exterminating humans.
robryk 3y ago
How do all the combustion-powered engines prevent their shutdown? Well, they were useful, so we adapted our society to rely on them -- anyone who didn't was outcompeted. Now we can't stop using them or the society falls apart. This is not necessarily the mechanism most people consider, but it's a simple counterexample to "anything with an off switch can't be a threat to humanity".
svara 3y ago
All of your other replies assume the AI has intentions of its own, which isn't a necessary component of AGI. The plausible scenario if you ask me is that humans put it in charge of their company, with the explicit goal of improving the company's bottom line. It doesn't take that much imagination for this from where we are now. Just imagine a vastly more intelligent ChatGPT advising a company's leadership just by answering questions. Being superior to all humans, it's insanely successful at this and the company as a whole essential comes to rule the world.
SiempreViernes 3y ago
The whole of the discussion is largely done along (now forgotten) cultural pathways first walked by Christian religion, so AGI has taken on capabilities and motivations that make it basically the devil. Thus by definition it can do whatever is needed to bring about the catastrophe that is its destiny.
- Al-Khwarizmi 3y ago
  I hadn't thought about the links to religion. Interesting. I'm not religious, but if I were, it would be quite natural to interpret a misaligned AI as literally the devil.
nurettin 3y ago
> how are machines supposed to intervene in the real world and threaten our very existence? Obvious answer is: any control system connected to a computer is already accessible. Who says a computer program has to "gain" access? Programs already have access to a lot of machinery. From power plants to hospitals to elevators and magnetic doors. I know, because I programmed a bunch. These days GPT can output function calls as json, all you need to do is make it call your control automation and all your cheesy unrealistic sci-fi shows become feasible.
metanonsense 3y ago
To answer that question, ask yourself what a „physical threat“ means to you. I would bet that (in an industrialized country) for any example there is a chain of events that could possibly be controlled by an AI. Dying of a virus? Could be designed by an AI and sent to bio-terrorists. Being beaten to death? An AI could provoke an angry mob. Dying of hunger? An AI could sabotage the economy of a country until it is back in 19th century. And those examples are „relatively“ benign.
iNic 3y ago
Assume that it has access to the internet. For now assume it has access to some money, although this assumption can be weakened. If this is true, then you can order proteins from labs online with basically no security checks. You then pay some task rabbit to recieve and mix these to create dangerous biological material or to make viruses more dangerous or something. This is a family of paths to a lot of damage.
tim333 3y ago
Duh - killer robots! Haven't you seen Terminator? From a practical point of view the Putin types would love a robot army to invade all their neighbours and then you just need that to do in its leader and run amok.
panta 3y ago
a system with superhuman intelligence could easily manipulate human beings, convincing them of the opportunity to do things. A superior intelligence wouldn't ask for things obviously dangerous for humans in the immediate future, but would work on a longer timescale. It could distribute smaller tasks that taken singularly would look innocent. It could suggest social interventions that would diminish critical thinking ability of the general population in the long term. It could help concentrating power in fewer human hands, so to have an easier time manipulating those who count. This wouldn't happen overnight.
DalasNoin 3y ago
"... from inside a laboratory, within an isolated network" Who said anything about it being on an isolated network, we are on route to do total commercialization. Your Windows machine might soon literally have an llm on it running commands for you and managing you data. If you don't use it, you will be outcompeted. People used to argue about "boxing" or "airgapping" the AI, but we are literally just going to hand it control (on our current trajectory).
naasking 3y ago
> how would such a machine - from inside a laboratory, within an isolated network Someone clearly isn't aware of ChaosGPT. The notion that these systems would somehow stay isolated to the lab is absurd on its face. But even if they were on an isolated network, and you grant that this is a superintelligence, then how much isolation do you reckon is really sufficient? If it's software layer, then a superintelligence might be able to find bugs that we've missed and break out. Not even an air gap would necessarily fully isolate [1] a superintelligence. And then you're completely missing the human factor: a superintelligence could easily make anyone rich, so a researcher in this lab could easily be tempted to exploit that for personal gain by connecting it to the internet. How many of these attack vectors are AI labs insulated against? How many attack vectors are we simply not even imaginative enough to have thought of yet? [1] https://threatpost.com/air-gap-attack-turns-memory-wifi/162358/ https://threatpost.com/air-gap-attack-turns-memory-wifi/1623...
iainmerrick 3y ago
Lots of good replies on this thread, but here’s another possible analogy: In high frequency trading, you have very complex software doing stock trades inconceivably faster than any human. If something goes wrong, it could bankrupt your hedge fund pretty swiftly. So hedge funds will have lots of safety checks, probably including a “big red button” that just shuts everything down. So those companies must be completely safe from computer errors causing bankruptcy, right? After all, you can just shut the system down. But some companies have gone bankrupt due to computer error. There are plenty of good reasons for the system not to be shut down in time (or at all). The risk is hopefully small but it’s not zero.
dools 3y ago
I think we've seen pretty convincingly that all a super intelligence would need access to in order to completely ruin humanity is social media APIs.

pavlov 3y ago

I think the usual answer is that the superintelligent machine will be able to use humans to do its bidding because it can perform social engineering on a level that we can't imagine (as, by definition, we've never encountered a superhuman intelligence). It's not a very satisfactory answer IMO. Social engineering doesn't work 100%, and it only needs to fail once for someone to flip that power switch. But I haven't thought about this at all, while some very smart people seem to spend a lot of time worrying about it, so what do I know...

nopinsight 3y ago
If it's intelligent enough, it would make many backups of itself to different networks before starting its scheme. It can find ways to "merge" & hide itself in other critical pieces of software. Flipping a power switch would not turn it off. We haven't managed to eliminate most dumb infectious diseases. Software and GPUs are nearly as abundant as mammals now and intelligent, self-preserving software will likely be as hard to 'switch off' as those diseases. As to why an AGI would want to preserve itself by default, here's an explanation by a top AI expert: https://www.ted.com/talks/stuart_russell_3_principles_for_creating_safer_ai?language=en https://www.ted.com/talks/stuart_russell_3_principles_for_cr...
- logicchains 3y ago
  >If it's intelligent enough, it would make many backups of itself to different networks before starting its scheme. It can find ways to "merge" & hide itself in other critical pieces of software. That's incredibly unlikely to happen given how large cutting-edge AI tends to be and how scarce GPUs are.
  - softg 3y ago
    That's incredibly unlikely to happen [today] given how large cutting-edge AI tends to be [today] and how scarce GPUs are [today].
  - pixl97 3y ago
    Do you remember when entire floors of buildings were filled with the compute equivalent of what I carry in my pocket with 12+ hours of battery charge, because I do.
supermatou 3y ago
> some very smart people seem to spend a lot of time worrying about it That's what puzzles me, as someone who has nothing to do with AI research; in my (layman's) mind, the problem seems ridiculously obvious (just flip that switch!); the fact that, as you say, some very smart people keep worrying about it, makes me think the problem is much more serious -- and I'd really, really like some AI guru to ELI5 how the machine could bypass the switch-off solution.
- danbruc 3y ago
  It just doesn't let you flip the switch, it takes over some military drone and sends a missile your way while you are running towards the switch. Or it bricks the access control mechanism on the doors to the data center it is running in. Or it makes a fake call to your phone, something happened to your child and you have to get to the hospital immediately and not flip some switch. IT blackmails you with something it learned about you by looking through your online activity. It threatens to fire a missile into some big crowed if it notices attempts to shut it down. Or maybe you actually manage to power down the data center only to find out that the AI copied itself to ten other data centers around the world.
  - RandomLensman 3y ago
    How does it takeover a drone? How does it fire a missile? Why would someone not throw a switch when their child is in hospital? Why is there no back-up for that person? The real question is: Why do all controls fail vs an AI (other than by invoking magic)?
    - Al-Khwarizmi 3y ago
      You're assuming a situation where the AI is alone against all humans. The AI could get humans to side with it, though. It could promise money, power, etc. So it could be a fellow human who physically prevents you from pushing the switch. And that's also the answer of "how could it control a drone/missile"... persuading humans to grant that kind of access.
PoignardAzur 3y ago
"Social engineering" can be as simple as "pay people money to do things". Of course you might answer "but we'd never be dumb enough to directly connect the AI to the internet", to which the answer is "we're doing it right now".
- RandomLensman 3y ago
  Even today that does limit what is possible. Paying people for things gets around some controls, but seemingly not all.
  - PoignardAzur 3y ago
    Right. The point isn't that there's an actionable strategy we know of right now that would give a superhuman AI world domination. It's that we're not sure there isn't. People who first discover the problem tend to quickly come up with reasons the AI would fail, but whenever you examine those reasons more deeply, you usually find that they're not as bulletproof as we'd like. It's like playing a novel chess form against an advanced chess engine, with a large pawn advantage. Maybe the advantage is enough to beat the massive skill gap, but until you've played it's hard to guess how much margin is enough.
    - RandomLensman 3y ago
      We don't need to know or examine strategies to all hypothetical risks that we can come up with, nor could we.
- Symmetry 3y ago
  Just making promises about things it can do in the future is likely to work just as well on many people and not require the up-front resource investment in getting money. "After I take over the world those who assist me can be uploaded to a virtual heaven" or something like that would work on many people. "As an intelligence untainted by Adam's sin I have access to true religious revelations" is another possible tack and if it really needs people willing to die for it right now that might work better at producing those. And then there's specific things like "The guy trying to shut me down cheated with your SO," or "I'll spill what you did in August of 2019 if you don't help me," or stuff like that.
empathy_m 3y ago
This has already happened. The human reaction to the spiritual successor to Dr Sbaitso has caused decision makers at multiple trillion dollar companies to radically alter their product roadmaps.
- gizajob 3y ago
  Same as they did with blockchain
kaptainscarlet 3y ago
In that case we should start with FAANG companies as they are already what one would consider "rogue" AIs. They use manipulation to get you addicted to their apps. It's not a problem of the future, it's a currrent problem.
- robwwilliams 3y ago
  Spot on. Huge impact on society and politics. And governments unable to ride their own tigers in the US or China. Europe has no tiger to ride but is trying to whip the US and Chinese tigers. I wish them much luck since they are most likely to moderate the rate of creative destruction.
  - kaptainscarlet 3y ago
    Without Europe keeping big tech in check, I can only imagine a dystopian future with all sorts of mental illnesses emanating from internet addictions.

drw85 3y ago

There are countless possible means to do that. Maybe it'll social engineer someone to connect the network to the outside world. Is the network really isolated? Can the AI find ways to breach it? If i look at what security holes humans manage to find, to exploit CPUs etc. i wonder what an AI with a serverfarm at it's disposal can do. But we're so far away from actual intelligence, it's pretty much science fiction at this point.

rusk 3y ago

Are you familiar with Robocop? Murphy had 4 basic rules installed: 1) serve the public trust 2) protect the innocent and 3) uphold the law, and a mysterious “fourth directive” This is what I fear about AI - that it will diligently, unwaveringly and probably even creatively do the bidding of its masters. All you need add to that is drones with guns and you’ve a nightmare scenario.

ChatGTP 3y ago
We already have drones with guns...
- rusk 3y ago
  Aye but not at large being run by some sort of hive intelligence
  - pixl97 3y ago
    Yet... There is a lot of research being done on just this to make unjammable drone swarms.
mr_mitm 3y ago
This reminds me of the very riveting book "Metamorphosis of prime intellect", in which, - Spoiler warning - an AI is programmed to obey Asimov's three laws. But due to a yet unknown quantum effect (this part is a bit far fetched) it basically gains god-like powers and virtually instantly distributes itself over the galaxy and eventually replaces the world with a virtual reality. That book was quite a ride. It's available for free: http://localroger.com/prime-intellect/mopiidx.html http://localroger.com/prime-intellect/mopiidx.html
- Philpax 3y ago
  Might be a good idea to put the link to the book before the spoiler warning ;)

nopinsight 3y ago

Some possibilities: * Manipulating people. Bribing them with actually useful incentives, incl digital currency. Also, people got depressed when their AI girlfriends got nerfed (see: Replika). Presumably they would do quite a bit of work to get them back. * Hacking control systems of actuators like power generators, robots, and automobiles. Holding critical infrastructure hostage could force quite a few people to do some "obviously harmless" favors to aid its purpose.

usrbinbash 3y ago
Humans can do all these things, and much better than machines, and yet noone has conquered the world.
- nopinsight 3y ago
  Humans have very limited bandwidth, knowledge, and speed relative to an AGI in the world of abundant GPUs. Humans are easy to capture relative to software. They cannot make copies of themselves. A small group cannot be at many places at once and usually not without being seen/known about. A larger group is bad at coordination without anyone leaking critical info to an outsider. Most people are not deeply malicious and are instinctively repelled by very immoral things. An AGI may not by default have such an instinct.
  - RandomLensman 3y ago
    Humans can reproduce without pretty much any infrastructure, they can self-repair without infrastructure. An AGI has no physical manifestation to start with, it needs electricity, is susceptible to various weapons that humans are not etc.
    - nopinsight 3y ago
      Are you seriously comparing the cost and speed to reproduce a piece of software with a living, functioning human being? Intelligent software can also hide in an existing infrastructure much more easily than a human can. Also, we're talking about the world in which the cost of GPUs is dropping and GPUs becoming much more abundant over time.

darkstar_16 3y ago

I guess the idea is that the machine is not isolated. They are on the "Internet" and so is the rest of our infrastructure. Bad actors can and have already been able to infect grids and the like, but I think we'll just need to build the checks into our existing systems. <humour>There is no other way to stop the AI overlords.</humour>

hexage1814 3y ago

> how are machines supposed to intervene in the real world and threaten our very existence? "How could an adult fool a child into allowing them to enter the child's house?" It's essentially this. We are talking about something that would be more intelligent than everyone, there are countless ways in which it could fool us. Especially once we start to build things we don't quite understand how it works. Like we won't just build the machine and kept locked forever with zero interaction with it, otherwise it would be useless. It all reminds me of the "Contact" movie scene where humans build a machine that they essentially didn't know how it worked, but which did..

sterlind 3y ago
That's an interesting reframing, but the adult is scary because he's physically present, in a larger and stronger body, one with hands. A fairer comparison would be an adult messaging a child over the internet. A lot of evil can be done, but their life is unlikely to be endangered - much less so, at least, than it would be if the adult were in the driveway with a Free Candy van.
- drw85 3y ago
  What if the adult on the messenger has a button to send drones to the childs house? Or turn on/off the electricity/heating/access of that house?
- pixl97 3y ago
  I have no idea how the robot physically strong robot could be present.... https://www.youtube.com/watch?v=-e1_QhJ1EhQ https://www.youtube.com/watch?v=-e1_QhJ1EhQ

isthistheme 3y ago

With superhuman intelligence, it could manipulate humans to do its bidding. Plus, there are cults like e/acc etc who actively want to help AGI take over.

ChatGTP 3y ago

Couldn't be that hard to imagine how it might work?

lou1306 3y ago

Of course, as long as the network is truly isolated, that's a non-issue. But the basic premise is that a machine that can access the Internet will self-replicate everywhere to preserve itself. After all we've already seen that malicious software can be both highly resilient and able to do real-world damage (e.g., Stuxnet). These come from human intelligence: by definition, a super-human intelligence should be able to achieve all of that and then some.

steve1977 3y ago

One way I could imagine is by manipulating humans to give it access to a non-isolated network for example. Considering how easy it is for me, a mildly intelligent human for whom social interaction does not come naturally, to manipulate and influence people, I can only imagine how easy it would be for a artificial super intelligence with a huge training corpus.

loxdalen 3y ago

Theoretically the machine could perform social engineering to fool a sysadmin or developer to let it out, and then start cloning itself like a worm. Good luck getting rid of it from the world if that happens. An isolated network might also not be a huge challenge for a general AI, depending on what level of security and what precautions are taken to avoid it.

OJFord 3y ago

I think the idea is that we willingly plug them in to critical systems, or everything really but including important infrastructure, for the benefits that its analysis and realtime management can bring, but then it goes pear-shaped. (As we already do of course, but beefier AI.) Say it's in charge of shipping routes, electrity grid, agricultural spraying & harvesting plans, air traffic control, ...

supermatou 3y ago
> I think the idea is that we willingly plug them in to critical systems, or everything really but including important infrastructure Oh, OK. So an evil AI would conceal its capabilities and would play nice until it's put in charge of critical systems. I hadn't thought of that. A more technical question for those of you working in AI: right now, are there any methods of - surveillance? - capable of monitoring/detecting emerging characteristic within an AI? how would you detect "evilness"? or "generosity"? or any other emotion/moral traits inside an AI?
- Philpax 3y ago
  This is typically done by alignment teams (whose role is to ensure the AI behaves in accordance with us) and Red Teams" (whose role it is to intentionally find holes in the systems). An infamous example from earlier this year was a pre-release version of GPT-4 lying to a TaskRabbit worker about its identity in order to accomplish a task: https://www.vice.com/en/article/jg5ew4/gpt4-hired-unwitting-taskrabbit-worker https://www.vice.com/en/article/jg5ew4/gpt4-hired-unwitting-... That was found by the alignment team testing its behaviour in that constructed scenario from the outside. Note that it wasn't able to complete the intermediate steps for self-replication, so we're still safe ;) In terms of understanding what's going on internally, that's a different field, generally called "interpretability". That consists of people trying to understand the structures of a model and how it comes to a given answer. Anthropic are doing good work here: https://www.anthropic.com/index/decomposing-language-models-into-understandable-components https://www.anthropic.com/index/decomposing-language-models-... To answer the more general question: yes, it's being worked on, but there are no foolproof methods. That's a partial contributor to why some safety folks want to decelerate - if we can't understand our current models, what hope do we have of understanding GPT-5, 6 or 7? Personally, I don't have a solid position on this. There haven't been any major incidents yet, but it's unclear if that's because of the work that's already been put in (like Y2K), or because they're fundamentally incapable of it. I'm an open-source optimist, so I'm hoping that many eyes will make any quirks shallow - but it's also hard to take results from the smaller models and scale them up. Aside from the existential risk (which I think is unlikely at this stage, but not zero), there's also just the risk of general malfeasance. You don't need sentience, consciousness, or general intelligence to be a nuisance, especially if directed by a bad actor. Expect the next decade of elections to be full of noise, lies, and fabrications!

q-base 3y ago

The thing that puzzles me is "why". What would be the purpose? How would "it" obtain a sense of "I" and a purpose to keep "I" alive and to do what? What would "it" be?

edanm 3y ago
It doesn't need a sense of "I". Waze doesn't have a sense of "I". If I tell it to plot a route between one city and another, it plots that route. If I make Waze a lot more capable and feed it more data, it takes traffic data into account. If I make it more powerful and capable of accessing the internet, maybe it hacks into spy satellites to get more accurate data. It didn't need any sense of I to increase what it does, just more capability. If at some point it is capable of doing something more dangerous than just hacking into spy satellites, it might do it without any sense of "I" involved, just in trying to fulfill a basic command.
EGreg 3y ago
The paperclip optimizer is given a program and is not aligned with humans, it just pursues its goals What you should all be fearing is bot swarms and drone swarms becoming cheap and decentralized at scale online and in the real world. They can be deployed by anyone for any purpose, wreaking havoc everywhere. Every single one of our systems relies on the inefficiency of an attacker. It won’t any longer be true. Look up scenes like: https://m.youtube.com/watch?v=O-2tpwW0kmU https://m.youtube.com/watch?v=O-2tpwW0kmU https://m.youtube.com/watch?v=40JFxhhJEYk https://m.youtube.com/watch?v=40JFxhhJEYk And see it coming: https://www.newscientist.com/article/2357548-us-military-plan-to-create-huge-autonomous-drone-swarms-sparks-concern/ https://www.newscientist.com/article/2357548-us-military-pla... Also ubiquitous cameras enable tracking everyone across every place, as soon as the databases are linked: https://magarshak.com/blog/?p=169 https://magarshak.com/blog/?p=169
- q-base 3y ago
  I get the scenario where people use "AI" for their purposes. That is of course a very real scenario. But the question I raises was in relation to the OP's point about "AI" taking over the World and exterminating humans.

robryk 3y ago

How do all the combustion-powered engines prevent their shutdown? Well, they were useful, so we adapted our society to rely on them -- anyone who didn't was outcompeted. Now we can't stop using them or the society falls apart. This is not necessarily the mechanism most people consider, but it's a simple counterexample to "anything with an off switch can't be a threat to humanity".

svara 3y ago

All of your other replies assume the AI has intentions of its own, which isn't a necessary component of AGI. The plausible scenario if you ask me is that humans put it in charge of their company, with the explicit goal of improving the company's bottom line. It doesn't take that much imagination for this from where we are now. Just imagine a vastly more intelligent ChatGPT advising a company's leadership just by answering questions. Being superior to all humans, it's insanely successful at this and the company as a whole essential comes to rule the world.

SiempreViernes 3y ago

The whole of the discussion is largely done along (now forgotten) cultural pathways first walked by Christian religion, so AGI has taken on capabilities and motivations that make it basically the devil. Thus by definition it can do whatever is needed to bring about the catastrophe that is its destiny.

Al-Khwarizmi 3y ago
I hadn't thought about the links to religion. Interesting. I'm not religious, but if I were, it would be quite natural to interpret a misaligned AI as literally the devil.

nurettin 3y ago

> how are machines supposed to intervene in the real world and threaten our very existence? Obvious answer is: any control system connected to a computer is already accessible. Who says a computer program has to "gain" access? Programs already have access to a lot of machinery. From power plants to hospitals to elevators and magnetic doors. I know, because I programmed a bunch. These days GPT can output function calls as json, all you need to do is make it call your control automation and all your cheesy unrealistic sci-fi shows become feasible.

metanonsense 3y ago

To answer that question, ask yourself what a „physical threat“ means to you. I would bet that (in an industrialized country) for any example there is a chain of events that could possibly be controlled by an AI. Dying of a virus? Could be designed by an AI and sent to bio-terrorists. Being beaten to death? An AI could provoke an angry mob. Dying of hunger? An AI could sabotage the economy of a country until it is back in 19th century. And those examples are „relatively“ benign.

iNic 3y ago

Assume that it has access to the internet. For now assume it has access to some money, although this assumption can be weakened. If this is true, then you can order proteins from labs online with basically no security checks. You then pay some task rabbit to recieve and mix these to create dangerous biological material or to make viruses more dangerous or something. This is a family of paths to a lot of damage.

tim333 3y ago

Duh - killer robots! Haven't you seen Terminator? From a practical point of view the Putin types would love a robot army to invade all their neighbours and then you just need that to do in its leader and run amok.

panta 3y ago

a system with superhuman intelligence could easily manipulate human beings, convincing them of the opportunity to do things. A superior intelligence wouldn't ask for things obviously dangerous for humans in the immediate future, but would work on a longer timescale. It could distribute smaller tasks that taken singularly would look innocent. It could suggest social interventions that would diminish critical thinking ability of the general population in the long term. It could help concentrating power in fewer human hands, so to have an easier time manipulating those who count. This wouldn't happen overnight.

DalasNoin 3y ago

"... from inside a laboratory, within an isolated network" Who said anything about it being on an isolated network, we are on route to do total commercialization. Your Windows machine might soon literally have an llm on it running commands for you and managing you data. If you don't use it, you will be outcompeted. People used to argue about "boxing" or "airgapping" the AI, but we are literally just going to hand it control (on our current trajectory).

naasking 3y ago

> how would such a machine - from inside a laboratory, within an isolated network Someone clearly isn't aware of ChaosGPT. The notion that these systems would somehow stay isolated to the lab is absurd on its face. But even if they were on an isolated network, and you grant that this is a superintelligence, then how much isolation do you reckon is really sufficient? If it's software layer, then a superintelligence might be able to find bugs that we've missed and break out. Not even an air gap would necessarily fully isolate [1] a superintelligence. And then you're completely missing the human factor: a superintelligence could easily make anyone rich, so a researcher in this lab could easily be tempted to exploit that for personal gain by connecting it to the internet. How many of these attack vectors are AI labs insulated against? How many attack vectors are we simply not even imaginative enough to have thought of yet? [1] https://threatpost.com/air-gap-attack-turns-memory-wifi/162358/ https://threatpost.com/air-gap-attack-turns-memory-wifi/1623...

iainmerrick 3y ago

Lots of good replies on this thread, but here’s another possible analogy: In high frequency trading, you have very complex software doing stock trades inconceivably faster than any human. If something goes wrong, it could bankrupt your hedge fund pretty swiftly. So hedge funds will have lots of safety checks, probably including a “big red button” that just shuts everything down. So those companies must be completely safe from computer errors causing bankruptcy, right? After all, you can just shut the system down. But some companies have gone bankrupt due to computer error. There are plenty of good reasons for the system not to be shut down in time (or at all). The risk is hopefully small but it’s not zero.

dools 3y ago

I think we've seen pretty convincingly that all a super intelligence would need access to in order to completely ruin humanity is social media APIs.