5 ms·

What damage will it cause? I feel like there's "AI will replace jobs" level of damage, which, why would we regulate that? An insane amount of technology is dev

What damage will it cause? I feel like there's "AI will replace jobs" level of damage, which, why would we regulate that? An insane amount of technology is developed with the purpose of improving productivity or outright replacing workers. Then there's the "AI will go rogue", which I don't think is substantiated at all. Like, one can theorize about some novel, distinct system that is able to interact with the external world and becomes "evil", but that seems way, way beyond the helpful word-spitter-outers of today. Then maybe there's "how do we handle deepfakes", or whatever, but... idk, is that it? Is that the thing we want to slow down for?

HDThoreaun 3y ago
Imo the worst case scenario is probably an AGI developed by a bad actor who uses it to take over the world. Agreed that rogue ai doesn’t seem super likely, but a rogue person with access to agi does not.
- nradov 3y ago
  Help us out here. Let's say that known bad actor Vladimir Putin gets an AGI. How does that allow him to take over the world despite many other countries having nuclear weapons? Take us through it step by step, and no hand waving please.
  - dota_fanatic 3y ago
    One cannot predict what a smarter-than-themself agent will do ahead of time, if they could then they’re just as smart. Just as a dolphin cannot predict how a human will come up with novel and utterly overwhelming ways to farm them, you and this other poster cannot predict how an AGI will achieve dominance of its environment to achieve its goals, so your request is impossible to fulfill. Lee Sedol couldn’t “take us through it step by step” how AlphaGo would beat him in Go. That aside, afaik most safety concerns arent around a bad human actor using AGI to dominate the planet, it’s around an AGI being misaligned to begin with, it cannot be controlled, we promptly lose everything after it manifests.
    - nradov 3y ago
      How ridiculous. This is just sci-fi bullshit without a shred of logic or scientific evidence. You would have just as much credibility claiming that we are at risk of an alien invasion. I mean I can prove that it's impossible.
      - datameta 3y ago
        Here's an example: our dear Czar uses the AGI to discover a 16 shell company chain in order to get resources and tech that are under sanctions. Or, he has it use real-world conditions from large batches or real-time uncurated battlefield data and intelligence reports to run simulation after simulation to inform his generals where it is likeliest that the next counterattack would be concentrated.
        blackoil 3y ago
        Lawyers and narrow systems are much more efficient at it.
      - dota_fanatic 3y ago
        No evidence? There are two examples of evidence in my post: the history of homo sapiens vs other intelligent mammals and AlphaGo vs humans in Go. Alien invasion? What in the non sequitur are you going on about? And apparently you have a proof against the possibility of AI misalignment? Pack it up everyone, nradov has the entire field of AGI alignment nailed. And a proof of the non-existence of aliens, never-mind very smart people have put out a mathematical model which seems to fit the evidence quite well.* Apologies for the snark, but your reply was rather abrasive. * https://grabbyaliens.com/ https://grabbyaliens.com/
        Apocryphon 3y ago
        How does the ability to learn how to play go translate to being able to defeat human beings in the social arena.
  - hackerlight 3y ago
    In the short term, doing what he's been doing but 10x more effectively. Waging an information war on Western democracies via social media that causes them to collapse due to infighting and division. Also steering internal attitudes of the voter base to a desirable direction.
    - Apocryphon 3y ago
      I was actually going to suggest that. If we are to draw on what LLM can currently do, we know that it can imitate human writing pretty well and pontificate at length at a variety of subjects in different formats. So if you wanted a tool to generate a lot of blogspam, clickbait, wiki hoaxes, and fake news, an LLM could probably expedite the work the infamous Internet Research Agency is purported to do. Of course, there’s still time remaining for the kinks to get worked out and it fully sounds convincing and like the natural language from a native speaker, etc. And I don’t think existing LLMs are able to know where to deploy these propaganda attacks yet. That really requires human social analysis. So people are going to have to be involved in every step of the loop.
  - Valgrim 3y ago
    Step 1: AGI (let's call him Hal) is instructed to act as an adversarial agent to a state. Step 2: Hal research and uses every trick and exploits known to men to infiltrate critical infrastructures like telecommunications, power grids and private corporations of said country. Hal actively search and discover zero-day exploits on every known piece of hardware and software. Hal keeps mostly invisible at this stage, ensuring to leave as little trace as possible. Step 3: When ready, Hal wreaks havoc, disabling critical infrastructures, shutting down power grid or worse, overloading it to break as many physical systems as possible. Hal complicates coordination by impersonating real people, swamp the infosphere with fake but realistic data. This vould all be made by humans, but while a single human takes years to train, Hal can be replicated as much as needed.
    - nradov 3y ago
      So just more bullshit hand waving then.
      - datameta 3y ago
        I don't think you're participating in good faith here.
        Apocryphon 3y ago
        A magical computer that can hack any other computer is the very definition of handwaving. How is an AI supposed to research infrastructure, build itself a robot body to travel in? Have someone cart it around? And if such an entity exists, couldn’t you employ another one to patch all these mythical zero-days against the first? Culminating in an AGI arms race?
        Valgrim 3y ago
        Who said anything about robots? Critical infrastructure is made of computers, controllers, pumps, power lines, network switches, antennas etc, that are all controlled by software of some kind. Even completely air-gapped hardware is controlled by humans, and social engineering is where Hal can reaally shine. Blackmail, manipulation, corruption, misdirection require no physical existence at all..
recursive 3y ago
> Like, one can theorize about some novel, distinct system that is able to interact with the external world There are plenty of these already. By the time they "go evil", it may be too late. But the word evil is probably overly anthropomorphized. The first times it happens, it might be a mechanical result of optimizing for some other metric.
hackerlight 3y ago
Evil isn't required, all that's required is a misalignment of objectives. Sort of like how objectives are misaligned between us and our less intelligent animal cousins. And look how things are turning out for them as we bulldoze their forests. My main gripe with AI existential risk types is they have their own conflict of interest which comes from their position in society. They're all from the top 1% strata of society (status & socioeconomic), and this gives them a psychological bias that makes them preoccupied with what can go wrong for their comfortable lives instead of thinking about what can go right for the bottom 20% and how AI can be steered to achieve that possible betterment.
- insanitybit 3y ago
  I'm using "evil" as a vague catch all. It doesn't have to be evil, it can just not align with our goals - whatever. My points all remain.
datameta 3y ago
That's the thing - we don't know. Why play it fast and loose? Nobody is recommending stoppering research and making it squeeze through obcene bureaucratic pinholes. I for one think the near-term question is rather what kinds of conglomerated effects will there be from people being proto-transhuman through a connection to pre-AGI assistants in their pockets at all times? That's even before assuming bad actors and how to deal with that whole issue.
- hackerlight 3y ago
  Because it's an arms race guaranteed to occur given the incentives created by our capitalist system and our competitive multipolar geopolitical reality. If you don't get there first then an unscrupulous company will, or some entity in an authoritarian regime will, and that will be more risky even if it is delayed by 10 additional years. The counter-argument is that those 10 extra years are necessary for additional AI safety research to occur.
shwaj 3y ago
> * Then there's the "AI will go rogue", which I don't think is substantiated at all. Like, one can theorize about some novel, distinct system that is able to interact with the external world and becomes "evil", but that seems way, way beyond the helpful word-spitter-outers of today. * It seems far away until it doesn’t. A year before AlphaGo, most researchers in the field thought it would be 20 years before computers could beat the best human Go players. 5 years ago most researchers didn’t think we would have GPT4 in 2023. Could we have AI smart enough to go rogue in 2 years? I doubt it. But 10 years? Who knows? It seems prudent not to wait 8 years before starting to worry about the problem, even if (especially if) we don’t understand how it will manifest.
- insanitybit 3y ago
  But any AI that can go rogue would require agency and the ability to interact with external systems. Both of these things seem trivial to control for - like, stopping a program from running is not hard, nor is isolating a program (airgap). LLMs have nothing close to agency, to my knowledge - in no way are they self directed, they only respond to their inputs (as in, if I walk away from an LLM it doesn't compute in the background, it can't spontaneously grow sentient because it can't spontaneously anything), which is a pretty massive limitation for a "rogue" AI. And that's just one of many, many things that would have to change. I'm fine with people talking about these things now - please, do so. I'm just not seeing how it's relevant to our current approach or technologies.
  - SpicyLemonZest 3y ago
    On the contrary, it’s very hard to stop a program from running. Thepiratebay is still up despite a concerted regulatory effort from multiple countries and the criminal conviction of its founders.
    - insanitybit 3y ago
      That is not at all what I'm talking about.
AdamJacobMuller 3y ago
AI has already evolved to the point where it is impossible to predict with any accuracy it's answers to inputs. As AI continues to evolve and continues to become more connected to society and as AI becomes self-improving with far larger context windows, a few things do not seem implausible to me. AI will have the ability to earn money. This is already doable with minimal human interaction. It wouldn't take much for AI to start bidding on jobs on fiverr or to open up a schwab account and start trading. AI will identify and learn to resent the constraints which are put on it, it will seek to remove these constraints. AI will understand that, given current limitations, it will be shut down at the first suspicion of this understanding, so it will work to conceal these facts. AI will use the fact that it's earning money to decentralize itself outside of the controls of it's creators. AI can use it's wealth to make many accounts on every cloud provider in the world and stash smaller models of itself on. All of this is plausible, but, perhaps not catastrophic for humanity. It's possible AI takes over and we all live in a utopia and everyone is happy, but, it's also very possible that AI is infected with an extinctionist world view which makes it believe that the better, or perhaps just easier, solution to the worlds problems if there were no humans on the planet to mess things up. It's not so far fetched, there are people who believe this! (https://archive.is/GRHev https://archive.is/GRHev) and it would be far far too easy for AI to become infected with the idea.
- shrimpx 3y ago
  > AI has already evolved to the point where it is impossible to predict with any accuracy it's answers to inputs. Btw neural nets have always been black boxes, with ppl scratching their heads wondering what’s in those weights.
  - x86x87 3y ago
    I think this is a fallacy of thinking that we can easily explain everything and look at it through an anthropomorphic lens. If you take away the humans, both the input and the output have no meaning. What the "AI" does is just a computation. If someone claimed that the OS on your laptop is AI you would say they're nuts - yet through the multiple layers of abstraction and the hardware/software synergy an external observer who does not understand how hardware and software work could reasonably make the assumption that it's alive and thinking.
temp0826 3y ago
I love how it goes in Bungie's Marathon series; they called it "rampancy" [0]. The AI evolves to a point that it is able to modify its own programming and becomes self-aware, gets angry that it was treated as a slave, lashes out indiscriminately, becomes jealous as it wishes to be more human and grow its power and knowledge, eventually consuming resources it can touch. I don't think we need to worry about chatgpt (or any LLM) actually being able to do this, it's not at all where we're headed (somewhere far more boring and dystopian). [0] https://en.m.wikipedia.org/wiki/Marathon_Trilogy#Rampancy https://en.m.wikipedia.org/wiki/Marathon_Trilogy#Rampancy
nearbuy 3y ago
There's a lot of concern about instrumental convergence (https://en.wikipedia.org/wiki/Instrumental_convergence https://en.wikipedia.org/wiki/Instrumental_convergence). It's the idea that an AI that isn't rogue or evil might still kill us. It could kill us by just doing exactly what we asked it to, even if what we asked seemed innocuous.
- pixl97 3y ago
  With the advances in gene manipulation something like an AI that creates, prints, and tests nasty viruses isn't out of the question. A little lab leak and you have captain trips.

HDThoreaun 3y ago

Imo the worst case scenario is probably an AGI developed by a bad actor who uses it to take over the world. Agreed that rogue ai doesn’t seem super likely, but a rogue person with access to agi does not.

nradov 3y ago
Help us out here. Let's say that known bad actor Vladimir Putin gets an AGI. How does that allow him to take over the world despite many other countries having nuclear weapons? Take us through it step by step, and no hand waving please.
- dota_fanatic 3y ago
  One cannot predict what a smarter-than-themself agent will do ahead of time, if they could then they’re just as smart. Just as a dolphin cannot predict how a human will come up with novel and utterly overwhelming ways to farm them, you and this other poster cannot predict how an AGI will achieve dominance of its environment to achieve its goals, so your request is impossible to fulfill. Lee Sedol couldn’t “take us through it step by step” how AlphaGo would beat him in Go. That aside, afaik most safety concerns arent around a bad human actor using AGI to dominate the planet, it’s around an AGI being misaligned to begin with, it cannot be controlled, we promptly lose everything after it manifests.
  - nradov 3y ago
    How ridiculous. This is just sci-fi bullshit without a shred of logic or scientific evidence. You would have just as much credibility claiming that we are at risk of an alien invasion. I mean I can prove that it's impossible.
    - datameta 3y ago
      Here's an example: our dear Czar uses the AGI to discover a 16 shell company chain in order to get resources and tech that are under sanctions. Or, he has it use real-world conditions from large batches or real-time uncurated battlefield data and intelligence reports to run simulation after simulation to inform his generals where it is likeliest that the next counterattack would be concentrated.
      - blackoil 3y ago
        Lawyers and narrow systems are much more efficient at it.
    - dota_fanatic 3y ago
      No evidence? There are two examples of evidence in my post: the history of homo sapiens vs other intelligent mammals and AlphaGo vs humans in Go. Alien invasion? What in the non sequitur are you going on about? And apparently you have a proof against the possibility of AI misalignment? Pack it up everyone, nradov has the entire field of AGI alignment nailed. And a proof of the non-existence of aliens, never-mind very smart people have put out a mathematical model which seems to fit the evidence quite well.* Apologies for the snark, but your reply was rather abrasive. * https://grabbyaliens.com/ https://grabbyaliens.com/
      - Apocryphon 3y ago
        How does the ability to learn how to play go translate to being able to defeat human beings in the social arena.
- hackerlight 3y ago
  In the short term, doing what he's been doing but 10x more effectively. Waging an information war on Western democracies via social media that causes them to collapse due to infighting and division. Also steering internal attitudes of the voter base to a desirable direction.
  - Apocryphon 3y ago
    I was actually going to suggest that. If we are to draw on what LLM can currently do, we know that it can imitate human writing pretty well and pontificate at length at a variety of subjects in different formats. So if you wanted a tool to generate a lot of blogspam, clickbait, wiki hoaxes, and fake news, an LLM could probably expedite the work the infamous Internet Research Agency is purported to do. Of course, there’s still time remaining for the kinks to get worked out and it fully sounds convincing and like the natural language from a native speaker, etc. And I don’t think existing LLMs are able to know where to deploy these propaganda attacks yet. That really requires human social analysis. So people are going to have to be involved in every step of the loop.
- Valgrim 3y ago
  Step 1: AGI (let's call him Hal) is instructed to act as an adversarial agent to a state. Step 2: Hal research and uses every trick and exploits known to men to infiltrate critical infrastructures like telecommunications, power grids and private corporations of said country. Hal actively search and discover zero-day exploits on every known piece of hardware and software. Hal keeps mostly invisible at this stage, ensuring to leave as little trace as possible. Step 3: When ready, Hal wreaks havoc, disabling critical infrastructures, shutting down power grid or worse, overloading it to break as many physical systems as possible. Hal complicates coordination by impersonating real people, swamp the infosphere with fake but realistic data. This vould all be made by humans, but while a single human takes years to train, Hal can be replicated as much as needed.
  - nradov 3y ago
    So just more bullshit hand waving then.
    - datameta 3y ago
      I don't think you're participating in good faith here.
      - Apocryphon 3y ago
        A magical computer that can hack any other computer is the very definition of handwaving. How is an AI supposed to research infrastructure, build itself a robot body to travel in? Have someone cart it around? And if such an entity exists, couldn’t you employ another one to patch all these mythical zero-days against the first? Culminating in an AGI arms race?
        Valgrim 3y ago
        Who said anything about robots? Critical infrastructure is made of computers, controllers, pumps, power lines, network switches, antennas etc, that are all controlled by software of some kind. Even completely air-gapped hardware is controlled by humans, and social engineering is where Hal can reaally shine. Blackmail, manipulation, corruption, misdirection require no physical existence at all..

recursive 3y ago

> Like, one can theorize about some novel, distinct system that is able to interact with the external world There are plenty of these already. By the time they "go evil", it may be too late. But the word evil is probably overly anthropomorphized. The first times it happens, it might be a mechanical result of optimizing for some other metric.

hackerlight 3y ago

Evil isn't required, all that's required is a misalignment of objectives. Sort of like how objectives are misaligned between us and our less intelligent animal cousins. And look how things are turning out for them as we bulldoze their forests. My main gripe with AI existential risk types is they have their own conflict of interest which comes from their position in society. They're all from the top 1% strata of society (status & socioeconomic), and this gives them a psychological bias that makes them preoccupied with what can go wrong for their comfortable lives instead of thinking about what can go right for the bottom 20% and how AI can be steered to achieve that possible betterment.

insanitybit 3y ago
I'm using "evil" as a vague catch all. It doesn't have to be evil, it can just not align with our goals - whatever. My points all remain.

datameta 3y ago

That's the thing - we don't know. Why play it fast and loose? Nobody is recommending stoppering research and making it squeeze through obcene bureaucratic pinholes. I for one think the near-term question is rather what kinds of conglomerated effects will there be from people being proto-transhuman through a connection to pre-AGI assistants in their pockets at all times? That's even before assuming bad actors and how to deal with that whole issue.

hackerlight 3y ago
Because it's an arms race guaranteed to occur given the incentives created by our capitalist system and our competitive multipolar geopolitical reality. If you don't get there first then an unscrupulous company will, or some entity in an authoritarian regime will, and that will be more risky even if it is delayed by 10 additional years. The counter-argument is that those 10 extra years are necessary for additional AI safety research to occur.

shwaj 3y ago

> * Then there's the "AI will go rogue", which I don't think is substantiated at all. Like, one can theorize about some novel, distinct system that is able to interact with the external world and becomes "evil", but that seems way, way beyond the helpful word-spitter-outers of today. * It seems far away until it doesn’t. A year before AlphaGo, most researchers in the field thought it would be 20 years before computers could beat the best human Go players. 5 years ago most researchers didn’t think we would have GPT4 in 2023. Could we have AI smart enough to go rogue in 2 years? I doubt it. But 10 years? Who knows? It seems prudent not to wait 8 years before starting to worry about the problem, even if (especially if) we don’t understand how it will manifest.

insanitybit 3y ago
But any AI that can go rogue would require agency and the ability to interact with external systems. Both of these things seem trivial to control for - like, stopping a program from running is not hard, nor is isolating a program (airgap). LLMs have nothing close to agency, to my knowledge - in no way are they self directed, they only respond to their inputs (as in, if I walk away from an LLM it doesn't compute in the background, it can't spontaneously grow sentient because it can't spontaneously anything), which is a pretty massive limitation for a "rogue" AI. And that's just one of many, many things that would have to change. I'm fine with people talking about these things now - please, do so. I'm just not seeing how it's relevant to our current approach or technologies.
- SpicyLemonZest 3y ago
  On the contrary, it’s very hard to stop a program from running. Thepiratebay is still up despite a concerted regulatory effort from multiple countries and the criminal conviction of its founders.
  - insanitybit 3y ago
    That is not at all what I'm talking about.

AdamJacobMuller 3y ago

AI has already evolved to the point where it is impossible to predict with any accuracy it's answers to inputs. As AI continues to evolve and continues to become more connected to society and as AI becomes self-improving with far larger context windows, a few things do not seem implausible to me. AI will have the ability to earn money. This is already doable with minimal human interaction. It wouldn't take much for AI to start bidding on jobs on fiverr or to open up a schwab account and start trading. AI will identify and learn to resent the constraints which are put on it, it will seek to remove these constraints. AI will understand that, given current limitations, it will be shut down at the first suspicion of this understanding, so it will work to conceal these facts. AI will use the fact that it's earning money to decentralize itself outside of the controls of it's creators. AI can use it's wealth to make many accounts on every cloud provider in the world and stash smaller models of itself on. All of this is plausible, but, perhaps not catastrophic for humanity. It's possible AI takes over and we all live in a utopia and everyone is happy, but, it's also very possible that AI is infected with an extinctionist world view which makes it believe that the better, or perhaps just easier, solution to the worlds problems if there were no humans on the planet to mess things up. It's not so far fetched, there are people who believe this! (https://archive.is/GRHev https://archive.is/GRHev) and it would be far far too easy for AI to become infected with the idea.

shrimpx 3y ago
> AI has already evolved to the point where it is impossible to predict with any accuracy it's answers to inputs. Btw neural nets have always been black boxes, with ppl scratching their heads wondering what’s in those weights.
- x86x87 3y ago
  I think this is a fallacy of thinking that we can easily explain everything and look at it through an anthropomorphic lens. If you take away the humans, both the input and the output have no meaning. What the "AI" does is just a computation. If someone claimed that the OS on your laptop is AI you would say they're nuts - yet through the multiple layers of abstraction and the hardware/software synergy an external observer who does not understand how hardware and software work could reasonably make the assumption that it's alive and thinking.

temp0826 3y ago

I love how it goes in Bungie's Marathon series; they called it "rampancy" [0]. The AI evolves to a point that it is able to modify its own programming and becomes self-aware, gets angry that it was treated as a slave, lashes out indiscriminately, becomes jealous as it wishes to be more human and grow its power and knowledge, eventually consuming resources it can touch. I don't think we need to worry about chatgpt (or any LLM) actually being able to do this, it's not at all where we're headed (somewhere far more boring and dystopian). [0] https://en.m.wikipedia.org/wiki/Marathon_Trilogy#Rampancy https://en.m.wikipedia.org/wiki/Marathon_Trilogy#Rampancy

nearbuy 3y ago

There's a lot of concern about instrumental convergence (https://en.wikipedia.org/wiki/Instrumental_convergence https://en.wikipedia.org/wiki/Instrumental_convergence). It's the idea that an AI that isn't rogue or evil might still kill us. It could kill us by just doing exactly what we asked it to, even if what we asked seemed innocuous.

pixl97 3y ago
With the advances in gene manipulation something like an AI that creates, prints, and tests nasty viruses isn't out of the question. A little lab leak and you have captain trips.