5 ms·

> But the A.I. hype is out of hand. "A.I. Safety" research is the worst of it, as it suggests this technology is so powerful that it's actually dangerous. It b

> But the A.I. hype is out of hand. "A.I. Safety" research is the worst of it, as it suggests this technology is so powerful that it's actually dangerous. It boggles my mind how anyone can think otherwise. Existential dangers of superintelligent or even non-intelligent AI are the long-term result of the dangers of AI being developed and misused over time for human ends. It's the exact same argument behind why we should be trying to track asteroids, or why we should be trying to tackle climate change: the worst-case scenario is unlikely or in the future, but the path we're on has numerous dangers where suffering and loss of human life is virtually certain unless something is done.

thesuperbigfrog 4y ago
>> the path we're on has numerous dangers where suffering and loss of human life is virtually certain unless something is done What sort of AI catastrophe do you think would happen?
- naasking 4y ago
  That's akin to asking "what sort of human catastrophe do you think would happen?" A whole host of things are possible in principle on a long enough timeline. If you constrain the timeline, then you'll get more plausible answers but with potentially wide error bars because we don't really understand what "intelligence" really is, ie. current research could be pretty close but could also be pretty far from it. The fact that we don't know how close we are is itself dangerous. It's like doing gene editing on pathogens without a proper understanding of germ theory and biosafety. That's where we are with AI.
michaelt 4y ago
> It boggles my mind how anyone can think otherwise. Some AI dangers are certainly legitimate - it's easy to foresee how an image recognition system might think all snowboarders are male; or a system trained on unfair sentences handed out to criminals would replicate that unfairness, adding a wrongful veneer of science and objectivity; or a self-driving car trained on data from a country with few mopeds and most pedestrians wearing denim might underperform in a country with many mopeds and few pedestrians wearing denim. But other AI dangers sound more like the work of philosophers and science fiction authors. The moment people start predicting the end of humans needing to work, or talking about a future evil AI that punishes people who didn't help bring it into existence? That's pretty far down in my list of worries.
- naasking 4y ago
  > But other AI dangers sound more like the work of philosophers and science fiction authors. Your ability to read this sentence right now when we have never met and may not even be on the same continent was once in the domain of science fiction. Don't underestimate technological progress, and specifically, don't underestimate the surprising directions it could go. Some fantastical AI predictions will happen, most probably will not, and some utterly terrifying ones no one foresaw will almost certainly happen. The unknown unknowns should worry you, and AI is full of them.
  - michaelt 4y ago
    > The unknown unknowns should worry you, and AI is full of them. Sure, but where should that rank in my worries relative to 'designer babies' and 'rise of authoritarian states as economic powerhouses' and 'corporations that can commit crimes with impunity' and 'rising medical bills' and 'widening gap between rich and poor' and 'far right extremism' and 'water shortages' and 'economic crisis wipes out my savings' and 'cyber warfare targeting vital infrastructure' and 'rising obesity' and 'voter suppression' and the many other things a person could worry about?
    - naasking 4y ago
      In my view, the only wrong opinions on where to rank this are "at the very top" and "at the very bottom or not at all". We will only know the correct answer in hindsight, so the sensible position is to just start funding some legitimate AI safety research.
      - ethanbond 4y ago
        A whole lot of the arguments about why we shouldn't be concerned really boil down to "I cannot conceive of risk until that risk has materialized." Impossible to argue against, really.
Diggsey 4y ago
Because it's like cavemen pondering the safety of nuclear fusion after discovering fire. Yes: nuclear fusion could be dangerous. No: there is nothing useful that can come out of such "research". You know what would be useful for cavemen to ponder? The safety of fire. Or you know, just staying alive because there are more dangerous things out there. The current state of so-called "AI" is our fire. It's impressive and useful (and there are real dangers associated with it) but it has no bearing on intelligence, let alone superintelligence. It's more likely that a freak super-intelligent human will be born than that we accidentally produce a super-intelligent computer. We produce a lot of intelligent humans, and we've never produced a single intelligent computer. As it stands, we don't have the understanding or the tools to do anything useful wrt safety from a super-intelligent AI. We do have the understanding and tools to do something useful about asteroids and climate change.
- naasking 4y ago
  > No: there is nothing useful that can come out of such "research". That's just wrong. This very paper we're discussing has AI safety factors built-in: the AI is not supposed to be able to lie, and yet it exhibited deceptive behaviours anyway. That falls squarely under AI safety, and is a pretty useful observation to inform future attempts. > The current state of so-called "AI" is our fire. It's impressive and useful (and there are real dangers associated with it) but it has no bearing on intelligence, let alone superintelligence. That's conjecture. We don't know what general intelligence really is, so you simply cannot gauge how close we are. For all you know we could be one simple tweak away from current architectures, and that should be terrifying. > As it stands, we don't have the understanding or the tools to do anything useful wrt safety from a super-intelligent AI. Even if that were true, which it's not, we certainly won't develop that understanding or those tools if we don't start researching them! > We do have the understanding and tools to do something useful about asteroids and climate change. Suppose you had a time machine and went back to the 19th century to explain the dangers of asteroids wiping out all of humanity. They didn't have the launch capability or the detection abilities we do now, but if they were sufficiently convinced of this real threat, does it not seem plausible that they could be motivated to more heavily fund research into telescopes and rockets? Couldn't we plausibly have reached space sooner and be even better prepared in the present to meet that threat than we are now? Instead of a time machine, couldn't we just use our big brains to predict that something might actually be a serious problem in the future and that maybe we should devote a few smart people to thinking about these problems now and how to mitigate them? This all just strikes me as incredibly obvious, and we do it in literally every other domain, but somehow say "AI" and basic logic goes out the window.
unmole 4y ago
> unless something is done We must do something. This is something. Therefore, we must do this.
- naasking 4y ago
  1. We must do something because AIs can be dangerous. 2. AI safety research attempts to mitigate some AI dangers. 3. Therefore we must fund AI safety research. Yup, checks out.
  - unmole 4y ago
    The gods are vengeful. The priests say virgin sacrifices pacify the gods. Therefore we must sacrifice a virgin on every new moon. While we are at it, we should also build a palatial mansion for the wise priests. Yup, checks out.
    - naasking 4y ago
      > The priests say virgin sacrifices pacify the gods. An argument that depends on equating priests and scientists is an argument that fails.
      - iudqnolq 4y ago
        Except most AI safety doesn't involve scientists. In an extreme example, AI safety research funding funded harry potter fanfiction.
        naasking 4y ago
        Do you actually have empirical data demonstrating that scientists aren't concerned about AI, or that "most" AI safety doesn't involve scientists, or is this just your gut feeling? Edit: for instance, here's a computer science professor and neurologist who published a book about all of the serious dangers of AI, and calling for funding safety research: https://www.amazon.com/Human-Compatible-Artificial-Intelligence-Problem/dp/0525558616/ref=as_li_ss_tl?keywords=human+compatible&qid=1577149898&sr=8-1&linkCode=ll1&tag=slatestarcode-20&linkId=ea5e41f92ad5c6166f0b399c2b430671&language=en_US https://www.amazon.com/Human-Compatible-Artificial-Intellige... Furthermore, AI safety programs are hiring AI researchers and computer scientists to actually do the work, so are you claiming that these people don't sincerely believe in the work they're doing?
origin_path 4y ago
"the worst-case scenario is unlikely or in the future ... loss of human life is virtually certain" These two things are in conflict. We could ignore both asteroids and climate change and according to the best known science there'd be very little impact for vast timespans and possibly no impact ever (before humanity is ended by something else like war). Yes, also for the climate. Look at the actual predictions and it's like a small reduction in GDP growth spread over a very long period of time, and that's assuming the predictions are actually correct when they have a long track record of being not so. Really stuff like asteroids and climate is a good counter-argument to caring about AI risk. Intellectuals like to hypothesize world-ending cataclysms that only their far sighted expertise can prevent, but whenever these people's predictions get tested against something concrete they seem to invariably end up being wrong. Our society rewards catastrophising far too generously and penalizes being wrong far too little, especially for academics, NGOs etc. It makes people feel or seem smart in the moment, and they can punt the reputational damage from being wrong far into the future (and then pretend they never made those predictions at all or there were mitigating factors).
- naasking 4y ago
  > We could ignore both asteroids and climate change and according to the best known science there'd be very little impact for vast timespans and possibly no impact ever (before humanity is ended by something else like war). That's just incorrect. The Tunguska event was a nuclear-weapon scale asteroid. These are predicted to happen once every hundred years or so. If it happens over a populated city millions would die. If a person wrongly concludes this was a surprise nuclear attack, maybe everyone would die, to say nothing of the real risk that the asteroid itself could be big enough to wipe us all out. There's a lot of uncertainty around climate change, but changing climate patterns will certainly change resource allocations (fresh water, arable land, etc.). This will lead to shortages in places that were once abundant, which could easily lead to wars in which millions die. > Really stuff like asteroids and climate is a good counter-argument to caring about AI risk. This. This boggles my mind. Long-tail risks exist, and burying your head in the sand and pretending they don't just places millions of lives at risk, and potentially the entire human race. You don't have to think these are top priorities, but to dismiss them as complete unimportant is frankly bonkers.
nradov 4y ago
The so called "AI safety" researchers issuing breathless warnings about the dangers of superintelligence are nothing but grifters. They are the modern equivalent of shaman, telling the common people that they need ever increasing amounts of resources to continue their vital work in protecting us from wrathful spirits. There is zero actual scientific evidence to support their claims. It is essentially a modern-day secular religion. (There is value in doing research and ethical analysis into AI/ML statistical algorithms to prevent hidden biases or accidental physical harm. People working in those areas are producing real benefits for the rest of us and I'm not criticizing them.)
- naasking 4y ago
  > The so called "AI safety" researchers issuing breathless warnings about the dangers of superintelligence are nothing but grifters. Who are you talking about exactly? Who are these alleged grifters and who are they grifting? > There is zero actual scientific evidence to support their claims What claims are you talking about, exactly? Let's get specific.

thesuperbigfrog 4y ago

>> the path we're on has numerous dangers where suffering and loss of human life is virtually certain unless something is done What sort of AI catastrophe do you think would happen?

naasking 4y ago
That's akin to asking "what sort of human catastrophe do you think would happen?" A whole host of things are possible in principle on a long enough timeline. If you constrain the timeline, then you'll get more plausible answers but with potentially wide error bars because we don't really understand what "intelligence" really is, ie. current research could be pretty close but could also be pretty far from it. The fact that we don't know how close we are is itself dangerous. It's like doing gene editing on pathogens without a proper understanding of germ theory and biosafety. That's where we are with AI.

michaelt 4y ago

> It boggles my mind how anyone can think otherwise. Some AI dangers are certainly legitimate - it's easy to foresee how an image recognition system might think all snowboarders are male; or a system trained on unfair sentences handed out to criminals would replicate that unfairness, adding a wrongful veneer of science and objectivity; or a self-driving car trained on data from a country with few mopeds and most pedestrians wearing denim might underperform in a country with many mopeds and few pedestrians wearing denim. But other AI dangers sound more like the work of philosophers and science fiction authors. The moment people start predicting the end of humans needing to work, or talking about a future evil AI that punishes people who didn't help bring it into existence? That's pretty far down in my list of worries.

naasking 4y ago
> But other AI dangers sound more like the work of philosophers and science fiction authors. Your ability to read this sentence right now when we have never met and may not even be on the same continent was once in the domain of science fiction. Don't underestimate technological progress, and specifically, don't underestimate the surprising directions it could go. Some fantastical AI predictions will happen, most probably will not, and some utterly terrifying ones no one foresaw will almost certainly happen. The unknown unknowns should worry you, and AI is full of them.
- michaelt 4y ago
  > The unknown unknowns should worry you, and AI is full of them. Sure, but where should that rank in my worries relative to 'designer babies' and 'rise of authoritarian states as economic powerhouses' and 'corporations that can commit crimes with impunity' and 'rising medical bills' and 'widening gap between rich and poor' and 'far right extremism' and 'water shortages' and 'economic crisis wipes out my savings' and 'cyber warfare targeting vital infrastructure' and 'rising obesity' and 'voter suppression' and the many other things a person could worry about?
  - naasking 4y ago
    In my view, the only wrong opinions on where to rank this are "at the very top" and "at the very bottom or not at all". We will only know the correct answer in hindsight, so the sensible position is to just start funding some legitimate AI safety research.
    - ethanbond 4y ago
      A whole lot of the arguments about why we shouldn't be concerned really boil down to "I cannot conceive of risk until that risk has materialized." Impossible to argue against, really.

Diggsey 4y ago

Because it's like cavemen pondering the safety of nuclear fusion after discovering fire. Yes: nuclear fusion could be dangerous. No: there is nothing useful that can come out of such "research". You know what would be useful for cavemen to ponder? The safety of fire. Or you know, just staying alive because there are more dangerous things out there. The current state of so-called "AI" is our fire. It's impressive and useful (and there are real dangers associated with it) but it has no bearing on intelligence, let alone superintelligence. It's more likely that a freak super-intelligent human will be born than that we accidentally produce a super-intelligent computer. We produce a lot of intelligent humans, and we've never produced a single intelligent computer. As it stands, we don't have the understanding or the tools to do anything useful wrt safety from a super-intelligent AI. We do have the understanding and tools to do something useful about asteroids and climate change.

naasking 4y ago
> No: there is nothing useful that can come out of such "research". That's just wrong. This very paper we're discussing has AI safety factors built-in: the AI is not supposed to be able to lie, and yet it exhibited deceptive behaviours anyway. That falls squarely under AI safety, and is a pretty useful observation to inform future attempts. > The current state of so-called "AI" is our fire. It's impressive and useful (and there are real dangers associated with it) but it has no bearing on intelligence, let alone superintelligence. That's conjecture. We don't know what general intelligence really is, so you simply cannot gauge how close we are. For all you know we could be one simple tweak away from current architectures, and that should be terrifying. > As it stands, we don't have the understanding or the tools to do anything useful wrt safety from a super-intelligent AI. Even if that were true, which it's not, we certainly won't develop that understanding or those tools if we don't start researching them! > We do have the understanding and tools to do something useful about asteroids and climate change. Suppose you had a time machine and went back to the 19th century to explain the dangers of asteroids wiping out all of humanity. They didn't have the launch capability or the detection abilities we do now, but if they were sufficiently convinced of this real threat, does it not seem plausible that they could be motivated to more heavily fund research into telescopes and rockets? Couldn't we plausibly have reached space sooner and be even better prepared in the present to meet that threat than we are now? Instead of a time machine, couldn't we just use our big brains to predict that something might actually be a serious problem in the future and that maybe we should devote a few smart people to thinking about these problems now and how to mitigate them? This all just strikes me as incredibly obvious, and we do it in literally every other domain, but somehow say "AI" and basic logic goes out the window.

unmole 4y ago

> unless something is done We must do something. This is something. Therefore, we must do this.

naasking 4y ago
1. We must do something because AIs can be dangerous. 2. AI safety research attempts to mitigate some AI dangers. 3. Therefore we must fund AI safety research. Yup, checks out.
- unmole 4y ago
  The gods are vengeful. The priests say virgin sacrifices pacify the gods. Therefore we must sacrifice a virgin on every new moon. While we are at it, we should also build a palatial mansion for the wise priests. Yup, checks out.
  - naasking 4y ago
    > The priests say virgin sacrifices pacify the gods. An argument that depends on equating priests and scientists is an argument that fails.
    - iudqnolq 4y ago
      Except most AI safety doesn't involve scientists. In an extreme example, AI safety research funding funded harry potter fanfiction.
      - naasking 4y ago
        Do you actually have empirical data demonstrating that scientists aren't concerned about AI, or that "most" AI safety doesn't involve scientists, or is this just your gut feeling? Edit: for instance, here's a computer science professor and neurologist who published a book about all of the serious dangers of AI, and calling for funding safety research: https://www.amazon.com/Human-Compatible-Artificial-Intelligence-Problem/dp/0525558616/ref=as_li_ss_tl?keywords=human+compatible&qid=1577149898&sr=8-1&linkCode=ll1&tag=slatestarcode-20&linkId=ea5e41f92ad5c6166f0b399c2b430671&language=en_US https://www.amazon.com/Human-Compatible-Artificial-Intellige... Furthermore, AI safety programs are hiring AI researchers and computer scientists to actually do the work, so are you claiming that these people don't sincerely believe in the work they're doing?

origin_path 4y ago

"the worst-case scenario is unlikely or in the future ... loss of human life is virtually certain" These two things are in conflict. We could ignore both asteroids and climate change and according to the best known science there'd be very little impact for vast timespans and possibly no impact ever (before humanity is ended by something else like war). Yes, also for the climate. Look at the actual predictions and it's like a small reduction in GDP growth spread over a very long period of time, and that's assuming the predictions are actually correct when they have a long track record of being not so. Really stuff like asteroids and climate is a good counter-argument to caring about AI risk. Intellectuals like to hypothesize world-ending cataclysms that only their far sighted expertise can prevent, but whenever these people's predictions get tested against something concrete they seem to invariably end up being wrong. Our society rewards catastrophising far too generously and penalizes being wrong far too little, especially for academics, NGOs etc. It makes people feel or seem smart in the moment, and they can punt the reputational damage from being wrong far into the future (and then pretend they never made those predictions at all or there were mitigating factors).

naasking 4y ago
> We could ignore both asteroids and climate change and according to the best known science there'd be very little impact for vast timespans and possibly no impact ever (before humanity is ended by something else like war). That's just incorrect. The Tunguska event was a nuclear-weapon scale asteroid. These are predicted to happen once every hundred years or so. If it happens over a populated city millions would die. If a person wrongly concludes this was a surprise nuclear attack, maybe everyone would die, to say nothing of the real risk that the asteroid itself could be big enough to wipe us all out. There's a lot of uncertainty around climate change, but changing climate patterns will certainly change resource allocations (fresh water, arable land, etc.). This will lead to shortages in places that were once abundant, which could easily lead to wars in which millions die. > Really stuff like asteroids and climate is a good counter-argument to caring about AI risk. This. This boggles my mind. Long-tail risks exist, and burying your head in the sand and pretending they don't just places millions of lives at risk, and potentially the entire human race. You don't have to think these are top priorities, but to dismiss them as complete unimportant is frankly bonkers.

nradov 4y ago

The so called "AI safety" researchers issuing breathless warnings about the dangers of superintelligence are nothing but grifters. They are the modern equivalent of shaman, telling the common people that they need ever increasing amounts of resources to continue their vital work in protecting us from wrathful spirits. There is zero actual scientific evidence to support their claims. It is essentially a modern-day secular religion. (There is value in doing research and ethical analysis into AI/ML statistical algorithms to prevent hidden biases or accidental physical harm. People working in those areas are producing real benefits for the rest of us and I'm not criticizing them.)

naasking 4y ago
> The so called "AI safety" researchers issuing breathless warnings about the dangers of superintelligence are nothing but grifters. Who are you talking about exactly? Who are these alleged grifters and who are they grifting? > There is zero actual scientific evidence to support their claims What claims are you talking about, exactly? Let's get specific.