5 ms·

Convince me that "x-risk research" won't be a bunch of out of touch academics handwaving and philosophising with their tenure as their primary concern and incen

Convince me that "x-risk research" won't be a bunch of out of touch academics handwaving and philosophising with their tenure as their primary concern and incentivised to say "you can't be too careful" while kicking the can down the road for a few more lifetimes? (You don't have to convince me; your position is like saying "we should wait for the perfect operating system and programming language before they get released to the world" and it's beaten by "worse is better" every time. The unfinished, inconsisent, flawed mess which you can have right now wins over the expensive flawless diamond in development estimated to be finished in just a few years. These models are out, the techniques are out, people have a taste for them, and the hardware to build them is only getting cheaper. Pandora's box is open, the genie's bottle is uncorked).

edouard-harris 4y ago
I mean, even if that is exactly what "x-risk research" turns out to be, surely even that's preferable to a catastrophic alternative, no? And by extension, isn't it also preferable to, say, a mere 10% chance of a catastrophic alternative?
- jodrellblank 4y ago
  > "surely even that's preferable to a catastrophic alternative, no?" Maybe? The current death rate is 150,000 humans per day, every day. It's only because we are accustomed to it that we don't think of it as a catastrophy; that's a World War II death count of 85 million people every 18 months. It's fifty Septebmer 11ths every day. What if a superintelligent AI can solve for climate change, solve for human cooperation, solve for vastly improved human health, solve for universal basic income which releives the drudgery of living for everyone, solve for immortality, solve for faster than light communication or travel, solve for xyz? How many human lives are the trade against the risk? But my second paragraph is, it doesn't matter whether it's preferable, events are in motion and aren't going to stop to let us off - it's preferable if we don't destroy the climate and kill a billion humans and make life on Earth much more difficult, but that's still on course. To me it's preferable to have clean air to breathe and people not being run over and killed by vehicles, but the market wants city streets for cars and air primarily for burining petrol and diesel and secondarily for humans to breathe and if they get asthsma and lung cancer, tough. I think the same will happen with AI, arguing that everyone should stop because we don't want Grey Goo or Paperclip Maximisers is unlikely to change the course of anything, just as it hasn't changed the course of anything up to now despite years and years and years of raising it as a concern.
  - theptip 4y ago
    I think that the benefits of AGI research are often omitted from the analysis, so I'm generally supportive of considering the cost/benefit. However I think you need to do a lot more work than just gesturing in the direction of very high potential benefits to actually convince anyone, in particular since we're dealing with extremely large numbers, that are extremely sensitive to small probabilities. EV = P(AlignedAI) * Utility(AGI) + P(1-AlignedAI) * Utility(ruin) (I'm aware that all I did up-thread was gesture in the direction of risks, but I think "unintended/un-measured existential risks" are in general more urgent to understand than "un-measured huge benefits"; there is no catching up from ruin, but you can often come back later and harvest fruit that you skipped earlier. Ideally we study both of course.)
- notahacker 4y ago
  If the catastrophic alternative is actually possible, who's to say the waffling academics aren't the ones to cause it? I'm being serious here: the AI model the x-risk people are worrying about here because it waffled about causing harm was originally developed by an entity founded by people with the explicit stated purpose of avoiding AI catastrophe. And one of the most popular things for people seeking x-risk funding to do is to write extremely long and detailed explanations of how and why AI is likely to harm humans. If I worried about the risk of LLMs achieving sentience and forming independent goals to destroy humanity based on the stuff they'd read, I'd want them to do less of that, not fund them to do more.
kelnos 4y ago
A flawed but useful operating system and programming language isn't likely to decide humanity is garbage and launch all nuclear weapons at once. A "worse is better" AGI could cause the end of humanity. I know that sounds overly dramatic, but I'm not remotely convinced that isn't possible, or even isn't likely. I agree with you that "x-risk" research could easily devolve into what you are worried about, but that doesn't mean we should ignore these risks and plow forward.
ShredKazoo 4y ago
>Pandora's box is open, the genie's bottle is uncorked As someone who's followed AI safety for over a decade now, it's been frustrating to see reactions flip from "it's too early to do any useful work!" to "it's too late to do any useful work!", with barely any time intervening. https://www.youtube.com/watch?v=0AW4nSq0hAc https://www.youtube.com/watch?v=0AW4nSq0hAc Perhaps it is worth actually reading a book like this one (posted to HN yesterday) before concluding that it's too late to do anything? https://betterwithout.ai/only-you-can-stop-an-AI-apocalypse https://betterwithout.ai/only-you-can-stop-an-AI-apocalypse
- jacquesm 4y ago
  This is how it always goes. Similar for climate change and lots of other problems that move slowly compared to the lifetime of a single human.
- nradov 4y ago
  AI safety is not a legitimate field. You have wasted your time. It's just a bunch of grifters posting alarmist tweets with no scientific evidence. You might as well be following "unicorn safety" or "ghost safety".
  - ShredKazoo 4y ago
    Do you think Stuart Russell (coauthor, with Peter Norvig, of the widely used textbook Artificial Intelligence: A Modern Approach) is a grifter? https://people.eecs.berkeley.edu/~russell/research/future/ https://people.eecs.berkeley.edu/~russell/research/future/ Does this review look like it only covers alarmist tweets? https://arxiv.org/pdf/1805.01109.pdf https://arxiv.org/pdf/1805.01109.pdf
    - nradov 4y ago
      Yes, Stuart Russell is a grifter. Some of the more advanced grifters have gone beyond tweeting and are now shilling low-effort books in an attempt to draw attention to themselves. Don't be fooled. If we want to talk about problems with biased data sets or using inappropriate AI algorithms for safety-critical applications then sure, let's address those issues. But the notion of some super intelligent computer coming to take over the world and kill everyone is just a stupid fantasy with no scientific basis.
naasking 4y ago
> Convince me that "x-risk research" won't be a bunch of out of touch academics handwaving and philosophising with their tenure https://scottaaronson.blog/?p=6823 https://scottaaronson.blog/?p=6823

edouard-harris 4y ago

I mean, even if that is exactly what "x-risk research" turns out to be, surely even that's preferable to a catastrophic alternative, no? And by extension, isn't it also preferable to, say, a mere 10% chance of a catastrophic alternative?

jodrellblank 4y ago
> "surely even that's preferable to a catastrophic alternative, no?" Maybe? The current death rate is 150,000 humans per day, every day. It's only because we are accustomed to it that we don't think of it as a catastrophy; that's a World War II death count of 85 million people every 18 months. It's fifty Septebmer 11ths every day. What if a superintelligent AI can solve for climate change, solve for human cooperation, solve for vastly improved human health, solve for universal basic income which releives the drudgery of living for everyone, solve for immortality, solve for faster than light communication or travel, solve for xyz? How many human lives are the trade against the risk? But my second paragraph is, it doesn't matter whether it's preferable, events are in motion and aren't going to stop to let us off - it's preferable if we don't destroy the climate and kill a billion humans and make life on Earth much more difficult, but that's still on course. To me it's preferable to have clean air to breathe and people not being run over and killed by vehicles, but the market wants city streets for cars and air primarily for burining petrol and diesel and secondarily for humans to breathe and if they get asthsma and lung cancer, tough. I think the same will happen with AI, arguing that everyone should stop because we don't want Grey Goo or Paperclip Maximisers is unlikely to change the course of anything, just as it hasn't changed the course of anything up to now despite years and years and years of raising it as a concern.
- theptip 4y ago
  I think that the benefits of AGI research are often omitted from the analysis, so I'm generally supportive of considering the cost/benefit. However I think you need to do a lot more work than just gesturing in the direction of very high potential benefits to actually convince anyone, in particular since we're dealing with extremely large numbers, that are extremely sensitive to small probabilities. EV = P(AlignedAI) * Utility(AGI) + P(1-AlignedAI) * Utility(ruin) (I'm aware that all I did up-thread was gesture in the direction of risks, but I think "unintended/un-measured existential risks" are in general more urgent to understand than "un-measured huge benefits"; there is no catching up from ruin, but you can often come back later and harvest fruit that you skipped earlier. Ideally we study both of course.)
notahacker 4y ago
If the catastrophic alternative is actually possible, who's to say the waffling academics aren't the ones to cause it? I'm being serious here: the AI model the x-risk people are worrying about here because it waffled about causing harm was originally developed by an entity founded by people with the explicit stated purpose of avoiding AI catastrophe. And one of the most popular things for people seeking x-risk funding to do is to write extremely long and detailed explanations of how and why AI is likely to harm humans. If I worried about the risk of LLMs achieving sentience and forming independent goals to destroy humanity based on the stuff they'd read, I'd want them to do less of that, not fund them to do more.

kelnos 4y ago

A flawed but useful operating system and programming language isn't likely to decide humanity is garbage and launch all nuclear weapons at once. A "worse is better" AGI could cause the end of humanity. I know that sounds overly dramatic, but I'm not remotely convinced that isn't possible, or even isn't likely. I agree with you that "x-risk" research could easily devolve into what you are worried about, but that doesn't mean we should ignore these risks and plow forward.

ShredKazoo 4y ago

>Pandora's box is open, the genie's bottle is uncorked As someone who's followed AI safety for over a decade now, it's been frustrating to see reactions flip from "it's too early to do any useful work!" to "it's too late to do any useful work!", with barely any time intervening. https://www.youtube.com/watch?v=0AW4nSq0hAc https://www.youtube.com/watch?v=0AW4nSq0hAc Perhaps it is worth actually reading a book like this one (posted to HN yesterday) before concluding that it's too late to do anything? https://betterwithout.ai/only-you-can-stop-an-AI-apocalypse https://betterwithout.ai/only-you-can-stop-an-AI-apocalypse

jacquesm 4y ago
This is how it always goes. Similar for climate change and lots of other problems that move slowly compared to the lifetime of a single human.
nradov 4y ago
AI safety is not a legitimate field. You have wasted your time. It's just a bunch of grifters posting alarmist tweets with no scientific evidence. You might as well be following "unicorn safety" or "ghost safety".
- ShredKazoo 4y ago
  Do you think Stuart Russell (coauthor, with Peter Norvig, of the widely used textbook Artificial Intelligence: A Modern Approach) is a grifter? https://people.eecs.berkeley.edu/~russell/research/future/ https://people.eecs.berkeley.edu/~russell/research/future/ Does this review look like it only covers alarmist tweets? https://arxiv.org/pdf/1805.01109.pdf https://arxiv.org/pdf/1805.01109.pdf
  - nradov 4y ago
    Yes, Stuart Russell is a grifter. Some of the more advanced grifters have gone beyond tweeting and are now shilling low-effort books in an attempt to draw attention to themselves. Don't be fooled. If we want to talk about problems with biased data sets or using inappropriate AI algorithms for safety-critical applications then sure, let's address those issues. But the notion of some super intelligent computer coming to take over the world and kill everyone is just a stupid fantasy with no scientific basis.

naasking 4y ago

> Convince me that "x-risk research" won't be a bunch of out of touch academics handwaving and philosophising with their tenure https://scottaaronson.blog/?p=6823 https://scottaaronson.blog/?p=6823