5 ms·

>Every time I see Yudkowsky’s writing I wonder why people take him seriously. Because he identified the potential risk of superhuman AI 20 years before almost

>Every time I see Yudkowsky’s writing I wonder why people take him seriously. Because he identified the potential risk of superhuman AI 20 years before almost everyone else. Sure, science fiction also identified that risk (as people here seem eager to point out), but he identified it as an actual real-world risk and has been trying to take real-world action on it for 20 years. No matter what else you think about him or the specifics of his ideas, for me that counts for something.

throwaway9274 3y ago
Twenty years ago was 2003. There are dozens of examples of AI x-risk thought experiments dating back to 1960. As far as I can tell, all he did was open a forum for people to write fanfic about these earlier ideas.
- bondarchuk 3y ago
  I opened the lesswrong homepage through archive.org on an arbitrary date in 2010, where I found a post about raising $100000 for a grant program. Most of that went towards papers and fellowships at the singularity institute AFAICT. https://web.archive.org/web/20100304171507/http://lesswrong.com/lw/1tn/the_last_days_of_the_singularity_challenge/ https://web.archive.org/web/20100304171507/http://lesswrong....
jonathankoren 3y ago
But it’s not a real world risk. Arguably, it’s even less of a real world risk because it’s a common sci-fi trope. The whole concept is farcical.
- ben_w 3y ago
  "Hacking" is also a common sci-fi trope, I don't think you can infer much from it appearing in sci-fi beyond "it makes for good sci-fi". As for real world risk… well, an AI with some kind of personality disorder might not be much of a realistic risk today, but even assuming it never is, there are still plenty of GOFAI that have gone wrong, either killing people directly when their bugs caused them to give patients lethal radiation doses, or nearly triggering WW3 because they weren't programmed to recognise the moon wasn't a Soviet bomber group, or causing economic catastrophes because the investors all reacted to the phrase "won the Nobel prize for economics" as a thought-terminating cliché, or categorising humans as gorillas if they had too much melanin[0], or promoted messages encouraging genocide to groups vulnerable to such messages thanks to a lack of oversight and a language barrier between platform owners and the messages in question. [0] Asimov's three laws was a plot device, and most of the works famously show how they can go wrong. Genuine question as I've not read them all: did that kind of failure mode ever come up in his fiction?
- bondarchuk 3y ago
  >Arguably, it’s even less of a real world risk because it’s a common sci-fi trope. It would be nice if the world worked that way. Then we could diminish any risk just by writing a lot of scifi about it :)
  - sdwr 3y ago
    That's exactly how it works. Invest thought into it before it happens, then when it becomes an issue, there's already awareness and possibilities have been mapped out.
    - TeMPOraL 3y ago
      Unless, of course, people then call decades of slow awareness building a "Silicon Valley conspiracy" of "AI one-percenters", or equivalent, as witnessed in this very thread...
dghlsakjg 3y ago
Hal 9000, the malevolent AI in 2001: a space Odyssey, was in one of the most respected films of all time three years before Yudkowsky was born. Plenty of people have been beating that drum for years.
- bondarchuk 3y ago
  I addressed science fiction in my comment. The crucial difference is taking it seriously and actually trying to do something tangible about slowing/stopping AI progress, which is not something Stanley Kubrick did.
  - varelse 3y ago
    [dead]
  - dghlsakjg 3y ago
    You edited the comment after I posted. Your original made no mention of sci-fi, and made it sound as if no one had imagined a malicious AI before yudkowski
    - bondarchuk 3y ago
      I did not edit my comment. (edit: you can see that https://news.ycombinator.com/item?id=38116456 https://news.ycombinator.com/item?id=38116456 responds to the scifi part of my comment as well and was posted before yours https://news.ycombinator.com/item?id=38116494 https://news.ycombinator.com/item?id=38116494 (if my assumption that ids are chronological is correct))
- oceanplexian 3y ago
  Mary Shelly wrote about the same philosophical issue in 1818 when she wrote Frankenstein. My take is that AI influencers don’t read a lot of books, because if they did they’d have much more intelligent commentary.
  - ben_w 3y ago
daveguy 3y ago
A poorly premised fantasy fiction horror story about what might happen during the interaction between a super intelligence and its creator makes for a great story, but it's not a blueprint for the future, much less a pressing concern. Misuse of current AI is far more concerning than anything dreamt up about AGI.
- bondarchuk 3y ago
  Yeah, for me EY's stories about an airgapped AI convincing their keepers to release it (or that other one about an AI destroying all humans because it's the most efficient way to make a gazillion paperclips) also miss the point. It's mostly the just the realization that superhuman AI might be dangerous through some kind of runaway feedback, as a very general idea, that I respect about EY's work.

throwaway9274 3y ago

Twenty years ago was 2003. There are dozens of examples of AI x-risk thought experiments dating back to 1960. As far as I can tell, all he did was open a forum for people to write fanfic about these earlier ideas.

bondarchuk 3y ago
I opened the lesswrong homepage through archive.org on an arbitrary date in 2010, where I found a post about raising $100000 for a grant program. Most of that went towards papers and fellowships at the singularity institute AFAICT. https://web.archive.org/web/20100304171507/http://lesswrong.com/lw/1tn/the_last_days_of_the_singularity_challenge/ https://web.archive.org/web/20100304171507/http://lesswrong....

jonathankoren 3y ago

But it’s not a real world risk. Arguably, it’s even less of a real world risk because it’s a common sci-fi trope. The whole concept is farcical.

ben_w 3y ago
"Hacking" is also a common sci-fi trope, I don't think you can infer much from it appearing in sci-fi beyond "it makes for good sci-fi". As for real world risk… well, an AI with some kind of personality disorder might not be much of a realistic risk today, but even assuming it never is, there are still plenty of GOFAI that have gone wrong, either killing people directly when their bugs caused them to give patients lethal radiation doses, or nearly triggering WW3 because they weren't programmed to recognise the moon wasn't a Soviet bomber group, or causing economic catastrophes because the investors all reacted to the phrase "won the Nobel prize for economics" as a thought-terminating cliché, or categorising humans as gorillas if they had too much melanin[0], or promoted messages encouraging genocide to groups vulnerable to such messages thanks to a lack of oversight and a language barrier between platform owners and the messages in question. [0] Asimov's three laws was a plot device, and most of the works famously show how they can go wrong. Genuine question as I've not read them all: did that kind of failure mode ever come up in his fiction?
bondarchuk 3y ago
>Arguably, it’s even less of a real world risk because it’s a common sci-fi trope. It would be nice if the world worked that way. Then we could diminish any risk just by writing a lot of scifi about it :)
- sdwr 3y ago
  That's exactly how it works. Invest thought into it before it happens, then when it becomes an issue, there's already awareness and possibilities have been mapped out.
  - TeMPOraL 3y ago
    Unless, of course, people then call decades of slow awareness building a "Silicon Valley conspiracy" of "AI one-percenters", or equivalent, as witnessed in this very thread...

dghlsakjg 3y ago

Hal 9000, the malevolent AI in 2001: a space Odyssey, was in one of the most respected films of all time three years before Yudkowsky was born. Plenty of people have been beating that drum for years.

bondarchuk 3y ago
I addressed science fiction in my comment. The crucial difference is taking it seriously and actually trying to do something tangible about slowing/stopping AI progress, which is not something Stanley Kubrick did.
- varelse 3y ago
  [dead]
- dghlsakjg 3y ago
  You edited the comment after I posted. Your original made no mention of sci-fi, and made it sound as if no one had imagined a malicious AI before yudkowski
  - bondarchuk 3y ago
    I did not edit my comment. (edit: you can see that https://news.ycombinator.com/item?id=38116456 https://news.ycombinator.com/item?id=38116456 responds to the scifi part of my comment as well and was posted before yours https://news.ycombinator.com/item?id=38116494 https://news.ycombinator.com/item?id=38116494 (if my assumption that ids are chronological is correct))
oceanplexian 3y ago
Mary Shelly wrote about the same philosophical issue in 1818 when she wrote Frankenstein. My take is that AI influencers don’t read a lot of books, because if they did they’d have much more intelligent commentary.
- ben_w 3y ago

daveguy 3y ago

A poorly premised fantasy fiction horror story about what might happen during the interaction between a super intelligence and its creator makes for a great story, but it's not a blueprint for the future, much less a pressing concern. Misuse of current AI is far more concerning than anything dreamt up about AGI.

bondarchuk 3y ago
Yeah, for me EY's stories about an airgapped AI convincing their keepers to release it (or that other one about an AI destroying all humans because it's the most efficient way to make a gazillion paperclips) also miss the point. It's mostly the just the realization that superhuman AI might be dangerous through some kind of runaway feedback, as a very general idea, that I respect about EY's work.