12 ms·

Every time I see Yudkowsky’s writing I wonder why people take him seriously. Just a couple of months back he was duped by a fake story about an airforce AI deci

by bnralt 3y ago

Every time I see Yudkowsky’s writing I wonder why people take him seriously. Just a couple of months back he was duped by a fake story about an airforce AI deciding to kill it’s operators in a simulation. It turned out the whole thing was thought experiment. Asking Yudkowsky what we should do about AI is like asking Green Peace what we should do about nuclear power.

dragonwriter 3y ago
> Every time I see Yudkowsky’s writing I wonder why people take him seriously. Its a perfect thing for rich, extremely privileged people to grab onto as a cause to present thenselves as (and maybe even feel like they are, depending on their capacity for self-delusion) “doing something for humanity” while continuing to ignore the substantive material condition of, as a first order approximation, everyone in the world, and their role in perpetuating it.
- 0x00_NULL 3y ago
  I lost total interest in every “Stop” argument when Elon Musk started a company to develop AGI during the pause that he signed on to and promoted. Yann LeCunn said that this was just an attempt to seize market share among the big companies and he couldn’t have been more right.
  - pclmulqdq 3y ago
    That is transparently what this letter is for most of its signatories. They lost the race (in one way or another) and they want time to catch back up. This is as true for the academics who made big contributions in the past but missed out on the latest stuff as it is for Elon Musk.
    - nineteen999 3y ago
      The joke will be on them when our open-source powered mechwarriors made of bulldozer parts come for them.
      - TeMPOraL 3y ago
        The joke will be on us all when your open-source powered mechwarriors start destroying dams and power plants and even take a stab at launching some nukes, because the models figured this is the most effective way to "get them", and weren't aligned enough to see why they should take a different approach.
        nineteen999 3y ago
        While my tongue was firmly in my cheek, and still is somewhat, whats to stop (for example) Boston Dynamics and OpenAI collaborating and beating us to your nightmare scenario first? And is that a situation you would prefer?
        TeMPOraL 3y ago
        Of course not. I'm leaning towards Yudkowsky's recommendation, which is to put a pause on all of it.
        pclmulqdq 3y ago
        I have a bridge to sell Yudkowsky and everyone else who believes that an "AI pause" is actually something that can happen. Global treaties pretty much all occur with implied threats of violence, and we don't have enough guns to force this across India, China, the middle east, the EU, and the US. Major AI development happens in all of those places.
        TeMPOraL 3y ago
        You're reading this wrong. Yudkowsky isn't saying that this will happen; in fact, it's apparent to him as much as to everyone else how unlikely is that. Yudkowsky's point is that this is the minimum and least difficult thing we need to do. LeCun and others on both sides of the "AI regulation" argument are busy arguing the color of the carpet, while the whole house is on fire.
- robotnikman 3y ago
  [flagged]
belter 3y ago
They will come to it... "Co-Founder of Greenpeace Envisions a Nuclear Future" - https://www.wired.com/2007/11/co-founder-of-greenpeace-envisions-a-nuclear-future/ https://www.wired.com/2007/11/co-founder-of-greenpeace-envis...
TeMPOraL 3y ago
If I had a cent for every time I see a HN semi-regular (i.e. not a throwaway account) falling for some rather transparent bullshit, I'd enjoy my retirement on a significant "FU money" cache... Everyone gets duped sometimes, often publicly. I remember skimming a summary of the story itself. Sounded like textbook failure mode of reinforcement learning. I was actually saddened to later learn it apparently was just a thought experiment, and not a simulated exercise - for a moment I hoped for a more visceral anecdote about a ML model finding an unusual solution after being trained too hard, rather than my go-to "AI beating a game in record time by finding and exploiting timing bugs" kind of stories.
- ben_w 3y ago
  Add an imaginary cent from me, if you've not already. This had me fooled for years: https://users.math.yale.edu/public_html/People/frame/Fractals/MandelSet/MandelMonk/MandelMonk.html https://users.math.yale.edu/public_html/People/frame/Fractal...
bondarchuk 3y ago
>Every time I see Yudkowsky’s writing I wonder why people take him seriously. Because he identified the potential risk of superhuman AI 20 years before almost everyone else. Sure, science fiction also identified that risk (as people here seem eager to point out), but he identified it as an actual real-world risk and has been trying to take real-world action on it for 20 years. No matter what else you think about him or the specifics of his ideas, for me that counts for something.
- throwaway9274 3y ago
  Twenty years ago was 2003. There are dozens of examples of AI x-risk thought experiments dating back to 1960. As far as I can tell, all he did was open a forum for people to write fanfic about these earlier ideas.
  - bondarchuk 3y ago
    I opened the lesswrong homepage through archive.org on an arbitrary date in 2010, where I found a post about raising $100000 for a grant program. Most of that went towards papers and fellowships at the singularity institute AFAICT. https://web.archive.org/web/20100304171507/http://lesswrong.com/lw/1tn/the_last_days_of_the_singularity_challenge/ https://web.archive.org/web/20100304171507/http://lesswrong....
- jonathankoren 3y ago
  But it’s not a real world risk. Arguably, it’s even less of a real world risk because it’s a common sci-fi trope. The whole concept is farcical.
  - ben_w 3y ago
    "Hacking" is also a common sci-fi trope, I don't think you can infer much from it appearing in sci-fi beyond "it makes for good sci-fi". As for real world risk… well, an AI with some kind of personality disorder might not be much of a realistic risk today, but even assuming it never is, there are still plenty of GOFAI that have gone wrong, either killing people directly when their bugs caused them to give patients lethal radiation doses, or nearly triggering WW3 because they weren't programmed to recognise the moon wasn't a Soviet bomber group, or causing economic catastrophes because the investors all reacted to the phrase "won the Nobel prize for economics" as a thought-terminating cliché, or categorising humans as gorillas if they had too much melanin[0], or promoted messages encouraging genocide to groups vulnerable to such messages thanks to a lack of oversight and a language barrier between platform owners and the messages in question. [0] Asimov's three laws was a plot device, and most of the works famously show how they can go wrong. Genuine question as I've not read them all: did that kind of failure mode ever come up in his fiction?
nyc_data_geek1 3y ago
Anybody subscribing to twitter is immediately disqualified as a credible, rational actor in my book.
- smilliken 3y ago
  A good heuristic if you care more about false positives than false negatives.
librexpr 3y ago
Yudkowsky was not fooled. He made three tweets on this subject in that time frame, which can all be seen using this link: https://nitter.net/ESYudkowsky/search?f=tweets&since=2023-05-30&until=2023-06-03 https://nitter.net/ESYudkowsky/search?f=tweets&since=2023-05... First tweet: > ...can we get confirmation on this being real? https://nitter.net/ESYudkowsky/status/1664313290317795330#m https://nitter.net/ESYudkowsky/status/1664313290317795330#m Second tweet, which is a reply: > Good they tested in sim, bad they didn't see it coming given how much the entire alignment field plus the previous fifty years of SF were warning in advance about this exact case https://nitter.net/ESYudkowsky/status/1664357633762140160#m https://nitter.net/ESYudkowsky/status/1664357633762140160#m Third tweet: > Disconfirmed. https://nitter.net/ESYudkowsky/status/1664639807002214401#m https://nitter.net/ESYudkowsky/status/1664639807002214401#m The second tweet does not explicitly say "conditional on this turning out to be real", but given that the immediately preceding tweet was expressing doubt, it is implicit from the context that that is what he meant.
firecall 3y ago
Indeed, that entire tweet was inane.

by bnralt 3y ago

dragonwriter 3y ago

> Every time I see Yudkowsky’s writing I wonder why people take him seriously. Its a perfect thing for rich, extremely privileged people to grab onto as a cause to present thenselves as (and maybe even feel like they are, depending on their capacity for self-delusion) “doing something for humanity” while continuing to ignore the substantive material condition of, as a first order approximation, everyone in the world, and their role in perpetuating it.

0x00_NULL 3y ago
I lost total interest in every “Stop” argument when Elon Musk started a company to develop AGI during the pause that he signed on to and promoted. Yann LeCunn said that this was just an attempt to seize market share among the big companies and he couldn’t have been more right.
- pclmulqdq 3y ago
  That is transparently what this letter is for most of its signatories. They lost the race (in one way or another) and they want time to catch back up. This is as true for the academics who made big contributions in the past but missed out on the latest stuff as it is for Elon Musk.
  - nineteen999 3y ago
    The joke will be on them when our open-source powered mechwarriors made of bulldozer parts come for them.
    - TeMPOraL 3y ago
      The joke will be on us all when your open-source powered mechwarriors start destroying dams and power plants and even take a stab at launching some nukes, because the models figured this is the most effective way to "get them", and weren't aligned enough to see why they should take a different approach.
      - nineteen999 3y ago
        While my tongue was firmly in my cheek, and still is somewhat, whats to stop (for example) Boston Dynamics and OpenAI collaborating and beating us to your nightmare scenario first? And is that a situation you would prefer?
        TeMPOraL 3y ago
        Of course not. I'm leaning towards Yudkowsky's recommendation, which is to put a pause on all of it.
        pclmulqdq 3y ago
        I have a bridge to sell Yudkowsky and everyone else who believes that an "AI pause" is actually something that can happen. Global treaties pretty much all occur with implied threats of violence, and we don't have enough guns to force this across India, China, the middle east, the EU, and the US. Major AI development happens in all of those places.
        TeMPOraL 3y ago
        You're reading this wrong. Yudkowsky isn't saying that this will happen; in fact, it's apparent to him as much as to everyone else how unlikely is that. Yudkowsky's point is that this is the minimum and least difficult thing we need to do. LeCun and others on both sides of the "AI regulation" argument are busy arguing the color of the carpet, while the whole house is on fire.
robotnikman 3y ago
[flagged]

belter 3y ago

They will come to it... "Co-Founder of Greenpeace Envisions a Nuclear Future" - https://www.wired.com/2007/11/co-founder-of-greenpeace-envisions-a-nuclear-future/ https://www.wired.com/2007/11/co-founder-of-greenpeace-envis...

TeMPOraL 3y ago

If I had a cent for every time I see a HN semi-regular (i.e. not a throwaway account) falling for some rather transparent bullshit, I'd enjoy my retirement on a significant "FU money" cache... Everyone gets duped sometimes, often publicly. I remember skimming a summary of the story itself. Sounded like textbook failure mode of reinforcement learning. I was actually saddened to later learn it apparently was just a thought experiment, and not a simulated exercise - for a moment I hoped for a more visceral anecdote about a ML model finding an unusual solution after being trained too hard, rather than my go-to "AI beating a game in record time by finding and exploiting timing bugs" kind of stories.

ben_w 3y ago
Add an imaginary cent from me, if you've not already. This had me fooled for years: https://users.math.yale.edu/public_html/People/frame/Fractals/MandelSet/MandelMonk/MandelMonk.html https://users.math.yale.edu/public_html/People/frame/Fractal...

bondarchuk 3y ago

>Every time I see Yudkowsky’s writing I wonder why people take him seriously. Because he identified the potential risk of superhuman AI 20 years before almost everyone else. Sure, science fiction also identified that risk (as people here seem eager to point out), but he identified it as an actual real-world risk and has been trying to take real-world action on it for 20 years. No matter what else you think about him or the specifics of his ideas, for me that counts for something.

throwaway9274 3y ago
Twenty years ago was 2003. There are dozens of examples of AI x-risk thought experiments dating back to 1960. As far as I can tell, all he did was open a forum for people to write fanfic about these earlier ideas.
- bondarchuk 3y ago
  I opened the lesswrong homepage through archive.org on an arbitrary date in 2010, where I found a post about raising $100000 for a grant program. Most of that went towards papers and fellowships at the singularity institute AFAICT. https://web.archive.org/web/20100304171507/http://lesswrong.com/lw/1tn/the_last_days_of_the_singularity_challenge/ https://web.archive.org/web/20100304171507/http://lesswrong....
jonathankoren 3y ago
But it’s not a real world risk. Arguably, it’s even less of a real world risk because it’s a common sci-fi trope. The whole concept is farcical.
- ben_w 3y ago
  "Hacking" is also a common sci-fi trope, I don't think you can infer much from it appearing in sci-fi beyond "it makes for good sci-fi". As for real world risk… well, an AI with some kind of personality disorder might not be much of a realistic risk today, but even assuming it never is, there are still plenty of GOFAI that have gone wrong, either killing people directly when their bugs caused them to give patients lethal radiation doses, or nearly triggering WW3 because they weren't programmed to recognise the moon wasn't a Soviet bomber group, or causing economic catastrophes because the investors all reacted to the phrase "won the Nobel prize for economics" as a thought-terminating cliché, or categorising humans as gorillas if they had too much melanin[0], or promoted messages encouraging genocide to groups vulnerable to such messages thanks to a lack of oversight and a language barrier between platform owners and the messages in question. [0] Asimov's three laws was a plot device, and most of the works famously show how they can go wrong. Genuine question as I've not read them all: did that kind of failure mode ever come up in his fiction?

nyc_data_geek1 3y ago

Anybody subscribing to twitter is immediately disqualified as a credible, rational actor in my book.

smilliken 3y ago
A good heuristic if you care more about false positives than false negatives.

librexpr 3y ago

Yudkowsky was not fooled. He made three tweets on this subject in that time frame, which can all be seen using this link: https://nitter.net/ESYudkowsky/search?f=tweets&since=2023-05-30&until=2023-06-03 https://nitter.net/ESYudkowsky/search?f=tweets&since=2023-05... First tweet: > ...can we get confirmation on this being real? https://nitter.net/ESYudkowsky/status/1664313290317795330#m https://nitter.net/ESYudkowsky/status/1664313290317795330#m Second tweet, which is a reply: > Good they tested in sim, bad they didn't see it coming given how much the entire alignment field plus the previous fifty years of SF were warning in advance about this exact case https://nitter.net/ESYudkowsky/status/1664357633762140160#m https://nitter.net/ESYudkowsky/status/1664357633762140160#m Third tweet: > Disconfirmed. https://nitter.net/ESYudkowsky/status/1664639807002214401#m https://nitter.net/ESYudkowsky/status/1664639807002214401#m The second tweet does not explicitly say "conditional on this turning out to be real", but given that the immediately preceding tweet was expressing doubt, it is implicit from the context that that is what he meant.

firecall 3y ago

Indeed, that entire tweet was inane.