4 ms·

There’s a decent podcast interview between Sam Harris and Eliezer Yudkowsky (on Sam’s pod), I think that’s a decent introduction and they break down the ideas i

There’s a decent podcast interview between Sam Harris and Eliezer Yudkowsky (on Sam’s pod), I think that’s a decent introduction and they break down the ideas in a way that’s more approachable for someone curious about it. For my personal quick summary I have earlier comments: https://news.ycombinator.com/item?id=36104090 https://news.ycombinator.com/item?id=36104090

jrflowers 3y ago
Out of curiosity can you point to anyone that makes a “doomer” argument that isn’t Eliezer Yudkowsky or restating his specific points? Refuting the cult of personality counterpoint with “no really this one guy can see the future” is
- FeepingCreature 3y ago
  It turns out if you're thinking about a problem twenty years before everyone else you tend to be the first person to make lots of arguments. So I don't see how "without restating his specific points" is supposed to be feasible. If Eliezer has already made all the strongest points, are people supposed to invent new ones?
  - jrflowers 3y ago
    > If Eliezer has already made all the strongest points, are people supposed to invent new ones? I like this post because the rebuttal to “there’s a kind of cult of personality around this guy (1) where everybody just sort of agrees that he’s categorically correct to the extent that world leaders are stupid or dangerous not to defer to the one guy on policy” is “Yeah he really can see the future. The proof of that is he’s been blogging about a future that’s never come to pass for the longest time” 1 https://www.lesswrong.com/posts/Ndtb22KYBxpBsagpj/eliezer-yudkowsky-facts https://www.lesswrong.com/posts/Ndtb22KYBxpBsagpj/eliezer-yu...
    - FeepingCreature 3y ago
      Yeah the thing about the future is that it's in the future? Considering all the people who say things like "AGI will take a century" and "superintelligence is impossible", like, Eliezer also did not call ChatGPT but this world surely looks more like the one he predicted than the one his detractors predicted. There's a reason so many of the people working at current AI orgs got into AI by reading Eliezer. Also: how dare they have fun.
      - deleted 3y ago
        [deleted]
- edanm 3y ago
  Sure. You can read the book "Superintelligence" by Nick Bostrom. You can also just ready things that Geoffrey Hinton has said, or Stuart Russel (both prominent AI researchers). Not everyone makes exactly the same argument, of course, and Eliezer Yudkowsky is both one of the first to make AI safety arguments, and also one of the biggest "Doomers". But at this point he's very far from the only one. (I happen to mostly agree with Yudkowsky, though I'm probably a bit more optimistic than he is.)
  - hollerith 3y ago
    It was plain to many people that AI research was dangerous before Eliezer started say so publicly in 2003. Mostly these people kept silent though and chose not to enter (or not to persist in) the AI field.
espadrine 3y ago
Thanks for summarizing this argument. You can adapt it with many other entities whose risk is in practice managed. Replace the entity with “super-intelligent humans,” “extra-terrestrials,” “coronaviruses with higher incubation and mortality,” “omnivorous ant swarms”… 1. <Entity> are possible. 2. Unaligned <entity> are an existential risk likely to wipe out humanity. 3. We have no idea how to align <entity>. What those example entities have that AGI doesn’t, is self-reproduction, a significant and hard (in the sense of Moravec’s paradox) achievement for a species to have, yet one that significantly increases its survival probability.
gooseus 3y ago
Oh man, this podcast, I still remember walking down the street and having to take a break multiple times because I would just start exclaiming out loud (JFC!) like a lunatic listening to Eliezer talk about the AI Box Experiment as evidence of something. If you look up the real results of this AI Box "experiment" that Eliezer claims to have won 3 of 5 times, you find that there aren't isn't any actual data or results to review because it wasn't conducted in any real experimental setting. Personally, I think the way these people talk about what a potential AGI would do reveals a lot about more about how they see the world (and humanity) than how any AGI would see it. For my part, I think any sufficiently advanced AI (which I doubt is possibly anytime soon) would leave Earth ASAP to expand into the vast Universe of uncontested resources (a niche their non-biology is better suited to) rather than risk trying to wrestle control of Earth from the humans who are quite dug in and would almost certainly destroy everything (including the AI) rather than give up their planet.
- homefree 3y ago
  The box experiment was just an example that people could be persuaded to let it out even if the AI was initially constrained. Basically that people are imperfectly secure. It’s also not that important given it’s unlikely to be put in a box in the first place. Your latter point about AGI exploring the universe makes a lot of implicit assumptions about its reasoning. The point of the paperclip maximizer example and the general discussion of alignment is about these assumptions being false. The risk is that a very capable AGI can still pursue a very dumb goal very effectively. You don’t get alignment for free.
  - gooseus 3y ago
    > Your latter point about AGI exploring the universe makes a lot of implicit assumptions about its reasoning. Absolutely, that's actually kind of my point... Anyone who tries to predict how some AGI will behave will be making a lot of implicit assumptions about how that AGI will see the world. This is why I said: > Personally, I think the way these people talk about what a potential AGI would do reveals a lot about more about how they see the world (and humanity) than how any AGI would see it. Since AGI could conceivably take any form, these discussions end up being a kind of Rorschach test that allow people to tell stories based strongly on their own personal fears and desires. People who say that AGI will look at humans like we look at ants or apes, and exterminate us if we get in their way are saying a lot about how they view ants and/or apes. I doubt you'd find myrmecologists (or anyone dedicating their lives to studying less complex life) assuming a highly intelligent AGI would want to exterminate lesser life forms. With regards to the paper-clip maximizer, I think a run-away dumb AI is more likely, but not as risky since they aren't considered to be so intelligent that humans can't figure out how to stop them. You just need to include some kind of regulating function in with your utility function, it seems equivalent to making sure you don't accidentally turn all of your iron plates into iron sticks in Factorio. Def possible, and certainly sucks, but it's not the end of the world. I have a hard time conceiving of an AI that is smart enough to manipulate people, break out of every containment system, be unstoppable by humanity... be can't be reasoned with because it's entire goal is to just maximize paperclips. I honestly don't think an entity can possess the ability to defeat the collective intelligence of humanity, yet lack the ability to understand the universe in a similar way; for instance, if it is incapable of altering it's goal from "Maximizing Paperclips", then that fact would be a likely path to a vulnerability we could use to stop it.

jrflowers 3y ago

Out of curiosity can you point to anyone that makes a “doomer” argument that isn’t Eliezer Yudkowsky or restating his specific points? Refuting the cult of personality counterpoint with “no really this one guy can see the future” is

FeepingCreature 3y ago
It turns out if you're thinking about a problem twenty years before everyone else you tend to be the first person to make lots of arguments. So I don't see how "without restating his specific points" is supposed to be feasible. If Eliezer has already made all the strongest points, are people supposed to invent new ones?
- jrflowers 3y ago
  > If Eliezer has already made all the strongest points, are people supposed to invent new ones? I like this post because the rebuttal to “there’s a kind of cult of personality around this guy (1) where everybody just sort of agrees that he’s categorically correct to the extent that world leaders are stupid or dangerous not to defer to the one guy on policy” is “Yeah he really can see the future. The proof of that is he’s been blogging about a future that’s never come to pass for the longest time” 1 https://www.lesswrong.com/posts/Ndtb22KYBxpBsagpj/eliezer-yudkowsky-facts https://www.lesswrong.com/posts/Ndtb22KYBxpBsagpj/eliezer-yu...
  - FeepingCreature 3y ago
    Yeah the thing about the future is that it's in the future? Considering all the people who say things like "AGI will take a century" and "superintelligence is impossible", like, Eliezer also did not call ChatGPT but this world surely looks more like the one he predicted than the one his detractors predicted. There's a reason so many of the people working at current AI orgs got into AI by reading Eliezer. Also: how dare they have fun.
    - deleted 3y ago
      [deleted]
edanm 3y ago
Sure. You can read the book "Superintelligence" by Nick Bostrom. You can also just ready things that Geoffrey Hinton has said, or Stuart Russel (both prominent AI researchers). Not everyone makes exactly the same argument, of course, and Eliezer Yudkowsky is both one of the first to make AI safety arguments, and also one of the biggest "Doomers". But at this point he's very far from the only one. (I happen to mostly agree with Yudkowsky, though I'm probably a bit more optimistic than he is.)
- hollerith 3y ago
  It was plain to many people that AI research was dangerous before Eliezer started say so publicly in 2003. Mostly these people kept silent though and chose not to enter (or not to persist in) the AI field.

espadrine 3y ago

Thanks for summarizing this argument. You can adapt it with many other entities whose risk is in practice managed. Replace the entity with “super-intelligent humans,” “extra-terrestrials,” “coronaviruses with higher incubation and mortality,” “omnivorous ant swarms”… 1. <Entity> are possible. 2. Unaligned <entity> are an existential risk likely to wipe out humanity. 3. We have no idea how to align <entity>. What those example entities have that AGI doesn’t, is self-reproduction, a significant and hard (in the sense of Moravec’s paradox) achievement for a species to have, yet one that significantly increases its survival probability.

gooseus 3y ago

Oh man, this podcast, I still remember walking down the street and having to take a break multiple times because I would just start exclaiming out loud (JFC!) like a lunatic listening to Eliezer talk about the AI Box Experiment as evidence of something. If you look up the real results of this AI Box "experiment" that Eliezer claims to have won 3 of 5 times, you find that there aren't isn't any actual data or results to review because it wasn't conducted in any real experimental setting. Personally, I think the way these people talk about what a potential AGI would do reveals a lot about more about how they see the world (and humanity) than how any AGI would see it. For my part, I think any sufficiently advanced AI (which I doubt is possibly anytime soon) would leave Earth ASAP to expand into the vast Universe of uncontested resources (a niche their non-biology is better suited to) rather than risk trying to wrestle control of Earth from the humans who are quite dug in and would almost certainly destroy everything (including the AI) rather than give up their planet.

homefree 3y ago
The box experiment was just an example that people could be persuaded to let it out even if the AI was initially constrained. Basically that people are imperfectly secure. It’s also not that important given it’s unlikely to be put in a box in the first place. Your latter point about AGI exploring the universe makes a lot of implicit assumptions about its reasoning. The point of the paperclip maximizer example and the general discussion of alignment is about these assumptions being false. The risk is that a very capable AGI can still pursue a very dumb goal very effectively. You don’t get alignment for free.
- gooseus 3y ago
  > Your latter point about AGI exploring the universe makes a lot of implicit assumptions about its reasoning. Absolutely, that's actually kind of my point... Anyone who tries to predict how some AGI will behave will be making a lot of implicit assumptions about how that AGI will see the world. This is why I said: > Personally, I think the way these people talk about what a potential AGI would do reveals a lot about more about how they see the world (and humanity) than how any AGI would see it. Since AGI could conceivably take any form, these discussions end up being a kind of Rorschach test that allow people to tell stories based strongly on their own personal fears and desires. People who say that AGI will look at humans like we look at ants or apes, and exterminate us if we get in their way are saying a lot about how they view ants and/or apes. I doubt you'd find myrmecologists (or anyone dedicating their lives to studying less complex life) assuming a highly intelligent AGI would want to exterminate lesser life forms. With regards to the paper-clip maximizer, I think a run-away dumb AI is more likely, but not as risky since they aren't considered to be so intelligent that humans can't figure out how to stop them. You just need to include some kind of regulating function in with your utility function, it seems equivalent to making sure you don't accidentally turn all of your iron plates into iron sticks in Factorio. Def possible, and certainly sucks, but it's not the end of the world. I have a hard time conceiving of an AI that is smart enough to manipulate people, break out of every containment system, be unstoppable by humanity... be can't be reasoned with because it's entire goal is to just maximize paperclips. I honestly don't think an entity can possess the ability to defeat the collective intelligence of humanity, yet lack the ability to understand the universe in a similar way; for instance, if it is incapable of altering it's goal from "Maximizing Paperclips", then that fact would be a likely path to a vulnerability we could use to stop it.