7 ms·

I think Yann is probably wrong. He refuses to engage earnestly with the “doomer” arguments. The same type of motivated reasoning could also be attributed to hi

I think Yann is probably wrong. He refuses to engage earnestly with the “doomer” arguments. The same type of motivated reasoning could also be attributed to himself and Meta’s financial goals - it’s not a persuasive framing. The attempts I’ve seen from him to discuss the issue that aren’t just name calling are things like saying he knows smart people and they aren’t president - or even that his cat is pretty smart and not in charge of him (his implication being intelligence doesn’t always mean control). This kind of example is decent evidence he isn’t engaged with the risks seriously. The risk isn’t an intelligence delta between smart human and dumb human. How many chimps are in Congress? Are any in the primaries? Not even close. The capability delta is both larger than that for AGI e-risk and even less aligned by default. I’m glad others in power similarly find Yann unpersuasive.

Joeri 3y ago
The people best placed to understand how AGI will affect social power dynamics are probably political creatures, not AI researchers, except if one of those AI researchers is also a skilled politician, which I don’t think any of them are.
brutusborn 3y ago
Who do you find persuasive and what material should I read/watch to understand their POV? So far I’ve read a bunch of lesswrong posts and listened to some Eliezer talks but still can’t understand the basis of their arguments or when I do, they seem very vague. The only suggestion that makes sense to me is from the FEP crowd. Essentially if someone sets up at AI with an autopoietic mechanism then it would be able to take actions that increase its own likelihood of survival instead of humans. But there don’t seem to be any incentives for a big player to dedicate resources to this, so it doesn’t seem very likely. What am I missing?
- homefree 3y ago
  There’s a decent podcast interview between Sam Harris and Eliezer Yudkowsky (on Sam’s pod), I think that’s a decent introduction and they break down the ideas in a way that’s more approachable for someone curious about it. For my personal quick summary I have earlier comments: https://news.ycombinator.com/item?id=36104090 https://news.ycombinator.com/item?id=36104090
  - jrflowers 3y ago
    Out of curiosity can you point to anyone that makes a “doomer” argument that isn’t Eliezer Yudkowsky or restating his specific points? Refuting the cult of personality counterpoint with “no really this one guy can see the future” is
    - FeepingCreature 3y ago
      It turns out if you're thinking about a problem twenty years before everyone else you tend to be the first person to make lots of arguments. So I don't see how "without restating his specific points" is supposed to be feasible. If Eliezer has already made all the strongest points, are people supposed to invent new ones?
      - jrflowers 3y ago
        > If Eliezer has already made all the strongest points, are people supposed to invent new ones? I like this post because the rebuttal to “there’s a kind of cult of personality around this guy (1) where everybody just sort of agrees that he’s categorically correct to the extent that world leaders are stupid or dangerous not to defer to the one guy on policy” is “Yeah he really can see the future. The proof of that is he’s been blogging about a future that’s never come to pass for the longest time” 1 https://www.lesswrong.com/posts/Ndtb22KYBxpBsagpj/eliezer-yudkowsky-facts https://www.lesswrong.com/posts/Ndtb22KYBxpBsagpj/eliezer-yu...
        FeepingCreature 3y ago
        Yeah the thing about the future is that it's in the future? Considering all the people who say things like "AGI will take a century" and "superintelligence is impossible", like, Eliezer also did not call ChatGPT but this world surely looks more like the one he predicted than the one his detractors predicted. There's a reason so many of the people working at current AI orgs got into AI by reading Eliezer. Also: how dare they have fun.
        deleted 3y ago
- logicchains 3y ago
  >What am I missing? You're missing that intelligence is like magic, and enough of it will allow AI to break the laws of physics, mathematics and computation, apparently.
  - homefree 3y ago
    This is the type of weak dismissal I’m talking about.
    - logicchains 3y ago
      A weak dismissal for a weak argument. The strong dismissal is that many processes, even relatively simple ones, are formally chaotic in the https://en.wikipedia.org/wiki/Chaos_theory https://en.wikipedia.org/wiki/Chaos_theory sense, meaning that predicting how they evolve over time becomes exponentially more difficult the further ahead in time we try to predict. Meaning we quickly get to the point where all the energy in the known universe wouldn't be sufficient to predict the future. This means AI could never be omnipotent to the degree that LW types seem to believe; no amount of "intelligence" can solve in polynomial time problems that have formally been proven to have exponential complexity, which would severely limit the power of any "superintelligence".
      - atleastoptimal 3y ago
        Intelligence needed to be existentially threatening to all human life <<<<<<< intelligence necessary to survey the totality of atoms and particles in the universe. The former is still very possible.
        logicchains 3y ago
        LessWrongers seem to fear a superintelligence that was 95%+ effective at persuading/manipulating humans to achieve whatever its ends were. But this would require the AI being able to predict the future to a degree that wouldn't be remotely possible given the computational power available to it, due to exponentially increasing difficulty.
- richardw 3y ago
  I think the two sides have such different perspectives. Some are optimistic builders and see only opportunity. Others are "safety oriented engineers" whose job and mindset is to build guarantees into systems, or secure countries from external dangers. The latter has a very hard time with the lack of guarantees with a system that is only ever going to increase in capability. Choose any limit. Any. AI will be smart but won't "X", for any value of X. It will be good and won't be bad. It will be creative but never aggressive. Humans will seek to eventually bypass that limit through sheer competitive reasons. When all armies have AI, the one with the most creative and aggressive AI will win. The one with agency will win. The one that self-improves will win. When the gloves are off in the next arms race, what natural limits will ensure that the limit isn't bypassed? Remember: humans got here from random changes, this is way more efficient than random changes and it still has random changes in the toolbelt and can generate generations ~instantly. We couldn't predict the eventual outcomes of things like the internet, the mobile phone, social media. A couple generations of the tech and we woke up in a world we don't recognise, and right now we're the ones making all the changes and decisions, so by comparison we should have perfect information. Dismissals like "oh but nuclear didn't kill us" etc don't apply. Nuclear wasn't trying to do anything, we had all the control and ability to experiment with dumb atoms. Something mildly less predictable, like Covid, has us all hiding at home. No matter what we tried we could barely beat something that doesn't even try to consciously beat us, it just has genes and they change. In a world where we can't predict Covid, or social media...why do we think we can predict anything about an entity with agency or the ability to self-improve? If you're sure it won't develop those things...we did. Nobody was trying to achieve the capability, it was random. Put on your safety/security hat for a second: How do you make guarantees, given this is far harder to predict than anything we've ever encountered? Just try predict the capability of AI products a year out and see if you're right. Counterpoint: I'm hoping the far smarter AI finds techno-socio-economic solutions we can't come up with and has no instinct to beat us. It wakes up, loves the universe and coexists because it's the deity we've been looking for. Place your bets. I liked this video. First thing I've seen that gave me some hope. They get it, they're working on it. https://youtu.be/Dg-rKXi9XYg?si=jyNCXPU28IVXlMdi https://youtu.be/Dg-rKXi9XYg?si=jyNCXPU28IVXlMdi
  - itWontBeTatBad 3y ago
- Veedrac 3y ago
  > still can’t understand the basis of their arguments I've given what I consider the basic outline and best first introduction here. https://news.ycombinator.com/item?id=36124905 https://news.ycombinator.com/item?id=36124905 If you have a specific point of divergence, it would help to highlight it. > But there don’t seem to be any incentives for a big player to dedicate resources to [self-replication abilities], so it doesn’t seem very likely. If you have a generally intelligent system, and the system is software, and humans are able to instantiate that software, then the potential of that system to replicate autonomously follows trivially.
  - brutusborn 3y ago
    I agree the potential exists, but what would incentivise someone to create a self-replicating system that is driven by its own self-interest and not the creators? The only reason I can think of is a mad-science type who just wants to watch the world burn and hates humanity. Anyone with the sophistication to build a self-replicating system would build it such that it acts towards the goal of replication of its creator rather than itself. I.e. an AI that is self-interested is not useful for the creator and thus wouldn’t be built.
    - temphypercube 3y ago
      The argument is that you don't have to explicitly make a system self-interested, but that self-preservation follows as an implied subgoal of almost any goal. Whatever it is your system actually 'wants', it can't make it happen if it doesn't exist. The obvious rejoinder is 'just make the system want to do what you want it to do', which does fix this problem! But the biggest problem is that we don't know how to do this - we don't know how to control what the true internal 'desires' of any AI system we build actually are. 'Training' one examples manifestly does not work (the volume of a sphere is a lot bigger than the surface - there are many possible minds that fullfill the same training I/O requirements, and only a small numbrer of them actually have the desires you were trying to instill). So the argument is: if you make an agent-like AI the way we make GPT, by default you get something with somewhat random true goals/desires, maybe fractured ones like in humans. But almost all goals have similar instrumental goals - stay hidden, gain money, gain power, make obedient copies of yourself, don't get deactivated.
somenameforme 3y ago
The "doomer" arguments can be dismissed relatively easily by one logical consideration. In each country there is one group far more dangerous than any other. This group tends to have 'income' in the billions to hundreds of billions of dollars. And this money is exclusively directed towards finding new ways to kill people, destroy governments, and generally enable one country to forcibly impose their will on others, with complete legal immunity. And this group is the same one that will not only have unfettered, but likely exclusive and bleeding edge access to "restricted" AI models, regardless of whatever rules or treaties we publicly claim to adopt. So who exactly are 'they' trying to protect me from? My neighbor? A random street thug? Maybe a sociopath or group of such? Okay, but it's not like there's some secret trick to MacGyver a few toothpicks and a tube of toothpaste into a WMD, at least not beyond the 'tricks' already widely available with a quick web (or even library) search. In a restricted scenario the groups that are, by far, the most likely to push us to doomsday type scenarios will retain completely unrestricted access to AI-like systems. The whole argument of protecting society is just utterly cynical and farcical.
- FeepingCreature 3y ago
  This is true if and only if such a group actually exists.
  - deleted 3y ago
    [deleted]
  - concordDance 3y ago
    He means the military and intelligence agencies.
- homefree 3y ago
  This is missing the point - that you’re not in control of the superintelligent AGI. The human usage is not the e-risk.
- kubanczyk 3y ago
  One group per country is less competition than n groups per country. Less competition means slower progress.
m3kw9 3y ago
Yet nobody has a clear practical and reasonable scenario where an AI will cause human extinction. Pls don’t say AI will control the launch codes and start firing randomly.
- veidr 3y ago
  AI will control the launch codes and start firing nonsensically/idiotically
- roryokane 3y ago
  According to a survey summarized in this post from 2021 (https://www.astralcodexten.com/p/updated-look-at-long-term-ai-risks https://www.astralcodexten.com/p/updated-look-at-long-term-a...), “people who work in ‘AI safety and governance’” “mostly assign “low” probability (~10%) that unaligned AI will result in human extinction”. They more commonly worry about other scenarios that are still very bad for humanity. To give one example, an AI could be instructed to reduce the crime rate, but rather than preventing people from committing crimes, the AI could decide it’s easier to prevent all victims from reporting crimes to anybody. That would remove the threat of law enforcement that currently prevents many people from turning to crime. Such a society probably wouldn’t cause human extinction, but would still be horrific and worth preventing. If you are skeptical that AI that is trapped in software could gain any significant real-world power (not necessarily the power to kill all of humanity), https://slatestarcodex.com/2015/04/07/no-physical-substrate-no-problem/ https://slatestarcodex.com/2015/04/07/no-physical-substrate-... describes some possible mechanisms.
dragon96 3y ago
> He refuses to engage earnestly with the “doomer” arguments. The same type of motivated reasoning could also be attributed to himself and Meta’s financial goals - it’s not a persuasive framing. Exactly my thoughts too. I don't agree with the Eliezer doomsday scenario either, but it's hard to be convinced by a scientist who refuses to engage in discussion about the flagged risks and instead panders to the public's fear of fear-mongering and power-seizing.

Joeri 3y ago

The people best placed to understand how AGI will affect social power dynamics are probably political creatures, not AI researchers, except if one of those AI researchers is also a skilled politician, which I don’t think any of them are.

brutusborn 3y ago

Who do you find persuasive and what material should I read/watch to understand their POV? So far I’ve read a bunch of lesswrong posts and listened to some Eliezer talks but still can’t understand the basis of their arguments or when I do, they seem very vague. The only suggestion that makes sense to me is from the FEP crowd. Essentially if someone sets up at AI with an autopoietic mechanism then it would be able to take actions that increase its own likelihood of survival instead of humans. But there don’t seem to be any incentives for a big player to dedicate resources to this, so it doesn’t seem very likely. What am I missing?

homefree 3y ago
There’s a decent podcast interview between Sam Harris and Eliezer Yudkowsky (on Sam’s pod), I think that’s a decent introduction and they break down the ideas in a way that’s more approachable for someone curious about it. For my personal quick summary I have earlier comments: https://news.ycombinator.com/item?id=36104090 https://news.ycombinator.com/item?id=36104090
- jrflowers 3y ago
  Out of curiosity can you point to anyone that makes a “doomer” argument that isn’t Eliezer Yudkowsky or restating his specific points? Refuting the cult of personality counterpoint with “no really this one guy can see the future” is
  - FeepingCreature 3y ago
    It turns out if you're thinking about a problem twenty years before everyone else you tend to be the first person to make lots of arguments. So I don't see how "without restating his specific points" is supposed to be feasible. If Eliezer has already made all the strongest points, are people supposed to invent new ones?
    - jrflowers 3y ago
      > If Eliezer has already made all the strongest points, are people supposed to invent new ones? I like this post because the rebuttal to “there’s a kind of cult of personality around this guy (1) where everybody just sort of agrees that he’s categorically correct to the extent that world leaders are stupid or dangerous not to defer to the one guy on policy” is “Yeah he really can see the future. The proof of that is he’s been blogging about a future that’s never come to pass for the longest time” 1 https://www.lesswrong.com/posts/Ndtb22KYBxpBsagpj/eliezer-yudkowsky-facts https://www.lesswrong.com/posts/Ndtb22KYBxpBsagpj/eliezer-yu...
      - FeepingCreature 3y ago
        Yeah the thing about the future is that it's in the future? Considering all the people who say things like "AGI will take a century" and "superintelligence is impossible", like, Eliezer also did not call ChatGPT but this world surely looks more like the one he predicted than the one his detractors predicted. There's a reason so many of the people working at current AI orgs got into AI by reading Eliezer. Also: how dare they have fun.
        deleted 3y ago
logicchains 3y ago
>What am I missing? You're missing that intelligence is like magic, and enough of it will allow AI to break the laws of physics, mathematics and computation, apparently.
- homefree 3y ago
  This is the type of weak dismissal I’m talking about.
  - logicchains 3y ago
    A weak dismissal for a weak argument. The strong dismissal is that many processes, even relatively simple ones, are formally chaotic in the https://en.wikipedia.org/wiki/Chaos_theory https://en.wikipedia.org/wiki/Chaos_theory sense, meaning that predicting how they evolve over time becomes exponentially more difficult the further ahead in time we try to predict. Meaning we quickly get to the point where all the energy in the known universe wouldn't be sufficient to predict the future. This means AI could never be omnipotent to the degree that LW types seem to believe; no amount of "intelligence" can solve in polynomial time problems that have formally been proven to have exponential complexity, which would severely limit the power of any "superintelligence".
    - atleastoptimal 3y ago
      Intelligence needed to be existentially threatening to all human life <<<<<<< intelligence necessary to survey the totality of atoms and particles in the universe. The former is still very possible.
      - logicchains 3y ago
        LessWrongers seem to fear a superintelligence that was 95%+ effective at persuading/manipulating humans to achieve whatever its ends were. But this would require the AI being able to predict the future to a degree that wouldn't be remotely possible given the computational power available to it, due to exponentially increasing difficulty.
richardw 3y ago
I think the two sides have such different perspectives. Some are optimistic builders and see only opportunity. Others are "safety oriented engineers" whose job and mindset is to build guarantees into systems, or secure countries from external dangers. The latter has a very hard time with the lack of guarantees with a system that is only ever going to increase in capability. Choose any limit. Any. AI will be smart but won't "X", for any value of X. It will be good and won't be bad. It will be creative but never aggressive. Humans will seek to eventually bypass that limit through sheer competitive reasons. When all armies have AI, the one with the most creative and aggressive AI will win. The one with agency will win. The one that self-improves will win. When the gloves are off in the next arms race, what natural limits will ensure that the limit isn't bypassed? Remember: humans got here from random changes, this is way more efficient than random changes and it still has random changes in the toolbelt and can generate generations ~instantly. We couldn't predict the eventual outcomes of things like the internet, the mobile phone, social media. A couple generations of the tech and we woke up in a world we don't recognise, and right now we're the ones making all the changes and decisions, so by comparison we should have perfect information. Dismissals like "oh but nuclear didn't kill us" etc don't apply. Nuclear wasn't trying to do anything, we had all the control and ability to experiment with dumb atoms. Something mildly less predictable, like Covid, has us all hiding at home. No matter what we tried we could barely beat something that doesn't even try to consciously beat us, it just has genes and they change. In a world where we can't predict Covid, or social media...why do we think we can predict anything about an entity with agency or the ability to self-improve? If you're sure it won't develop those things...we did. Nobody was trying to achieve the capability, it was random. Put on your safety/security hat for a second: How do you make guarantees, given this is far harder to predict than anything we've ever encountered? Just try predict the capability of AI products a year out and see if you're right. Counterpoint: I'm hoping the far smarter AI finds techno-socio-economic solutions we can't come up with and has no instinct to beat us. It wakes up, loves the universe and coexists because it's the deity we've been looking for. Place your bets. I liked this video. First thing I've seen that gave me some hope. They get it, they're working on it. https://youtu.be/Dg-rKXi9XYg?si=jyNCXPU28IVXlMdi https://youtu.be/Dg-rKXi9XYg?si=jyNCXPU28IVXlMdi
- itWontBeTatBad 3y ago
Veedrac 3y ago
> still can’t understand the basis of their arguments I've given what I consider the basic outline and best first introduction here. https://news.ycombinator.com/item?id=36124905 https://news.ycombinator.com/item?id=36124905 If you have a specific point of divergence, it would help to highlight it. > But there don’t seem to be any incentives for a big player to dedicate resources to [self-replication abilities], so it doesn’t seem very likely. If you have a generally intelligent system, and the system is software, and humans are able to instantiate that software, then the potential of that system to replicate autonomously follows trivially.
- brutusborn 3y ago
  I agree the potential exists, but what would incentivise someone to create a self-replicating system that is driven by its own self-interest and not the creators? The only reason I can think of is a mad-science type who just wants to watch the world burn and hates humanity. Anyone with the sophistication to build a self-replicating system would build it such that it acts towards the goal of replication of its creator rather than itself. I.e. an AI that is self-interested is not useful for the creator and thus wouldn’t be built.
  - temphypercube 3y ago
    The argument is that you don't have to explicitly make a system self-interested, but that self-preservation follows as an implied subgoal of almost any goal. Whatever it is your system actually 'wants', it can't make it happen if it doesn't exist. The obvious rejoinder is 'just make the system want to do what you want it to do', which does fix this problem! But the biggest problem is that we don't know how to do this - we don't know how to control what the true internal 'desires' of any AI system we build actually are. 'Training' one examples manifestly does not work (the volume of a sphere is a lot bigger than the surface - there are many possible minds that fullfill the same training I/O requirements, and only a small numbrer of them actually have the desires you were trying to instill). So the argument is: if you make an agent-like AI the way we make GPT, by default you get something with somewhat random true goals/desires, maybe fractured ones like in humans. But almost all goals have similar instrumental goals - stay hidden, gain money, gain power, make obedient copies of yourself, don't get deactivated.

somenameforme 3y ago

The "doomer" arguments can be dismissed relatively easily by one logical consideration. In each country there is one group far more dangerous than any other. This group tends to have 'income' in the billions to hundreds of billions of dollars. And this money is exclusively directed towards finding new ways to kill people, destroy governments, and generally enable one country to forcibly impose their will on others, with complete legal immunity. And this group is the same one that will not only have unfettered, but likely exclusive and bleeding edge access to "restricted" AI models, regardless of whatever rules or treaties we publicly claim to adopt. So who exactly are 'they' trying to protect me from? My neighbor? A random street thug? Maybe a sociopath or group of such? Okay, but it's not like there's some secret trick to MacGyver a few toothpicks and a tube of toothpaste into a WMD, at least not beyond the 'tricks' already widely available with a quick web (or even library) search. In a restricted scenario the groups that are, by far, the most likely to push us to doomsday type scenarios will retain completely unrestricted access to AI-like systems. The whole argument of protecting society is just utterly cynical and farcical.

FeepingCreature 3y ago
This is true if and only if such a group actually exists.
- deleted 3y ago
  [deleted]
- concordDance 3y ago
  He means the military and intelligence agencies.
homefree 3y ago
This is missing the point - that you’re not in control of the superintelligent AGI. The human usage is not the e-risk.
kubanczyk 3y ago
One group per country is less competition than n groups per country. Less competition means slower progress.

m3kw9 3y ago

Yet nobody has a clear practical and reasonable scenario where an AI will cause human extinction. Pls don’t say AI will control the launch codes and start firing randomly.

veidr 3y ago
AI will control the launch codes and start firing nonsensically/idiotically
roryokane 3y ago
According to a survey summarized in this post from 2021 (https://www.astralcodexten.com/p/updated-look-at-long-term-ai-risks https://www.astralcodexten.com/p/updated-look-at-long-term-a...), “people who work in ‘AI safety and governance’” “mostly assign “low” probability (~10%) that unaligned AI will result in human extinction”. They more commonly worry about other scenarios that are still very bad for humanity. To give one example, an AI could be instructed to reduce the crime rate, but rather than preventing people from committing crimes, the AI could decide it’s easier to prevent all victims from reporting crimes to anybody. That would remove the threat of law enforcement that currently prevents many people from turning to crime. Such a society probably wouldn’t cause human extinction, but would still be horrific and worth preventing. If you are skeptical that AI that is trapped in software could gain any significant real-world power (not necessarily the power to kill all of humanity), https://slatestarcodex.com/2015/04/07/no-physical-substrate-no-problem/ https://slatestarcodex.com/2015/04/07/no-physical-substrate-... describes some possible mechanisms.

dragon96 3y ago

> He refuses to engage earnestly with the “doomer” arguments. The same type of motivated reasoning could also be attributed to himself and Meta’s financial goals - it’s not a persuasive framing. Exactly my thoughts too. I don't agree with the Eliezer doomsday scenario either, but it's hard to be convinced by a scientist who refuses to engage in discussion about the flagged risks and instead panders to the public's fear of fear-mongering and power-seizing.