5 ms·

Yeah, seeing how bad the rebuttals are to superintelligence e-risk really does make me feel like we’re doomed. They’re frustratingly dumb, I’d like to see some

Yeah, seeing how bad the rebuttals are to superintelligence e-risk really does make me feel like we’re doomed. They’re frustratingly dumb, I’d like to see some good arguments or steelman but they’re honestly very hard to find. Ultimately it seems like there’s nothing most individuals can do so just live your life as you would and hope the timeline is farther out than it seems. Worse than the Cold War nuclear risk imo, at least in that case it was possible for humans to stupidly build thousands and then decide not to use them (and it’s relatively easy to restrict uranium access/control the development of nukes). Not really the case with superintelligent AGI. People have a bad heuristic for what tailrisk exists. They think extinction talk is impossible or crazy. We won’t get the opportunity to mess up and try again in this case. https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/ https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...

AbrahamParangi 3y ago
They’re hard to find because you’re comparing arguments for god to arguments against god. They’re both more or less evidence free so the better argument is the one that is more intellectually satisfying, which the positive case always is.
- gonehome 3y ago
  I find the religious arguments for god’s existence similarly bad to the denials of unaligned ASI e-risk, but in the god case at least it doesn’t matter as much since bad epistemology there is a lot less likely to lead to human extinction. It’s a logic argument. 1. Super Intelligent AGI is possible. 2. Unaligned super intelligent AGI is an e-risk likely to wipe out humanity. 3. We have no idea how to align AGI. There seems to be more consensus around 1 now than there was even 3 years ago. I think most also agree with 3. 2 is where people are incredulous, but the incredulity is backed by (imo) bad reasoning. People think of a smart person they know as an example of something “smarter” and use that as a justification of why it’s not an issue. We’re constrained in all sorts of ways (head size, energy) and the distance of intelligence from a dumb human to Einstein is very tiny on the overall spectrum.
  - AbrahamParangi 3y ago
    Arguably in the theology case it matters even more! If god exists your misaligned omnipotent AI already exists and has promised you infinite torment for not believing! It’s just totally evidence free reasoning from axioms that are chosen by vibes alone. Why evil AI god and not the Christian god? Why not Huitzilopochtli, who demands sacrifice? The answer is that this is the wrong question. No argumentation can be usefully made either for or against.
    - gonehome 3y ago
      It matters less because in the theology case it’s a lot easier to dismiss 1. - the divine religious arguments for a supernatural god are super weak so the details don’t matter. It’s much more likely humans just making up myths. With AI we’re seeing the capabilities improve rapidly and the arguments about why AGI is impossible or will be constrained for some reason are the weak ones.
      - usrbinbash 3y ago
        > we’re seeing the capabilities improve rapidly Yes, but towards what? How do we know that, say, Transformer based LLMs are closer to superintelligence than earlier architectures? To make such an assumption, there would need to be something that we could measure to track the process. To the best of my knowledge, there is no accurate definition of intelligence, nor superintelligence. So how would we know where on the scale of [intelligent-------superintelligent] a given system is, or whether it even is on that scale?
  - YeGoblynQueenne 3y ago
    >> 1. Super Intelligent AGI is possible. When?
    - gonehome 3y ago
      The when is harder to know. If it’s possible then we need to figure out alignment first (which currently doesn’t look promising). People are famously bad at predicting when right up until they have already done it. “In 1901, two years before helping build the first heavier-than-air flyer, Wilbur Wright told his brother that powered flight was fifty years away. “In 1939, three years before he personally oversaw the first critical chain reaction in a pile of uranium bricks, Enrico Fermi voiced 90% confidence that it was impossible to use uranium to sustain a fission chain reaction. I believe Fermi also said a year after that, aka two years before the denouement, that if net power from fission was even possible (as he then granted some greater plausibility) then it would be fifty years off; but for this I neglected to keep the citation. “And of course if you’re not the Wright Brothers or Enrico Fermi, you will be even more surprised. Most of the world learned that atomic weapons were now a thing when they woke up to the headlines about Hiroshima. There were esteemed intellectuals saying four years after the Wright Flyer that heavier-than-air flight was impossible, because knowledge propagated more slowly back then.” https://intelligence.org/2017/10/13/fire-alarm/ https://intelligence.org/2017/10/13/fire-alarm/
- NumberWangMan 3y ago
  Really? I'm pretty worried about AGI that's not god, just a bit smarter than us, and because we're lazy and the incentives are to give power to the AI, we just start putting it in control of everything. It gets smarter, captures regulators just like oil companies did, and we end up losing control of things. Even though if we could coordinate, we might decide to want to stop things, coordination is really hard.
mattmanser 3y ago
Personally I find people with the superintelligence fetish frustratingly dumb too. Far more likely is that an AI will get depressed, bored, get obsessed with some niche maths or navel gaze itself into nothingness. And that's before we even think about whether it would even have the motivation. I think there's a reason humans plateaued where they did, very intelligent people often really struggle in the real world.
- jabradoodle 3y ago
  What is your heuristic to determine what is likely? The plateau of human intelligence isn't especially relevant given we are the product of evolution rather than engineering. Personally,I think we stand a pretty decent chance of making a horrible mess of this all well before reaching AGI. However, a machine that could improve itself would not be limited in anyway that humans are; and I think we know little of how it would behave. That is assuming it can't be controlled; otherwise it will behave as directed by whoever controls it.
- mitthrowaway2 3y ago
  > Far more likely is that an AI will get depressed, bored, get obsessed with some niche maths or navel gaze itself into nothingness. When the "people with the superintelligence fetish" worry about AI, at least they provide a reason why it's likely that AI will make humans extinct.

AbrahamParangi 3y ago

They’re hard to find because you’re comparing arguments for god to arguments against god. They’re both more or less evidence free so the better argument is the one that is more intellectually satisfying, which the positive case always is.

gonehome 3y ago
I find the religious arguments for god’s existence similarly bad to the denials of unaligned ASI e-risk, but in the god case at least it doesn’t matter as much since bad epistemology there is a lot less likely to lead to human extinction. It’s a logic argument. 1. Super Intelligent AGI is possible. 2. Unaligned super intelligent AGI is an e-risk likely to wipe out humanity. 3. We have no idea how to align AGI. There seems to be more consensus around 1 now than there was even 3 years ago. I think most also agree with 3. 2 is where people are incredulous, but the incredulity is backed by (imo) bad reasoning. People think of a smart person they know as an example of something “smarter” and use that as a justification of why it’s not an issue. We’re constrained in all sorts of ways (head size, energy) and the distance of intelligence from a dumb human to Einstein is very tiny on the overall spectrum.
- AbrahamParangi 3y ago
  Arguably in the theology case it matters even more! If god exists your misaligned omnipotent AI already exists and has promised you infinite torment for not believing! It’s just totally evidence free reasoning from axioms that are chosen by vibes alone. Why evil AI god and not the Christian god? Why not Huitzilopochtli, who demands sacrifice? The answer is that this is the wrong question. No argumentation can be usefully made either for or against.
  - gonehome 3y ago
    It matters less because in the theology case it’s a lot easier to dismiss 1. - the divine religious arguments for a supernatural god are super weak so the details don’t matter. It’s much more likely humans just making up myths. With AI we’re seeing the capabilities improve rapidly and the arguments about why AGI is impossible or will be constrained for some reason are the weak ones.
    - usrbinbash 3y ago
      > we’re seeing the capabilities improve rapidly Yes, but towards what? How do we know that, say, Transformer based LLMs are closer to superintelligence than earlier architectures? To make such an assumption, there would need to be something that we could measure to track the process. To the best of my knowledge, there is no accurate definition of intelligence, nor superintelligence. So how would we know where on the scale of [intelligent-------superintelligent] a given system is, or whether it even is on that scale?
- YeGoblynQueenne 3y ago
  >> 1. Super Intelligent AGI is possible. When?
  - gonehome 3y ago
    The when is harder to know. If it’s possible then we need to figure out alignment first (which currently doesn’t look promising). People are famously bad at predicting when right up until they have already done it. “In 1901, two years before helping build the first heavier-than-air flyer, Wilbur Wright told his brother that powered flight was fifty years away. “In 1939, three years before he personally oversaw the first critical chain reaction in a pile of uranium bricks, Enrico Fermi voiced 90% confidence that it was impossible to use uranium to sustain a fission chain reaction. I believe Fermi also said a year after that, aka two years before the denouement, that if net power from fission was even possible (as he then granted some greater plausibility) then it would be fifty years off; but for this I neglected to keep the citation. “And of course if you’re not the Wright Brothers or Enrico Fermi, you will be even more surprised. Most of the world learned that atomic weapons were now a thing when they woke up to the headlines about Hiroshima. There were esteemed intellectuals saying four years after the Wright Flyer that heavier-than-air flight was impossible, because knowledge propagated more slowly back then.” https://intelligence.org/2017/10/13/fire-alarm/ https://intelligence.org/2017/10/13/fire-alarm/
NumberWangMan 3y ago
Really? I'm pretty worried about AGI that's not god, just a bit smarter than us, and because we're lazy and the incentives are to give power to the AI, we just start putting it in control of everything. It gets smarter, captures regulators just like oil companies did, and we end up losing control of things. Even though if we could coordinate, we might decide to want to stop things, coordination is really hard.

mattmanser 3y ago

Personally I find people with the superintelligence fetish frustratingly dumb too. Far more likely is that an AI will get depressed, bored, get obsessed with some niche maths or navel gaze itself into nothingness. And that's before we even think about whether it would even have the motivation. I think there's a reason humans plateaued where they did, very intelligent people often really struggle in the real world.

jabradoodle 3y ago
What is your heuristic to determine what is likely? The plateau of human intelligence isn't especially relevant given we are the product of evolution rather than engineering. Personally,I think we stand a pretty decent chance of making a horrible mess of this all well before reaching AGI. However, a machine that could improve itself would not be limited in anyway that humans are; and I think we know little of how it would behave. That is assuming it can't be controlled; otherwise it will behave as directed by whoever controls it.
mitthrowaway2 3y ago
> Far more likely is that an AI will get depressed, bored, get obsessed with some niche maths or navel gaze itself into nothingness. When the "people with the superintelligence fetish" worry about AI, at least they provide a reason why it's likely that AI will make humans extinct.