4 ms·

While the persons mentionned in the articles are indubitably most of the most well-informed people in the world. They are also the most likely to have internali

While the persons mentionned in the articles are indubitably most of the most well-informed people in the world. They are also the most likely to have internalized that their work is leading to superhuman intelligence/AGI. But is it really realistic ? So they have a strong bias towards imagining the most catastrophic scenario.

Sharlin 26d ago
There have been extremely smart people worried about this exact scenario for 20 years or more. Nothing about this is new. In any case, what's the probability at which a possible extinction event becomes a risk worth taking? Even if there's just a 1% likelihood of current research bringing about a superintelligent AGI, and just a 1% likelihood of that AGI causing an existential catastrophe, no rational person should accept the risk, unless it was clear that not accepting it would yield an even worse outcome.
- chasd00 26d ago
  superintelligent AGI is what everyone invested in and why these companies have insane valuations. If they all slowed and said "well wait, we're going to hold on AGI" it would be the biggest rug pull in financial history.
  - Sharlin 26d ago
    Yeah, let's risk extinction in order to avoid some made-up numbers going down! Not an insane take at all.
- BobaFloutist 26d ago
  > Even if there's just a 1% likelihood This is like me saying "Even if there's just a 1% likelihood of me getting struck by lightning..." You're implying that 1% likelihood is the floor, because 1 is the lowest natural number and feels like a good default "low percentage", but there's zero justification for putting the floor that high. Conjuring an unsupported "low" probability and multiplying it by a massive outcome to make it seem significant is one of the most irritating ways people launder their opinions(/gut feelings) through "math".
  - Sharlin 26d ago
    For the record, I consider that P(dangerously unaligned | AGI from current research) >> 0.01. A more reasonable estimate given, what we know, would be 0.99. With regard to the second probability, P(AGI | current LLM research), it would certainly be overconfidence bordering on arrogance to assign it a value significantly lower than 0.01.
    - BobaFloutist 26d ago
      Once again, you're writing up nicely formal formulae with zero justification. It's fine to eyeball it and give your gut feeling, but it's disingenuous to present that as probabilistic fact. Edit: Also, you've downgraded your claim from "Extinction event" to "dangerously unaligned"
PowerElectronix 26d ago
People working for those companies have either drank the koolaid or have equity enough to play along until they can cash out. I imagine saying you're developing skynet is better than saying you are developing a very cool, extremely expensive to run chatbot that can do math and code.

Sharlin 26d ago

There have been extremely smart people worried about this exact scenario for 20 years or more. Nothing about this is new. In any case, what's the probability at which a possible extinction event becomes a risk worth taking? Even if there's just a 1% likelihood of current research bringing about a superintelligent AGI, and just a 1% likelihood of that AGI causing an existential catastrophe, no rational person should accept the risk, unless it was clear that not accepting it would yield an even worse outcome.

chasd00 26d ago
superintelligent AGI is what everyone invested in and why these companies have insane valuations. If they all slowed and said "well wait, we're going to hold on AGI" it would be the biggest rug pull in financial history.
- Sharlin 26d ago
  Yeah, let's risk extinction in order to avoid some made-up numbers going down! Not an insane take at all.
BobaFloutist 26d ago
> Even if there's just a 1% likelihood This is like me saying "Even if there's just a 1% likelihood of me getting struck by lightning..." You're implying that 1% likelihood is the floor, because 1 is the lowest natural number and feels like a good default "low percentage", but there's zero justification for putting the floor that high. Conjuring an unsupported "low" probability and multiplying it by a massive outcome to make it seem significant is one of the most irritating ways people launder their opinions(/gut feelings) through "math".
- Sharlin 26d ago
  For the record, I consider that P(dangerously unaligned | AGI from current research) >> 0.01. A more reasonable estimate given, what we know, would be 0.99. With regard to the second probability, P(AGI | current LLM research), it would certainly be overconfidence bordering on arrogance to assign it a value significantly lower than 0.01.
  - BobaFloutist 26d ago
    Once again, you're writing up nicely formal formulae with zero justification. It's fine to eyeball it and give your gut feeling, but it's disingenuous to present that as probabilistic fact. Edit: Also, you've downgraded your claim from "Extinction event" to "dangerously unaligned"

PowerElectronix 26d ago

People working for those companies have either drank the koolaid or have equity enough to play along until they can cash out. I imagine saying you're developing skynet is better than saying you are developing a very cool, extremely expensive to run chatbot that can do math and code.