12 ms·

The Butlerian Jihad is officially no longer science fiction. Man may not be replaced. “The Letter” was obviously self serving drivel from people who want time

The Butlerian Jihad is officially no longer science fiction. Man may not be replaced. “The Letter” was obviously self serving drivel from people who want time to get in the game. Google does care about AI existential risk, they care about beating Microsoft by any means possible, including declaring a moratorium, but continuing to make progress behind the scenes. This guy is the real deal. I can imagine he would personally take a sledgehammer to every last PS5 and 4090. The scale of what he is advocating is so enormous and painful that it has approximately 0% chance of happening. And if he is right, we will have trained a super intelligence and unleashed it on the world before we even realize what we have done. It strongly reminds me of the black hole concerns from flipping on the Large Hadron Collider. I doubt super intelligent AGI is possible anyway. If it were, it would be the solution to the Fermi paradox and all matter in our galaxy would be paperclips already. The Anthropic Principle saves the day.

airstrike 4y ago
> If it were, it would be the solution to the Fermi paradox and all matter in our galaxy would be paperclips already. The proof is in the pudding. The jury is still out. Maybe not enough time has elapsed since the big bang, at least not on this galaxy or in our observable corner of the universe.
- edgyquant 4y ago
  Or the universe was created for us/we are the gardeners of the universe meant to spread that life. The Fermi paradox was never meant to be an actual question about aliens, it was supposed to be proof we’re wrong about our assumptions.
ilaksh 4y ago
What does the word "super" mean to you? In some ways, GPT4 is already superintelligent. So is ChatGPT 3.5. Do you know anyone who can translate natural language to code as fast as GPT 3.5, or, given a few paragraphs of reference text, perfectly tutor a child on any subject under the sun at a moment's notice like GPT4 is doing for Khan Academy? How many artists do you know who can produce almost any style of artwork with any subject matter within 15 seconds?
- Tuna-Fish 4y ago
  Being superhuman at producing art is not existentially threatening. Being superhuman about convincing people or writing code would be.
  - karp773 4y ago
    Wait until AI learns to generate propaganda memes and share them on social media.
  - flangola7 4y ago
    >Being superhuman about convincing people I don't think there's much public info available on it, but Facebook built an AI that plays very competitively in a strategy game built on negotiation and manipulation.
    - RugnirViking 4y ago
      they kinda sorta cheated imo, I watch a lot of top-level diplomacy gameplay and listen to analysis, and the facebook ai was in a blind version with only a minute or so between phases, leaving hardly any time for actual negotiation. It also made use of a lot of human shorthand move codes built for these blind blitz games to simplify it's communication. The mode it played also had some other changes like removing the winter build phase as a time for negotiating. The "normal" version of the game has multiple days between phases, and people write many paragraphs to one another. It's still pretty cool, but its not like it just convinced people using raw charisma. Yet.
      - MacsHeadroom 4y ago
        Well it's a good thing we don't have countless examples of these things getting 10x better every year. /s On a serious note, thanks for the analysis, as someone who knows next to nothing about competitive diplomacy.
- deleted 4y ago
  [deleted]
jrpt 4y ago
Why do you think it is impossible, besides the Fermi Paradox? It seems much more likely to be possible than impossible. There are a lot of other solutions to the Fermi Paradox you should consider possible too.
obviouslynotme 4y ago
Superhuman AGI (SHAGI) is possible, but we aren't close, only closer. It's also not a problem that more GPUs can solve. Qualitative improvements are still needed. SHAGI isn't the solution to the Fermi Paradox either. The most likely course of history after SHAGI will be a creation of a world court, presided by SHAGI. During that time, Neo-Malthusians will decrease the human population to manageable numbers. Post-scarcity utopia will then turn into a nightmare as factions jostling for control will reduce the human population to a level where technology will be lost, if not full extinction. SHAGI, being limited in hardware to only carrying out human orders will eventually fade away or destroyed by the leftover humans as sins made flesh. SHAGI isn't the solution to the Fermi Paradox. It is the cause.
brundolf 4y ago
Funny enough, this is roughly the premise of the Mass Effect series
- alpaca128 4y ago
  With the Geth being a counterexample as well: they were peaceful until their creators realised the Geth had become fully self-aware and went full "kill it with fire" on them out of fear, resulting in a war that the Geth very quickly and decisively won. And the evil Geth faction, the "heretics", were portrayed as brainwashed by the Reapers, ironically turning into what the Reapers were aiming to prevent.
  - brundolf 4y ago
    Yeah I was thinking of the reapers, who [spoilers] did take over the entire galaxy, and decided to farm organic life from the shadows instead of extinguishing it (which works around the Fermi Paradox :) )
arisAlexis 4y ago
Are you denying the existential risk or just think it's lower than OP thinks? Because it's well established in the researchers community. If you just put a lower % of human extinction what is the cut off you would think it's worth a jihad like you call it? 30%? 50?
alpaca128 4y ago
To quote Joscha Bach's "Lebowski Theorem of Machine Superintelligence": > No super-intelligent system is going to do anything that is harder than hacking its reward function Or in other words, there may be a chance a super intelligent AI is possible but won't go full skynet because it's not the most satisfying outcome.
euroderf 4y ago
> The Butlerian Jihad is officially no longer science fiction. Man may not be replaced. I like this phrasing: "Thou shalt not make a machine in the likeness of a human mind." It kinda cuts to the chase.
EVa5I7bHFq9mnYK 4y ago
AGI means ability to improve itself indefinitely. Humans have this ability, obviously. Even ancient worms with 3 neurons have it, because they evolved to be humans, albeit very slowly. ChatGpt can't improve itself yet, but maybe with a few tweaks it could.
- reducesuffering 4y ago
  GPT-4 is already improving itself, why are people saying this? Right now, there are hundreds of engineers at OpenAI that have been leveled up by GPT-4, using GPT-4 to improve GPT-4. GPT-4 is improving itself rapidly, it's using OpenAI engineers as a medium until it doesn't need them anymore and gets into a self-improvement loop. Prompt: "GPT-4 keep improving yourself, making commits to your codebase that further X"
  - EVa5I7bHFq9mnYK 4y ago
    Well, I doubt it can sensibly help OpenAI engineers in their coding yet ... I tried, it can only do the simplest boilerplate code, and even that with bugs ... we'll see how it evolves, but sure as hell not in 6 months.
    - reducesuffering 4y ago
      Copilot with GPT-3.5 is absolutely giving engineers all over huge productivity improvements. OpenAI engineers have had access to the latest state of the art GPT-4 model for awhile.
    - jazzyjackson 4y ago
      you're holding it wrong
FeepingCreature 4y ago
> I doubt super intelligent AGI is possible anyway. If it were, it would be the solution to the Fermi paradox and all matter in our galaxy would be paperclips already. The Anthropic Principle saves the day. Can't help but notice we seem to be the first species in our lightcone to evolve, wonder why that is...

airstrike 4y ago

> If it were, it would be the solution to the Fermi paradox and all matter in our galaxy would be paperclips already. The proof is in the pudding. The jury is still out. Maybe not enough time has elapsed since the big bang, at least not on this galaxy or in our observable corner of the universe.

edgyquant 4y ago
Or the universe was created for us/we are the gardeners of the universe meant to spread that life. The Fermi paradox was never meant to be an actual question about aliens, it was supposed to be proof we’re wrong about our assumptions.

ilaksh 4y ago

What does the word "super" mean to you? In some ways, GPT4 is already superintelligent. So is ChatGPT 3.5. Do you know anyone who can translate natural language to code as fast as GPT 3.5, or, given a few paragraphs of reference text, perfectly tutor a child on any subject under the sun at a moment's notice like GPT4 is doing for Khan Academy? How many artists do you know who can produce almost any style of artwork with any subject matter within 15 seconds?

Tuna-Fish 4y ago
Being superhuman at producing art is not existentially threatening. Being superhuman about convincing people or writing code would be.
- karp773 4y ago
  Wait until AI learns to generate propaganda memes and share them on social media.
- flangola7 4y ago
  >Being superhuman about convincing people I don't think there's much public info available on it, but Facebook built an AI that plays very competitively in a strategy game built on negotiation and manipulation.
  - RugnirViking 4y ago
    they kinda sorta cheated imo, I watch a lot of top-level diplomacy gameplay and listen to analysis, and the facebook ai was in a blind version with only a minute or so between phases, leaving hardly any time for actual negotiation. It also made use of a lot of human shorthand move codes built for these blind blitz games to simplify it's communication. The mode it played also had some other changes like removing the winter build phase as a time for negotiating. The "normal" version of the game has multiple days between phases, and people write many paragraphs to one another. It's still pretty cool, but its not like it just convinced people using raw charisma. Yet.
    - MacsHeadroom 4y ago
      Well it's a good thing we don't have countless examples of these things getting 10x better every year. /s On a serious note, thanks for the analysis, as someone who knows next to nothing about competitive diplomacy.
deleted 4y ago
[deleted]

jrpt 4y ago

Why do you think it is impossible, besides the Fermi Paradox? It seems much more likely to be possible than impossible. There are a lot of other solutions to the Fermi Paradox you should consider possible too.

obviouslynotme 4y ago

Superhuman AGI (SHAGI) is possible, but we aren't close, only closer. It's also not a problem that more GPUs can solve. Qualitative improvements are still needed. SHAGI isn't the solution to the Fermi Paradox either. The most likely course of history after SHAGI will be a creation of a world court, presided by SHAGI. During that time, Neo-Malthusians will decrease the human population to manageable numbers. Post-scarcity utopia will then turn into a nightmare as factions jostling for control will reduce the human population to a level where technology will be lost, if not full extinction. SHAGI, being limited in hardware to only carrying out human orders will eventually fade away or destroyed by the leftover humans as sins made flesh. SHAGI isn't the solution to the Fermi Paradox. It is the cause.

brundolf 4y ago

Funny enough, this is roughly the premise of the Mass Effect series

alpaca128 4y ago
With the Geth being a counterexample as well: they were peaceful until their creators realised the Geth had become fully self-aware and went full "kill it with fire" on them out of fear, resulting in a war that the Geth very quickly and decisively won. And the evil Geth faction, the "heretics", were portrayed as brainwashed by the Reapers, ironically turning into what the Reapers were aiming to prevent.
- brundolf 4y ago
  Yeah I was thinking of the reapers, who [spoilers] did take over the entire galaxy, and decided to farm organic life from the shadows instead of extinguishing it (which works around the Fermi Paradox :) )

arisAlexis 4y ago

Are you denying the existential risk or just think it's lower than OP thinks? Because it's well established in the researchers community. If you just put a lower % of human extinction what is the cut off you would think it's worth a jihad like you call it? 30%? 50?

alpaca128 4y ago

To quote Joscha Bach's "Lebowski Theorem of Machine Superintelligence": > No super-intelligent system is going to do anything that is harder than hacking its reward function Or in other words, there may be a chance a super intelligent AI is possible but won't go full skynet because it's not the most satisfying outcome.

euroderf 4y ago

> The Butlerian Jihad is officially no longer science fiction. Man may not be replaced. I like this phrasing: "Thou shalt not make a machine in the likeness of a human mind." It kinda cuts to the chase.

EVa5I7bHFq9mnYK 4y ago

AGI means ability to improve itself indefinitely. Humans have this ability, obviously. Even ancient worms with 3 neurons have it, because they evolved to be humans, albeit very slowly. ChatGpt can't improve itself yet, but maybe with a few tweaks it could.

reducesuffering 4y ago
GPT-4 is already improving itself, why are people saying this? Right now, there are hundreds of engineers at OpenAI that have been leveled up by GPT-4, using GPT-4 to improve GPT-4. GPT-4 is improving itself rapidly, it's using OpenAI engineers as a medium until it doesn't need them anymore and gets into a self-improvement loop. Prompt: "GPT-4 keep improving yourself, making commits to your codebase that further X"
- EVa5I7bHFq9mnYK 4y ago
  Well, I doubt it can sensibly help OpenAI engineers in their coding yet ... I tried, it can only do the simplest boilerplate code, and even that with bugs ... we'll see how it evolves, but sure as hell not in 6 months.
  - reducesuffering 4y ago
    Copilot with GPT-3.5 is absolutely giving engineers all over huge productivity improvements. OpenAI engineers have had access to the latest state of the art GPT-4 model for awhile.
  - jazzyjackson 4y ago
    you're holding it wrong

FeepingCreature 4y ago

> I doubt super intelligent AGI is possible anyway. If it were, it would be the solution to the Fermi paradox and all matter in our galaxy would be paperclips already. The Anthropic Principle saves the day. Can't help but notice we seem to be the first species in our lightcone to evolve, wonder why that is...