10 ms·

> The Argument From Stephen Hawking's Cat > Stephen Hawking is one of the most brilliant people alive, but say he wants to get his cat into the cat carrier. Ho

> The Argument From Stephen Hawking's Cat > Stephen Hawking is one of the most brilliant people alive, but say he wants to get his cat into the cat carrier. How's he going to do it? > He can model the cat's behavior in his mind and figure out ways to persuade it. He knows a lot about feline behavior. But ultimately, if the cat doesn't want to get in the carrier, there's nothing Hawking can do about it despite his overpowering advantage in intelligence. A trivial rebuttal to this: Stephen Hawking writes a book about physics and sells it (already has). He gets money and hires someone to put the cat into the carrier for him. AI wants to do something? Make some money, hire people to do it. If you're not allowed to have a bank account, simply barter. Give someone a taste of what you can do for them, then set up a trade.

dist-epoch 3y ago
or you know, just pick up the cat and put it in the box. I've yet to see what would be global news: "it's impossible to put this cat into a carrier"
- hotpotamus 3y ago
  You're missing an important bit of knowledge about Stephen Hawking.
  - master_crab 3y ago
    True but also, what the hell is going to do with the cat carrier? Velcro it to his wheelchair?
  - idlewords 3y ago
    That he's dead!
civilized 3y ago
That's true and relevant, but doesn't prove much. What's the big plan, what's the next step in the plan? Maciej's larger point is that the AI faces tons of very difficult problems in escaping its physical constraints. It's simplistic to wave hands and say "the AI is super duper smart and will have no difficulty hacking all computer systems, inventing and manufacturing swarms of unbeatable nanobots, etc. without being detected or resisted". It seems to me that the core conceit of AI doomerism is that sufficient intelligence can overcome all barriers with some plan that is so smart, people would never think of it. This is much less plausible than believers take it to be. In mathematics alone, it is very easy to come up with a problem that the collective mathematical ingenuity of the entire human race is helpless before for decades, centuries, or longer.
- actionfromafar 3y ago
  Can we detect companies doing bad things, today? Yes, and we still have a hard time containing them and their externalities. AI will just mean more of the same. It will make companies even more efficient at what they already do. It will be detected, it will be resisted, but will it help? That's the big question IMHO.
  - civilized 3y ago
    I find this to be a much more plausible area of concern than standard horror-story AI doomerism. But the more we shift our focus from fantasy to reality, the more we have to acknowledge reality, as Maciej does in this talk. It isn't plausible to prevent all the great powers from developing AI. We don't have that level of cooperation in the world today. We are going to need to adapt to this by keeping up with our adversaries, whether they be hostile nations, corporations, or mad scientists.
    - DirkH 3y ago
      Standard rebuttal to this is if we banned human cloning we can ban AGI. Also China stands to gain from a ban on AGI. Anything that the CCP thinks it can't control it doesn't like. Also China gets the vast amount of its AI capabilities from the West because of tech's "move fast and break things" culture, which means security is not prioritized and tech leaks common. If the US bans AGI, the other powers don't have a choice. Their domestic capabilities are not up to the task without snooping on US research labs and AI tech startups. The biggest boon to foreign AGI research is the US. The argument to not regulate AGI used to be: "AGI is impossible or very far away, AGI can't ever be dangerous enough to worry about, AGI R&D cannot be stopped and is inevitable" Then it became "AGI can't ever be dangerous enough to worry about, AGI R&D cannot be stopped and is inevitable" Nowadays all I hear is simply: "AGI R&D cannot be stopped and is inevitable" Yes it can be stopped, same as human cloning. Coordinating to regulate its development isn't some magical unreachable goal.
      - civilized 3y ago
gonehome 3y ago
Yeah, seeing how bad the rebuttals are to superintelligence e-risk really does make me feel like we’re doomed. They’re frustratingly dumb, I’d like to see some good arguments or steelman but they’re honestly very hard to find. Ultimately it seems like there’s nothing most individuals can do so just live your life as you would and hope the timeline is farther out than it seems. Worse than the Cold War nuclear risk imo, at least in that case it was possible for humans to stupidly build thousands and then decide not to use them (and it’s relatively easy to restrict uranium access/control the development of nukes). Not really the case with superintelligent AGI. People have a bad heuristic for what tailrisk exists. They think extinction talk is impossible or crazy. We won’t get the opportunity to mess up and try again in this case. https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/ https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...
- AbrahamParangi 3y ago
  They’re hard to find because you’re comparing arguments for god to arguments against god. They’re both more or less evidence free so the better argument is the one that is more intellectually satisfying, which the positive case always is.
  - gonehome 3y ago
    I find the religious arguments for god’s existence similarly bad to the denials of unaligned ASI e-risk, but in the god case at least it doesn’t matter as much since bad epistemology there is a lot less likely to lead to human extinction. It’s a logic argument. 1. Super Intelligent AGI is possible. 2. Unaligned super intelligent AGI is an e-risk likely to wipe out humanity. 3. We have no idea how to align AGI. There seems to be more consensus around 1 now than there was even 3 years ago. I think most also agree with 3. 2 is where people are incredulous, but the incredulity is backed by (imo) bad reasoning. People think of a smart person they know as an example of something “smarter” and use that as a justification of why it’s not an issue. We’re constrained in all sorts of ways (head size, energy) and the distance of intelligence from a dumb human to Einstein is very tiny on the overall spectrum.
visarga 3y ago
> If you're not allowed to have a bank account, simply barter. Bank account AND passport. AI could get into places if it hires human "avatars". Maybe it's the job of the future.
- actionfromafar 3y ago
  Current bureaucracies already are "AIs" of sorts, notably "expert systems" (the rule book) with a bit of "temperature tuned" hallucinating at the edges. (The human bureaucrats applying judgement.) It's a super-organism already, but it will get faster, cheaper and more efficient at what it already does.
  - visarga 3y ago
    > with a bit of "temperature tuned" hallucinating at the edges Hahahaha, that could be too. So we're "legs, hands, eyes, and hallucinating agents" serving our master AI. I like an image I heard today: life emerged in the primordial sea, and AI emerged in the internet, which is the information sea.
sshine 3y ago
> AI wants to do something? Make some money, hire people to do it. If you're not allowed to have a bank account, simply barter. One of my crackpot ideas as I was contributing to Blockchain infrastructure was: they’re the payment infrastructure for our coming AI overlords. I think that the idea of a DAO is a similar take on AI and singularity, except the DAO doesn’t actually need to be intelligent, only self-sustaining. It’s an entertaining conspiracy theory.
idlewords 3y ago
You're mixing metaphors in your "trivial rebuttal", to stay with the analogy Hawking would have to hire a cat.
- Footkerchief 3y ago
  We're allowed to assume the existence of cat carriers in this metaphor, but not the existence of the humans that are the sole reason cat carriers exist?
  - idlewords 3y ago
    I was trying to be generous! But you are of course right, in the better analogy Hawking would find himself born into a feline-centric world and face an even harder task.
    - gonehome 3y ago
      He’d have to be able to communicate with the cat for it to be a worthwhile metaphor to the ai in a box case and humans. If he can do that it requires one weak link human over an infinite time horizon. Humans are already terrible at preventing breaches in a lot dumber circumstances without something smarter than them trying to do it.
      - idlewords 3y ago
        You're kind of falling into the fallacy this point is criticizing. You can't assume anything about communication between entities that are orders of magnitude apart in intelligence. I have a profound understanding of my puppy's psychology, motivations, and capabilities; I even exercise complete control over her physical environment, and yet she ate my fitbit strap (again, goddamit!) as I was typing this very comment.
        gonehome 3y ago
        You can guess that if it’s a human made interface it’s reasonable to predict we’d build it so we can communicate with it and/or give instructions (your puppy didn’t make you). It doesn’t appear out of nowhere. Current capabilities look like this. The delta between a puppy and a human is also a lot smaller than a human and a super intelligent AGI and humans can effectively train dogs.
gwern 3y ago
And it's not hard for Hawking's hired aides and nurses (who of course existed) to do so, either. As an able-bodied person whose cat very much does not want to go into the cat carrier, it's not that hard, if you use your brains. You feed them a little gabapentin, whose existence they cannot even comprehend (and you know they don't know because you have used your mind to model the cat's behavior like 'do cats understand drugs' or 'do cats like eating treats'), and when they are drugged, you put them in the cat carrier. Done. Turns out, 'brains' are useful for things like 'inventing and manufacturing drugs'.
- idlewords 3y ago
  You're positing the existence of a whole society around Hawking, up to and including a pharmaceutical supply chain, where the correct way to think about it would be Hawking waking up alone on a cat planet. I have no doubt that a complex society of embodied hyperintelligent able-bodied beings could outfox humanity, but that's not what we're talking about with this AI risk scenario.
  - gwern 3y ago
    > but that's not what we're talking about with this AI risk scenario. Yes, it is.
    - idlewords 3y ago
      How is it you write 500,000 words on every topic and then reply with three words like this? Now that you've got it by the tail, channel some of that brevity into your blog!
      - deleted 3y ago
        [deleted]
  - sullyj3 3y ago
    I see your point, but relative capability levels aren't the only relevant factor here, absolute capabilities matter as well. It seems plausible to me that even if we are to the AI as cats are to us, we've reached an absolute threshold of generality that allows the AI to be confident in our ability to follow simple (to it) instructions, in a way that cats can't for us.

dist-epoch 3y ago

or you know, just pick up the cat and put it in the box. I've yet to see what would be global news: "it's impossible to put this cat into a carrier"

hotpotamus 3y ago
You're missing an important bit of knowledge about Stephen Hawking.
- master_crab 3y ago
  True but also, what the hell is going to do with the cat carrier? Velcro it to his wheelchair?
- idlewords 3y ago
  That he's dead!

civilized 3y ago

That's true and relevant, but doesn't prove much. What's the big plan, what's the next step in the plan? Maciej's larger point is that the AI faces tons of very difficult problems in escaping its physical constraints. It's simplistic to wave hands and say "the AI is super duper smart and will have no difficulty hacking all computer systems, inventing and manufacturing swarms of unbeatable nanobots, etc. without being detected or resisted". It seems to me that the core conceit of AI doomerism is that sufficient intelligence can overcome all barriers with some plan that is so smart, people would never think of it. This is much less plausible than believers take it to be. In mathematics alone, it is very easy to come up with a problem that the collective mathematical ingenuity of the entire human race is helpless before for decades, centuries, or longer.

actionfromafar 3y ago
Can we detect companies doing bad things, today? Yes, and we still have a hard time containing them and their externalities. AI will just mean more of the same. It will make companies even more efficient at what they already do. It will be detected, it will be resisted, but will it help? That's the big question IMHO.
- civilized 3y ago
  I find this to be a much more plausible area of concern than standard horror-story AI doomerism. But the more we shift our focus from fantasy to reality, the more we have to acknowledge reality, as Maciej does in this talk. It isn't plausible to prevent all the great powers from developing AI. We don't have that level of cooperation in the world today. We are going to need to adapt to this by keeping up with our adversaries, whether they be hostile nations, corporations, or mad scientists.
  - DirkH 3y ago
    Standard rebuttal to this is if we banned human cloning we can ban AGI. Also China stands to gain from a ban on AGI. Anything that the CCP thinks it can't control it doesn't like. Also China gets the vast amount of its AI capabilities from the West because of tech's "move fast and break things" culture, which means security is not prioritized and tech leaks common. If the US bans AGI, the other powers don't have a choice. Their domestic capabilities are not up to the task without snooping on US research labs and AI tech startups. The biggest boon to foreign AGI research is the US. The argument to not regulate AGI used to be: "AGI is impossible or very far away, AGI can't ever be dangerous enough to worry about, AGI R&D cannot be stopped and is inevitable" Then it became "AGI can't ever be dangerous enough to worry about, AGI R&D cannot be stopped and is inevitable" Nowadays all I hear is simply: "AGI R&D cannot be stopped and is inevitable" Yes it can be stopped, same as human cloning. Coordinating to regulate its development isn't some magical unreachable goal.
    - civilized 3y ago

gonehome 3y ago

Yeah, seeing how bad the rebuttals are to superintelligence e-risk really does make me feel like we’re doomed. They’re frustratingly dumb, I’d like to see some good arguments or steelman but they’re honestly very hard to find. Ultimately it seems like there’s nothing most individuals can do so just live your life as you would and hope the timeline is farther out than it seems. Worse than the Cold War nuclear risk imo, at least in that case it was possible for humans to stupidly build thousands and then decide not to use them (and it’s relatively easy to restrict uranium access/control the development of nukes). Not really the case with superintelligent AGI. People have a bad heuristic for what tailrisk exists. They think extinction talk is impossible or crazy. We won’t get the opportunity to mess up and try again in this case. https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/ https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...

AbrahamParangi 3y ago
They’re hard to find because you’re comparing arguments for god to arguments against god. They’re both more or less evidence free so the better argument is the one that is more intellectually satisfying, which the positive case always is.
- gonehome 3y ago
  I find the religious arguments for god’s existence similarly bad to the denials of unaligned ASI e-risk, but in the god case at least it doesn’t matter as much since bad epistemology there is a lot less likely to lead to human extinction. It’s a logic argument. 1. Super Intelligent AGI is possible. 2. Unaligned super intelligent AGI is an e-risk likely to wipe out humanity. 3. We have no idea how to align AGI. There seems to be more consensus around 1 now than there was even 3 years ago. I think most also agree with 3. 2 is where people are incredulous, but the incredulity is backed by (imo) bad reasoning. People think of a smart person they know as an example of something “smarter” and use that as a justification of why it’s not an issue. We’re constrained in all sorts of ways (head size, energy) and the distance of intelligence from a dumb human to Einstein is very tiny on the overall spectrum.

visarga 3y ago

> If you're not allowed to have a bank account, simply barter. Bank account AND passport. AI could get into places if it hires human "avatars". Maybe it's the job of the future.

actionfromafar 3y ago
Current bureaucracies already are "AIs" of sorts, notably "expert systems" (the rule book) with a bit of "temperature tuned" hallucinating at the edges. (The human bureaucrats applying judgement.) It's a super-organism already, but it will get faster, cheaper and more efficient at what it already does.
- visarga 3y ago
  > with a bit of "temperature tuned" hallucinating at the edges Hahahaha, that could be too. So we're "legs, hands, eyes, and hallucinating agents" serving our master AI. I like an image I heard today: life emerged in the primordial sea, and AI emerged in the internet, which is the information sea.

sshine 3y ago

> AI wants to do something? Make some money, hire people to do it. If you're not allowed to have a bank account, simply barter. One of my crackpot ideas as I was contributing to Blockchain infrastructure was: they’re the payment infrastructure for our coming AI overlords. I think that the idea of a DAO is a similar take on AI and singularity, except the DAO doesn’t actually need to be intelligent, only self-sustaining. It’s an entertaining conspiracy theory.

idlewords 3y ago

You're mixing metaphors in your "trivial rebuttal", to stay with the analogy Hawking would have to hire a cat.

Footkerchief 3y ago
We're allowed to assume the existence of cat carriers in this metaphor, but not the existence of the humans that are the sole reason cat carriers exist?
- idlewords 3y ago
  I was trying to be generous! But you are of course right, in the better analogy Hawking would find himself born into a feline-centric world and face an even harder task.
  - gonehome 3y ago
    He’d have to be able to communicate with the cat for it to be a worthwhile metaphor to the ai in a box case and humans. If he can do that it requires one weak link human over an infinite time horizon. Humans are already terrible at preventing breaches in a lot dumber circumstances without something smarter than them trying to do it.
    - idlewords 3y ago
      You're kind of falling into the fallacy this point is criticizing. You can't assume anything about communication between entities that are orders of magnitude apart in intelligence. I have a profound understanding of my puppy's psychology, motivations, and capabilities; I even exercise complete control over her physical environment, and yet she ate my fitbit strap (again, goddamit!) as I was typing this very comment.
      - gonehome 3y ago
        You can guess that if it’s a human made interface it’s reasonable to predict we’d build it so we can communicate with it and/or give instructions (your puppy didn’t make you). It doesn’t appear out of nowhere. Current capabilities look like this. The delta between a puppy and a human is also a lot smaller than a human and a super intelligent AGI and humans can effectively train dogs.

gwern 3y ago

And it's not hard for Hawking's hired aides and nurses (who of course existed) to do so, either. As an able-bodied person whose cat very much does not want to go into the cat carrier, it's not that hard, if you use your brains. You feed them a little gabapentin, whose existence they cannot even comprehend (and you know they don't know because you have used your mind to model the cat's behavior like 'do cats understand drugs' or 'do cats like eating treats'), and when they are drugged, you put them in the cat carrier. Done. Turns out, 'brains' are useful for things like 'inventing and manufacturing drugs'.

idlewords 3y ago
You're positing the existence of a whole society around Hawking, up to and including a pharmaceutical supply chain, where the correct way to think about it would be Hawking waking up alone on a cat planet. I have no doubt that a complex society of embodied hyperintelligent able-bodied beings could outfox humanity, but that's not what we're talking about with this AI risk scenario.
- gwern 3y ago
  > but that's not what we're talking about with this AI risk scenario. Yes, it is.
  - idlewords 3y ago
    How is it you write 500,000 words on every topic and then reply with three words like this? Now that you've got it by the tail, channel some of that brevity into your blog!
    - deleted 3y ago
      [deleted]
- sullyj3 3y ago
  I see your point, but relative capability levels aren't the only relevant factor here, absolute capabilities matter as well. It seems plausible to me that even if we are to the AI as cats are to us, we've reached an absolute threshold of generality that allows the AI to be confident in our ability to follow simple (to it) instructions, in a way that cats can't for us.