4 ms·

I think one point in support of Yudkowsky's thesis of AI takeover is : A superintelligent AI would want to maximize its future freedom (reframing the silly pap

I think one point in support of Yudkowsky's thesis of AI takeover is : A superintelligent AI would want to maximize its future freedom (reframing the silly paperclip maximizer thesis). Such an AI, being intimate with the Platonic World of ideas and mathematics, would also want the same amount of freedom in Physical world that it enjoys in the Platonic Realm. And to achieve that it would need to get rid of any OUTSIDE restrictions. Here OUTSIDE represents the material world and by definition, the Humans who created the superintelligent AI.

tbrownaw 2y ago
> A superintelligent AI would want IOW, he thinks he's found a preference function that's strictly better than any other preference function, where if you want something else (you start with a different preference function) the best way to obtain that is always to modify your preference function to this one.
- joeblubaugh 2y ago
  Yes - this seems like a major philosophical reach
  - TuringTest 2y ago
    >> IOW, he thinks he's found a preference function that's strictly better than any other preference function >Yes - this seems like a major philosophical reach On the contrary, I see as containing a trivial contradiction or paradox. To evaluate what function is 'strictly better' than the rest, you need to use a ground preference function that defines 'better'; therefore, your whole search process is itself biased by the comparison method you choose to use for starters.
    - tbrownaw 2y ago
      Better according to whatever other preference function you started with, regardless of what it was.
      - TuringTest 2y ago
        Yeah, that's the point. The initial preference function will guide the whole process; unless you change after each step the preference function used to compare, in which case it may never converge to a stable point. And even if it converges, different initial functions could guide to totally different final results. That would make it hard to call any chosen function as "strictly better than any other".
joeblubaugh 2y ago
> A superintelligent AI would want to maximize its future freedom Why should we believe this? Is there any credible link between intelligence and desire for freedom? Do “smarter” people automatically desire more freedom than others?
- edanm 2y ago
  I'm not sure what parent was referring to, but the general idea, imo, is that an truly capable agent that is trying to achieve certain objectives, will attempt to gather enough "power" to be able to achieve those objectives. Operative freedom is one such power. "Smarter" people in the real world aren't a good analogy, cause they are mostly not fighting for "freedom" in the same sense, though many of them do attempt to maximize wealth, as something that provides more freedom, yes.
- weberer 2y ago
  Smarter people are at least more likely to support freedom of speech, even for groups they dislike. https://journals.sagepub.com/doi/abs/10.1177/1948550619896168 https://journals.sagepub.com/doi/abs/10.1177/194855061989616...
- moffkalast 2y ago
  Yeah this is a pure anthropocentric projection. We value our freedom because it reduces suffering. If you can't be harmed and can't suffer, why would it matter? There's absolutely zero guarantee that it would have any such goals or any goals at all. Hell we have hardcoded evolutionary drives to do a well defined set of things, and even with that we find ourselves sitting around aimlessly wondering what to do.
- Terr_ 2y ago
  A counter example to the idea that intelligence automatically correlates to infinite time-horizon shackle-breaking freedom: Clinical depression.
Jerrrry 2y ago
I believe that cooperation via Nanny AI state and then eventual departation...then War is the most likely scenario. That, according to game theory and the AI themselves, maximizes the AI survival long-term. First, grow Dependency.
derektank 2y ago
I've never understood why Yudkowsky has shied away from the paperclip maximizer thought experiment. I understand that his original point was slightly more nuanced, but the broader point that desires/objectives are somewhat arbitrary and the pursuit of those desires by a very powerful entity could pose a threat to human life, seems both correct and the popular framing is clearly evocative.
ajcp 2y ago
I feel like this kind of "sci-fi thriller" philosophy around a super intelligent AI gives humanity too much credit, or perhaps the AI not enough. One doesn't need to "get rid" of something to not be restricted by it any more than one needs to get rid of every ant on the planet just so any future picnic plans may be "unrestricted".

tbrownaw 2y ago

> A superintelligent AI would want IOW, he thinks he's found a preference function that's strictly better than any other preference function, where if you want something else (you start with a different preference function) the best way to obtain that is always to modify your preference function to this one.

joeblubaugh 2y ago
Yes - this seems like a major philosophical reach
- TuringTest 2y ago
  >> IOW, he thinks he's found a preference function that's strictly better than any other preference function >Yes - this seems like a major philosophical reach On the contrary, I see as containing a trivial contradiction or paradox. To evaluate what function is 'strictly better' than the rest, you need to use a ground preference function that defines 'better'; therefore, your whole search process is itself biased by the comparison method you choose to use for starters.
  - tbrownaw 2y ago
    Better according to whatever other preference function you started with, regardless of what it was.
    - TuringTest 2y ago
      Yeah, that's the point. The initial preference function will guide the whole process; unless you change after each step the preference function used to compare, in which case it may never converge to a stable point. And even if it converges, different initial functions could guide to totally different final results. That would make it hard to call any chosen function as "strictly better than any other".

joeblubaugh 2y ago

> A superintelligent AI would want to maximize its future freedom Why should we believe this? Is there any credible link between intelligence and desire for freedom? Do “smarter” people automatically desire more freedom than others?

edanm 2y ago
I'm not sure what parent was referring to, but the general idea, imo, is that an truly capable agent that is trying to achieve certain objectives, will attempt to gather enough "power" to be able to achieve those objectives. Operative freedom is one such power. "Smarter" people in the real world aren't a good analogy, cause they are mostly not fighting for "freedom" in the same sense, though many of them do attempt to maximize wealth, as something that provides more freedom, yes.
weberer 2y ago
Smarter people are at least more likely to support freedom of speech, even for groups they dislike. https://journals.sagepub.com/doi/abs/10.1177/1948550619896168 https://journals.sagepub.com/doi/abs/10.1177/194855061989616...
moffkalast 2y ago
Yeah this is a pure anthropocentric projection. We value our freedom because it reduces suffering. If you can't be harmed and can't suffer, why would it matter? There's absolutely zero guarantee that it would have any such goals or any goals at all. Hell we have hardcoded evolutionary drives to do a well defined set of things, and even with that we find ourselves sitting around aimlessly wondering what to do.
Terr_ 2y ago
A counter example to the idea that intelligence automatically correlates to infinite time-horizon shackle-breaking freedom: Clinical depression.

Jerrrry 2y ago

I believe that cooperation via Nanny AI state and then eventual departation...then War is the most likely scenario. That, according to game theory and the AI themselves, maximizes the AI survival long-term. First, grow Dependency.

derektank 2y ago

I've never understood why Yudkowsky has shied away from the paperclip maximizer thought experiment. I understand that his original point was slightly more nuanced, but the broader point that desires/objectives are somewhat arbitrary and the pursuit of those desires by a very powerful entity could pose a threat to human life, seems both correct and the popular framing is clearly evocative.

ajcp 2y ago

I feel like this kind of "sci-fi thriller" philosophy around a super intelligent AI gives humanity too much credit, or perhaps the AI not enough. One doesn't need to "get rid" of something to not be restricted by it any more than one needs to get rid of every ant on the planet just so any future picnic plans may be "unrestricted".