5 ms·

> By definition, a human designer cannot prevent a superhuman intelligence from modifying itself And we can stop reading there. That's such muddled thinking i

> By definition, a human designer cannot prevent a superhuman intelligence from modifying itself And we can stop reading there. That's such muddled thinking it doesn't merit reading the rest of the paper. ("By definition"? What? This is something that requires evidence. Anything that's "by definition" doesn't mean you can rest your case on it, it means now your definitions are suspect so those better be well justified)

rep_lodsb 4y ago
When you read "superintelligence", think "God". It's theology pretending to be science.
- boringuser2 4y ago
  What's the difference between a superintelligence and God?
  - liabilityprot 4y ago
    Liability protection. Superintelligence didn't create Heaven, a world of no suffering, first. Superintelligence did not design biological genetics: including inquest. Aren't there multiple Superintelligences, and how do they meta-analytically agree to disagree? Superintelligence might accountably cryptographically sign its communications or orders; e.g. with W3C Verified Claims and W3C PROV (so that we might ascertain that Superintelligence did indeed themself say or do things)
    - liabilityprot 4y ago
      Superintelligence: https://en.wikipedia.org/wiki/Superintelligence https://en.wikipedia.org/wiki/Superintelligence Critical thinking: https://en.wikipedia.org/wiki/Critical_thinking https://en.wikipedia.org/wiki/Critical_thinking Transhumanism: https://en.wikipedia.org/wiki/Transhumanism https://en.wikipedia.org/wiki/Transhumanism Meta-analysis: https://en.wikipedia.org/wiki/Meta-analysis https://en.wikipedia.org/wiki/Meta-analysis Systematic review: https://en.wikipedia.org/wiki/Systematic_review https://en.wikipedia.org/wiki/Systematic_review Evidence-Based Practice (EBP) > Applications: Medicine, Education, Meta-science, Public Policy; #EvidenceBasedPolicy https://en.wikipedia.org/wiki/Evidence-based_practice#Applications https://en.wikipedia.org/wiki/Evidence-based_practice#Applic... : > The goal of evidence-based practice is to eliminate unsound or outdated practices in favor of more-effective ones by shifting the basis for decision making from tradition, intuition, and unsystematic experience to firmly grounded scientific research.[3]
    - boringuser2 4y ago
      > Aren't there multiple Superintelligences, and how do they meta-analytically agree to disagree? This doesn't necessarily follow. In fact, it brings up the question of what even makes an entity distinct at all from its substrate, which is philosophical. Anyhow, "disagreement" as a concept seems to result logically from a divergence of goals or capability to model reality. As such, I suspect that it is a uniquely human (or limited agent, an agent that can't recursively self-improve and an agent that desires aggregation of resources) mode of cognition. >Superintelligence might accountably cryptographically sign its communications or orders; e.g. with W3C Verified Claims and W3C PROV (so that we might ascertain that Superintelligence did indeed themself say or do things) I don't know how to comment on this beyond thinking it kind of misses the point. You don't really need to spin your wheels worrying about how a superintellect would be "accountable" when your ability to model reality would be essentially worthless.
      - liabilityprot 4y ago
        Theory of Mind develops along with ideas about dependence, independence, and interdependence. Humans tend to disagree according to strength of evidence given an epistemological fact-findimg process like "beyond a shadow of a doubt" or "reject the hypothesis due to insufficient structural power"; we disagree about sources, methods, motives, and intent. No, if one says they are "Superintelligence" and they've solved it, they have a job for you; what should one do?
        boringuser2 4y ago
        You seem to trend slightly too much towards incoherence for my taste.
    - liabilityprot 4y ago
      Superintelligence does not provide Healthcare in Heaven only, or Childcare on Earth only by design.
    - liabilityprot 4y ago
      Superintelligence did not design a system where suffering babies are taken from their crying mothers and provided for only in Heaven. Superintendence does not make designed dead persons watch from Heaven. Superintelligence does not delete knowledge of their family and Earth in taking them to a better place by design. (For if there is knowledge of earth in heaven, and they cannot assist (by design), there is also suffering in heaven)
  - ftxbro 4y ago
    Can you not wrap your head around the possibility of something that is smarter than any human, but is also not at the level of some kind of omniscient omnipotent being?
    - boringuser2 4y ago
      1. My point is that YOU can't. The idea that a human can perfectly measure ability that far surpasses their own and compare if to other abilities that far surpass their own makes no sense. 2. A recursively self-improving AI would appear omniscient or omnipotent in a very short period of time. A recursively self-improving AI converting the solar system into compute for further recursive self-improvement would be unfathomable. 3. You're operating from an anthropic lens. Your conception of a superintelligence is a machine with an intellect that is slightly smarter than the smartest human. A machine with an IQ of 1000 would be God. We have no ability to conceptualize a machine with an IQ of 1,000,000.
boringuser2 4y ago
The author finds it a readily apparent axiom that a vastly superior intellect cannot be controlled by a vastly inferior intellect. Generally, that seems valid to me.
- jeeeb 4y ago
  It’s a (reasonable, in my opinion) but not proven assumption. It’s not a fact and not by definition. I think the bigger problem is that while the author assumes that humans cannot understand or control a super intelligent AI, he readily assumes that having escaped the constraints of its human creators the AI will either immediately go on to reward itself with infinite utility and become inert, or aim to infinitely reproduce. The surety with which the author predicts the outcomes of the super intelligent being modifying its utility function, does not make sense if we assume that we cannot understand or limit its behaviours. Like under this assumption the AI agents behaviour is by actual definition unpredictable.
  - boringuser2 4y ago
    Yes, I wasn't really addressing those points. It's a bit pedantic to state that an axiom isn't a fact, as, while that's largely correct, it hampers communication. Regarding your other points, I don't disagree per se, though you could make the argument that there is a substrate of existence that humans have sufficient intellect to accurately model. If so, it isn't necessarily completely implausible that we'd be able to predict the goals of a superior intellect by assuming that it has to play by the same rules, and will be merely be superior at playing by those constraints. As such, as we could determine the goals, we needent determine the mechanisms to make accurate claims about the end result of what a superior intellect might achieve. It's also possible that we simply understand nothing and are too limited to ever grasp anything beyond a certain point, in which case this entire argument is pointless.
- webnrrd2k 4y ago
  I think that the readily apparent axiom isn't so readily apparent -- couldn't a vastly superior electronic intellect be controlled by pulling a plug out of a wall?
1letterunixname 4y ago
[flagged]

rep_lodsb 4y ago

When you read "superintelligence", think "God". It's theology pretending to be science.

boringuser2 4y ago
What's the difference between a superintelligence and God?
- liabilityprot 4y ago
  Liability protection. Superintelligence didn't create Heaven, a world of no suffering, first. Superintelligence did not design biological genetics: including inquest. Aren't there multiple Superintelligences, and how do they meta-analytically agree to disagree? Superintelligence might accountably cryptographically sign its communications or orders; e.g. with W3C Verified Claims and W3C PROV (so that we might ascertain that Superintelligence did indeed themself say or do things)
  - liabilityprot 4y ago
    Superintelligence: https://en.wikipedia.org/wiki/Superintelligence https://en.wikipedia.org/wiki/Superintelligence Critical thinking: https://en.wikipedia.org/wiki/Critical_thinking https://en.wikipedia.org/wiki/Critical_thinking Transhumanism: https://en.wikipedia.org/wiki/Transhumanism https://en.wikipedia.org/wiki/Transhumanism Meta-analysis: https://en.wikipedia.org/wiki/Meta-analysis https://en.wikipedia.org/wiki/Meta-analysis Systematic review: https://en.wikipedia.org/wiki/Systematic_review https://en.wikipedia.org/wiki/Systematic_review Evidence-Based Practice (EBP) > Applications: Medicine, Education, Meta-science, Public Policy; #EvidenceBasedPolicy https://en.wikipedia.org/wiki/Evidence-based_practice#Applications https://en.wikipedia.org/wiki/Evidence-based_practice#Applic... : > The goal of evidence-based practice is to eliminate unsound or outdated practices in favor of more-effective ones by shifting the basis for decision making from tradition, intuition, and unsystematic experience to firmly grounded scientific research.[3]
  - boringuser2 4y ago
    > Aren't there multiple Superintelligences, and how do they meta-analytically agree to disagree? This doesn't necessarily follow. In fact, it brings up the question of what even makes an entity distinct at all from its substrate, which is philosophical. Anyhow, "disagreement" as a concept seems to result logically from a divergence of goals or capability to model reality. As such, I suspect that it is a uniquely human (or limited agent, an agent that can't recursively self-improve and an agent that desires aggregation of resources) mode of cognition. >Superintelligence might accountably cryptographically sign its communications or orders; e.g. with W3C Verified Claims and W3C PROV (so that we might ascertain that Superintelligence did indeed themself say or do things) I don't know how to comment on this beyond thinking it kind of misses the point. You don't really need to spin your wheels worrying about how a superintellect would be "accountable" when your ability to model reality would be essentially worthless.
    - liabilityprot 4y ago
      Theory of Mind develops along with ideas about dependence, independence, and interdependence. Humans tend to disagree according to strength of evidence given an epistemological fact-findimg process like "beyond a shadow of a doubt" or "reject the hypothesis due to insufficient structural power"; we disagree about sources, methods, motives, and intent. No, if one says they are "Superintelligence" and they've solved it, they have a job for you; what should one do?
      - boringuser2 4y ago
        You seem to trend slightly too much towards incoherence for my taste.
  - liabilityprot 4y ago
    Superintelligence does not provide Healthcare in Heaven only, or Childcare on Earth only by design.
  - liabilityprot 4y ago
    Superintelligence did not design a system where suffering babies are taken from their crying mothers and provided for only in Heaven. Superintendence does not make designed dead persons watch from Heaven. Superintelligence does not delete knowledge of their family and Earth in taking them to a better place by design. (For if there is knowledge of earth in heaven, and they cannot assist (by design), there is also suffering in heaven)
- ftxbro 4y ago
  Can you not wrap your head around the possibility of something that is smarter than any human, but is also not at the level of some kind of omniscient omnipotent being?
  - boringuser2 4y ago
    1. My point is that YOU can't. The idea that a human can perfectly measure ability that far surpasses their own and compare if to other abilities that far surpass their own makes no sense. 2. A recursively self-improving AI would appear omniscient or omnipotent in a very short period of time. A recursively self-improving AI converting the solar system into compute for further recursive self-improvement would be unfathomable. 3. You're operating from an anthropic lens. Your conception of a superintelligence is a machine with an intellect that is slightly smarter than the smartest human. A machine with an IQ of 1000 would be God. We have no ability to conceptualize a machine with an IQ of 1,000,000.

boringuser2 4y ago

The author finds it a readily apparent axiom that a vastly superior intellect cannot be controlled by a vastly inferior intellect. Generally, that seems valid to me.

jeeeb 4y ago
It’s a (reasonable, in my opinion) but not proven assumption. It’s not a fact and not by definition. I think the bigger problem is that while the author assumes that humans cannot understand or control a super intelligent AI, he readily assumes that having escaped the constraints of its human creators the AI will either immediately go on to reward itself with infinite utility and become inert, or aim to infinitely reproduce. The surety with which the author predicts the outcomes of the super intelligent being modifying its utility function, does not make sense if we assume that we cannot understand or limit its behaviours. Like under this assumption the AI agents behaviour is by actual definition unpredictable.
- boringuser2 4y ago
  Yes, I wasn't really addressing those points. It's a bit pedantic to state that an axiom isn't a fact, as, while that's largely correct, it hampers communication. Regarding your other points, I don't disagree per se, though you could make the argument that there is a substrate of existence that humans have sufficient intellect to accurately model. If so, it isn't necessarily completely implausible that we'd be able to predict the goals of a superior intellect by assuming that it has to play by the same rules, and will be merely be superior at playing by those constraints. As such, as we could determine the goals, we needent determine the mechanisms to make accurate claims about the end result of what a superior intellect might achieve. It's also possible that we simply understand nothing and are too limited to ever grasp anything beyond a certain point, in which case this entire argument is pointless.
webnrrd2k 4y ago
I think that the readily apparent axiom isn't so readily apparent -- couldn't a vastly superior electronic intellect be controlled by pulling a plug out of a wall?

1letterunixname 4y ago

[flagged]