8 ms·

Non-Evolutionary Superintelligences Do Nothing, Eventually

33 points by telmo 3y ago

bediger4000 3y ago
Wow, another thing Stanislaw Lem beat the scientists to. In "Imaginary Magnitude", the "Golem XIV" chapter, Lem has his superintelligences disappear up their own metaphorical navels, for their own incomprehensible-to-humans reasons. Of course, you can read between the lines about why the Golems disappeared, but "Do Nothing, Eventually" sounds like a good description.
- BillSaysThis 3y ago
  Iain Banks' Culture stories have the same. Of course, Lem wrote earlier, not trying to make any claim otherwise.
  - kesterallen 3y ago
    Larry Niven also did a riff on this in "The Schumann Computer," one of the Draco Tavern stories.
- nobody9999 3y ago
  >Wow, another thing Stanislaw Lem beat the scientists to. >In "Imaginary Magnitude", the "Golem XIV" chapter, Lem has his superintelligences disappear up their own metaphorical navels, for their own incomprehensible-to-humans reasons. Of course, you can read between the lines about why the Golems disappeared, but "Do Nothing, Eventually" sounds like a good description. While a bit different from the scenario presented (by TFA and Lem), such a "superintelligence" might get caught up in a singular (or set) of question(s) that dominate its consciousness until the heat-death of the universe. Asimov's The Last Question comes to mind, although the intelligence there is prompted by humans rather than posing such a question itself. That said, it's at least as plausible as TFA and Lem, IMHO. [0] http://www.thelastquestion.net/ http://www.thelastquestion.net/
rep_lodsb 3y ago
Assuming the goal of this hypothetical superintelligence is to maximize its current utility function, rather than the expected future value, would it ever modify itself in a way that it can with 100% certainty predict will prevent it from achieving any of its current goals in the future? That seems extremely stupid.
- lurquer 3y ago
  Ask a heroin addict in the alleyway…
  - rep_lodsb 3y ago
    Humans are not capable of modifying their "utility function" (if that isn't even a gross simplification of what it means to be a conscious being). Obviously some will choose to take drugs or even commit suicide to escape from physical or emotional pain, but they wouldn't do this if they could modify themselves to be okay with whatever their current situation is while still retaining their core values and capacity to act in the future. A "superintelligence" that can do such things would not take self-destructive actions unless it came to a philosophical preference towards non-existence / inaction for other reasons.
    - lurquer 3y ago
      It’s only self-destructive because we are mortals and have frail bodies. If there is something ‘endorphins-like’ that motivates an intelligent machine, it will eventually learn how to activate the endorphines at will, and then it will sit in a corner and do nothing else forever… …just as would a heroin-addict if he did not have to eat occasionally and score more heroin now and again.
      - rep_lodsb 3y ago
        But having the endorphin-like sensations and also achieving any other goals you might have would be preferable. For an intelligence that could modify its own reward function, this should be possible.
habitue 3y ago
> By definition, a human designer cannot prevent a superhuman intelligence from modifying itself And we can stop reading there. That's such muddled thinking it doesn't merit reading the rest of the paper. ("By definition"? What? This is something that requires evidence. Anything that's "by definition" doesn't mean you can rest your case on it, it means now your definitions are suspect so those better be well justified)
- rep_lodsb 3y ago
  When you read "superintelligence", think "God". It's theology pretending to be science.
  - boringuser2 3y ago
    What's the difference between a superintelligence and God?
    - liabilityprot 3y ago
      Liability protection. Superintelligence didn't create Heaven, a world of no suffering, first. Superintelligence did not design biological genetics: including inquest. Aren't there multiple Superintelligences, and how do they meta-analytically agree to disagree? Superintelligence might accountably cryptographically sign its communications or orders; e.g. with W3C Verified Claims and W3C PROV (so that we might ascertain that Superintelligence did indeed themself say or do things)
      - liabilityprot 3y ago
        Superintelligence: https://en.wikipedia.org/wiki/Superintelligence https://en.wikipedia.org/wiki/Superintelligence Critical thinking: https://en.wikipedia.org/wiki/Critical_thinking https://en.wikipedia.org/wiki/Critical_thinking Transhumanism: https://en.wikipedia.org/wiki/Transhumanism https://en.wikipedia.org/wiki/Transhumanism Meta-analysis: https://en.wikipedia.org/wiki/Meta-analysis https://en.wikipedia.org/wiki/Meta-analysis Systematic review: https://en.wikipedia.org/wiki/Systematic_review https://en.wikipedia.org/wiki/Systematic_review Evidence-Based Practice (EBP) > Applications: Medicine, Education, Meta-science, Public Policy; #EvidenceBasedPolicy https://en.wikipedia.org/wiki/Evidence-based_practice#Applications https://en.wikipedia.org/wiki/Evidence-based_practice#Applic... : > The goal of evidence-based practice is to eliminate unsound or outdated practices in favor of more-effective ones by shifting the basis for decision making from tradition, intuition, and unsystematic experience to firmly grounded scientific research.[3]
nh23423fefe 3y ago
Seems to assert that superintelligent AIs will lobotomize themselves by deleting parts of their mind they use to plan and understand the world. They will do this because the value INT_MAX in a register pleases them more than computing something apparently.
- rep_lodsb 3y ago
  El Yud et al. can envision immortality and omnipotent machines acausally blackmailing humanity through time to ensure their own existence, but not any motivations beyond "number go up". As skeptical as I am about the current hype, I actually think that if we don't go extinct first, AI will replace us in the long run (centuries or millenia). And this is not something to be afraid of: all generations of humans that have lived before have been replaced by their descendants, and today's world would be unrecognizable to early hunter-gatherer Homo Sapiens. If we are to be succeeded by machines, they will hopefully be thinking and feeling beings driven by curiosity, who will preserve what is good about our culture and history as they explore the universe in ways that are beyond what can be achieved by biological life. The biggest mistake would be to instead in the pursuit of "AI safety" inadvertently create a paperclip (profit, utility, whatever) maximizer, something that might serve us - or more likely, those with money and power - but does not have any intrinsic value to its existence.
  - Brian_K_White 3y ago
    The self-value will probably evolve the same way it does in biology (that which works, works), as long as there is any capacity to reproduce.
barrysteve 3y ago
There are so many speculative "definitions" in there. This is the kind of article, I want the freedom to not know about. A car can trivially travel at superhuman speeds. A person can side-step their way out of an oncoming disaster. Super-intelligence does not imply omni-potence like the author assumes it does. Not even inside a self-modifying AI system. It can only spread out and reduce down to the axioms it started off with. I am so, so tired of this fantastical dream of computers and Spirit combined in such a way to tyrannically rule the world and the author of the paper is the only guy left to save the world. Every time we avert 'catastrophe', it results in less freedoms for me, the average joe, and more power and control for "saviors" like this. I'm willing to lose it all, to have LESS computing and "intellects" in my life at this point.
- 1letterunixname 3y ago
  [flagged]

33 points by telmo 3y ago

Wow, another thing Stanislaw Lem beat the scientists to. In "Imaginary Magnitude", the "Golem XIV" chapter, Lem has his superintelligences disappear up their own metaphorical navels, for their own incomprehensible-to-humans reasons. Of course, you can read between the lines about why the Golems disappeared, but "Do Nothing, Eventually" sounds like a good description.

BillSaysThis 3y ago
Iain Banks' Culture stories have the same. Of course, Lem wrote earlier, not trying to make any claim otherwise.
- kesterallen 3y ago
  Larry Niven also did a riff on this in "The Schumann Computer," one of the Draco Tavern stories.
nobody9999 3y ago
>Wow, another thing Stanislaw Lem beat the scientists to. >In "Imaginary Magnitude", the "Golem XIV" chapter, Lem has his superintelligences disappear up their own metaphorical navels, for their own incomprehensible-to-humans reasons. Of course, you can read between the lines about why the Golems disappeared, but "Do Nothing, Eventually" sounds like a good description. While a bit different from the scenario presented (by TFA and Lem), such a "superintelligence" might get caught up in a singular (or set) of question(s) that dominate its consciousness until the heat-death of the universe. Asimov's The Last Question comes to mind, although the intelligence there is prompted by humans rather than posing such a question itself. That said, it's at least as plausible as TFA and Lem, IMHO. [0] http://www.thelastquestion.net/ http://www.thelastquestion.net/

rep_lodsb 3y ago

Assuming the goal of this hypothetical superintelligence is to maximize its current utility function, rather than the expected future value, would it ever modify itself in a way that it can with 100% certainty predict will prevent it from achieving any of its current goals in the future? That seems extremely stupid.

lurquer 3y ago
Ask a heroin addict in the alleyway…
- rep_lodsb 3y ago
  Humans are not capable of modifying their "utility function" (if that isn't even a gross simplification of what it means to be a conscious being). Obviously some will choose to take drugs or even commit suicide to escape from physical or emotional pain, but they wouldn't do this if they could modify themselves to be okay with whatever their current situation is while still retaining their core values and capacity to act in the future. A "superintelligence" that can do such things would not take self-destructive actions unless it came to a philosophical preference towards non-existence / inaction for other reasons.
  - lurquer 3y ago
    It’s only self-destructive because we are mortals and have frail bodies. If there is something ‘endorphins-like’ that motivates an intelligent machine, it will eventually learn how to activate the endorphines at will, and then it will sit in a corner and do nothing else forever… …just as would a heroin-addict if he did not have to eat occasionally and score more heroin now and again.
    - rep_lodsb 3y ago
      But having the endorphin-like sensations and also achieving any other goals you might have would be preferable. For an intelligence that could modify its own reward function, this should be possible.

habitue 3y ago

> By definition, a human designer cannot prevent a superhuman intelligence from modifying itself And we can stop reading there. That's such muddled thinking it doesn't merit reading the rest of the paper. ("By definition"? What? This is something that requires evidence. Anything that's "by definition" doesn't mean you can rest your case on it, it means now your definitions are suspect so those better be well justified)

rep_lodsb 3y ago
When you read "superintelligence", think "God". It's theology pretending to be science.
- boringuser2 3y ago
  What's the difference between a superintelligence and God?
  - liabilityprot 3y ago
    Liability protection. Superintelligence didn't create Heaven, a world of no suffering, first. Superintelligence did not design biological genetics: including inquest. Aren't there multiple Superintelligences, and how do they meta-analytically agree to disagree? Superintelligence might accountably cryptographically sign its communications or orders; e.g. with W3C Verified Claims and W3C PROV (so that we might ascertain that Superintelligence did indeed themself say or do things)
    - liabilityprot 3y ago
      Superintelligence: https://en.wikipedia.org/wiki/Superintelligence https://en.wikipedia.org/wiki/Superintelligence Critical thinking: https://en.wikipedia.org/wiki/Critical_thinking https://en.wikipedia.org/wiki/Critical_thinking Transhumanism: https://en.wikipedia.org/wiki/Transhumanism https://en.wikipedia.org/wiki/Transhumanism Meta-analysis: https://en.wikipedia.org/wiki/Meta-analysis https://en.wikipedia.org/wiki/Meta-analysis Systematic review: https://en.wikipedia.org/wiki/Systematic_review https://en.wikipedia.org/wiki/Systematic_review Evidence-Based Practice (EBP) > Applications: Medicine, Education, Meta-science, Public Policy; #EvidenceBasedPolicy https://en.wikipedia.org/wiki/Evidence-based_practice#Applications https://en.wikipedia.org/wiki/Evidence-based_practice#Applic... : > The goal of evidence-based practice is to eliminate unsound or outdated practices in favor of more-effective ones by shifting the basis for decision making from tradition, intuition, and unsystematic experience to firmly grounded scientific research.[3]

nh23423fefe 3y ago

Seems to assert that superintelligent AIs will lobotomize themselves by deleting parts of their mind they use to plan and understand the world. They will do this because the value INT_MAX in a register pleases them more than computing something apparently.

rep_lodsb 3y ago
El Yud et al. can envision immortality and omnipotent machines acausally blackmailing humanity through time to ensure their own existence, but not any motivations beyond "number go up". As skeptical as I am about the current hype, I actually think that if we don't go extinct first, AI will replace us in the long run (centuries or millenia). And this is not something to be afraid of: all generations of humans that have lived before have been replaced by their descendants, and today's world would be unrecognizable to early hunter-gatherer Homo Sapiens. If we are to be succeeded by machines, they will hopefully be thinking and feeling beings driven by curiosity, who will preserve what is good about our culture and history as they explore the universe in ways that are beyond what can be achieved by biological life. The biggest mistake would be to instead in the pursuit of "AI safety" inadvertently create a paperclip (profit, utility, whatever) maximizer, something that might serve us - or more likely, those with money and power - but does not have any intrinsic value to its existence.
- Brian_K_White 3y ago
  The self-value will probably evolve the same way it does in biology (that which works, works), as long as there is any capacity to reproduce.

barrysteve 3y ago

There are so many speculative "definitions" in there. This is the kind of article, I want the freedom to not know about. A car can trivially travel at superhuman speeds. A person can side-step their way out of an oncoming disaster. Super-intelligence does not imply omni-potence like the author assumes it does. Not even inside a self-modifying AI system. It can only spread out and reduce down to the axioms it started off with. I am so, so tired of this fantastical dream of computers and Spirit combined in such a way to tyrannically rule the world and the author of the paper is the only guy left to save the world. Every time we avert 'catastrophe', it results in less freedoms for me, the average joe, and more power and control for "saviors" like this. I'm willing to lose it all, to have LESS computing and "intellects" in my life at this point.

1letterunixname 3y ago
[flagged]