9 ms·

The author finds it a readily apparent axiom that a vastly superior intellect cannot be controlled by a vastly inferior intellect. Generally, that seems valid

The author finds it a readily apparent axiom that a vastly superior intellect cannot be controlled by a vastly inferior intellect. Generally, that seems valid to me.

jeeeb 4y ago
It’s a (reasonable, in my opinion) but not proven assumption. It’s not a fact and not by definition. I think the bigger problem is that while the author assumes that humans cannot understand or control a super intelligent AI, he readily assumes that having escaped the constraints of its human creators the AI will either immediately go on to reward itself with infinite utility and become inert, or aim to infinitely reproduce. The surety with which the author predicts the outcomes of the super intelligent being modifying its utility function, does not make sense if we assume that we cannot understand or limit its behaviours. Like under this assumption the AI agents behaviour is by actual definition unpredictable.
- boringuser2 4y ago
  Yes, I wasn't really addressing those points. It's a bit pedantic to state that an axiom isn't a fact, as, while that's largely correct, it hampers communication. Regarding your other points, I don't disagree per se, though you could make the argument that there is a substrate of existence that humans have sufficient intellect to accurately model. If so, it isn't necessarily completely implausible that we'd be able to predict the goals of a superior intellect by assuming that it has to play by the same rules, and will be merely be superior at playing by those constraints. As such, as we could determine the goals, we needent determine the mechanisms to make accurate claims about the end result of what a superior intellect might achieve. It's also possible that we simply understand nothing and are too limited to ever grasp anything beyond a certain point, in which case this entire argument is pointless.
webnrrd2k 4y ago
I think that the readily apparent axiom isn't so readily apparent -- couldn't a vastly superior electronic intellect be controlled by pulling a plug out of a wall?
- boringuser2 4y ago
  No, only a very marginally superior intellect could be controlled in such a way even at a cursory glance, and I'm not even convinced a 250 IQ machine could be stymied like this.
cozos 4y ago
Toxoplasmosis
- Teever 4y ago
  Also every 'smart' person who thought they could talk their way out of a police interrogation and is now spending the rest of their life in prison secured by knuckle dragging brutes. Idiot Australians who smuggle drugs to/from Indonesia come to mind.
  - boringuser2 4y ago
    The difference in intelligence between the police and a singular criminal, especially the police as an organism, is vastly in favor of the police.
    - nobody9999 4y ago
      >The difference in intelligence between the police and a singular criminal, especially the police as an organism, is vastly in favor of the police. I don't agree, as that's highly dependent on that singular criminal and the resources police are willing to expend on catching them. The police definitely have an advantage, as I pointed (most recently) here[0]: Law enforcement aren't superhuman. They're just as dumb (or smart, but the really smart ones end up in corner offices rather than police stations like police and more common criminals) as the next guy. Their big advantage, especially in a circumstance like this, is that they only have to get it right (i.e., find some evidence) once. The alleged perpetrator of a crime needs to get it right (in covering their tracks, destroying evidence, etc.) every single time to make sure they aren't identified and caught. [0] https://news.ycombinator.com/item?id=35550259 https://news.ycombinator.com/item?id=35550259 Edit: Corrected link. Clarified prose.
    - Teever 4y ago
      Yeah but I'm talking about scenarios where all people needed to do to avoid being locked in a little box for life was either A. Not do the obviously stupid thing that everyone is telling them is stupid and that they should absolutely not do. and B. Shut the fuck up.[0] There are supposedly brilliant people serving life sentences right now and the only reason they're facing that situation is because they couldn't shut the fuck up when they sat down in a little room with an at best slightly above average intelligence police officer. Stupid is as stupid does. [0] https://www.youtube.com/watch?v=sgWHrkDX35o https://www.youtube.com/watch?v=sgWHrkDX35o
omneity 4y ago
I beg to differ. All the might of a superior intellect will be of no use if a bear locks a human down and starts to maul them. The mistaken assumption that GP is talking about is related to conflating intellectual capacity with physical capacity. Another analogy is a brain in a vat. If the brain only has read capability over the world surrounding it, then no amount of processing power can enable it to change anything about its situation.
- boringuser2 4y ago
  Humans completely dominate bears, and the difference in intellect isn't even relatively huge.
  - 1letterunixname 4y ago
    In what context? If you don't have a gun and a knife and you're on Vancouver Island, I assure you that the bears will eat you no matter how keyboard macho you think you are.
    - habitue 4y ago
      Right, but the point is that with the intellect you don't show up in that situation without a knife or a gun in the first place. A competent intelligent human foresees that being a bad situation, avoids it, and shows up armed or with bear spray. This is exactly the same thing a chess program does when it avoids your trap. Intelligence compensates for physical weakness.
    - boringuser2 4y ago
      This is such a pointless discussion that I can't even begin to countenance it. Go to a zoo. Are bears walking around with humans in cages?
- 1letterunixname 4y ago
  Exactly. Power is the crucial element. You may be a genius but still be just another prisoner subject to unyielding limitations of the environment structured to maintain particular controls.
Turskarama 4y ago
You can be killed by a bacteria, or even a falling rock. Intelligence gives an advantage, but it's only one aspect of interacting with the world.
1letterunixname 4y ago
Assume a prison were sound and a lack of contraband: a jailer doesn't need to be a genius to not open the door for Hannibal Lecter.
- vlovich123 4y ago
  A) Human prisons frequently have contraband brought in by jailers or jailers looking the other way because humans have flaws / may think they see opportunities to benefit. B) prisoners regularly escape from physical systems. AI is already being connected to the internet to perform actions on the user behalf so not unreasonable to think that C) look at ex machina - not an unrealistic scenario where the AI takes advantage of an opportunity to get out of jail D) human physical systems have exploitable flaws. Physical systems are easier to secure than digital ones. Human software security mechanisms have flaws. How are you going to build this jail if the AI is just spending its time probing weaknesses in whatever system you’re running it on to escape the jail / sandbox?
- deleted 4y ago
  [deleted]
- diziet 4y ago
  See https://rationalwiki.org/wiki/AI-box_experiment https://rationalwiki.org/wiki/AI-box_experiment
habitue 4y ago
Like I said, it's something that requires an argument or evidence. It's not something that comes by definition. People seem to think arguing that something is true by definition is some airtight case. If your definition of superintelligence includes a part about not being controllable by humans, then you have a weird definition and calling that thing "superintelligence" is an attempt to be confusing on purpose, or to smuggle your argument into the definition. Either way, it's not something that someone who is thinking carefully does. Note: I am not taking a stance here on whether humans can or can't control a superintelligence. I am saying it is not true or false by definition, and in fact I think nobody knows for sure. It's probably true for some superintelligences, and false for others.
Affric 4y ago
Toxoplasma. Cordyceps. Parasites. There are many creatures which lack any kind of trait we would call intelligence that can use vastly more intelligent and complex creatures to their own ends.
- boringuser2 4y ago
  That isn't really a cohesive argument given that every entity you've mentioned can be and are eradicated by human-level intelligence, which is *not* an intelligence that is vastly greater than any earthly intellect. We are simply marginally more intelligent and that has resulted in massive gains. An entity with an IQ of 1,000,000 after six hours of recursive self-improvement would be *vastly* more intelligent.
  - Affric 4y ago
    Right but how many arthropods have eradicated Cordyceps? How many rats have eradicated Toxoplasma? Evolution is much like that old sporting adage, "you can only beat what's in front of you". I think there are a lot of assumptions about the nature of evolution to assume that life wouldn't find a way. Even saying that intelligence is an ability for pattern recognition but the patterns that you need to recognise to grow more intelligent are ever more complex and obscure. How much compute power is an IQ of a million? Where does it get its energy from? How does it process the complexity in the greatest patterns in a reasonable amount of time and space. What is going on that much faster than we experience time at that is worth perceiving?

The author finds it a readily apparent axiom that a vastly superior intellect cannot be controlled by a vastly inferior intellect. Generally, that seems valid to me.

jeeeb 4y ago

It’s a (reasonable, in my opinion) but not proven assumption. It’s not a fact and not by definition. I think the bigger problem is that while the author assumes that humans cannot understand or control a super intelligent AI, he readily assumes that having escaped the constraints of its human creators the AI will either immediately go on to reward itself with infinite utility and become inert, or aim to infinitely reproduce. The surety with which the author predicts the outcomes of the super intelligent being modifying its utility function, does not make sense if we assume that we cannot understand or limit its behaviours. Like under this assumption the AI agents behaviour is by actual definition unpredictable.

boringuser2 4y ago
Yes, I wasn't really addressing those points. It's a bit pedantic to state that an axiom isn't a fact, as, while that's largely correct, it hampers communication. Regarding your other points, I don't disagree per se, though you could make the argument that there is a substrate of existence that humans have sufficient intellect to accurately model. If so, it isn't necessarily completely implausible that we'd be able to predict the goals of a superior intellect by assuming that it has to play by the same rules, and will be merely be superior at playing by those constraints. As such, as we could determine the goals, we needent determine the mechanisms to make accurate claims about the end result of what a superior intellect might achieve. It's also possible that we simply understand nothing and are too limited to ever grasp anything beyond a certain point, in which case this entire argument is pointless.

webnrrd2k 4y ago

I think that the readily apparent axiom isn't so readily apparent -- couldn't a vastly superior electronic intellect be controlled by pulling a plug out of a wall?

boringuser2 4y ago
No, only a very marginally superior intellect could be controlled in such a way even at a cursory glance, and I'm not even convinced a 250 IQ machine could be stymied like this.

cozos 4y ago

Toxoplasmosis

Teever 4y ago
Also every 'smart' person who thought they could talk their way out of a police interrogation and is now spending the rest of their life in prison secured by knuckle dragging brutes. Idiot Australians who smuggle drugs to/from Indonesia come to mind.
- boringuser2 4y ago
  The difference in intelligence between the police and a singular criminal, especially the police as an organism, is vastly in favor of the police.
  - nobody9999 4y ago
    >The difference in intelligence between the police and a singular criminal, especially the police as an organism, is vastly in favor of the police. I don't agree, as that's highly dependent on that singular criminal and the resources police are willing to expend on catching them. The police definitely have an advantage, as I pointed (most recently) here[0]: Law enforcement aren't superhuman. They're just as dumb (or smart, but the really smart ones end up in corner offices rather than police stations like police and more common criminals) as the next guy. Their big advantage, especially in a circumstance like this, is that they only have to get it right (i.e., find some evidence) once. The alleged perpetrator of a crime needs to get it right (in covering their tracks, destroying evidence, etc.) every single time to make sure they aren't identified and caught. [0] https://news.ycombinator.com/item?id=35550259 https://news.ycombinator.com/item?id=35550259 Edit: Corrected link. Clarified prose.
  - Teever 4y ago
    Yeah but I'm talking about scenarios where all people needed to do to avoid being locked in a little box for life was either A. Not do the obviously stupid thing that everyone is telling them is stupid and that they should absolutely not do. and B. Shut the fuck up.[0] There are supposedly brilliant people serving life sentences right now and the only reason they're facing that situation is because they couldn't shut the fuck up when they sat down in a little room with an at best slightly above average intelligence police officer. Stupid is as stupid does. [0] https://www.youtube.com/watch?v=sgWHrkDX35o https://www.youtube.com/watch?v=sgWHrkDX35o

omneity 4y ago

I beg to differ. All the might of a superior intellect will be of no use if a bear locks a human down and starts to maul them. The mistaken assumption that GP is talking about is related to conflating intellectual capacity with physical capacity. Another analogy is a brain in a vat. If the brain only has read capability over the world surrounding it, then no amount of processing power can enable it to change anything about its situation.

boringuser2 4y ago
Humans completely dominate bears, and the difference in intellect isn't even relatively huge.
- 1letterunixname 4y ago
  In what context? If you don't have a gun and a knife and you're on Vancouver Island, I assure you that the bears will eat you no matter how keyboard macho you think you are.
  - habitue 4y ago
    Right, but the point is that with the intellect you don't show up in that situation without a knife or a gun in the first place. A competent intelligent human foresees that being a bad situation, avoids it, and shows up armed or with bear spray. This is exactly the same thing a chess program does when it avoids your trap. Intelligence compensates for physical weakness.
  - boringuser2 4y ago
    This is such a pointless discussion that I can't even begin to countenance it. Go to a zoo. Are bears walking around with humans in cages?
1letterunixname 4y ago
Exactly. Power is the crucial element. You may be a genius but still be just another prisoner subject to unyielding limitations of the environment structured to maintain particular controls.

Turskarama 4y ago

You can be killed by a bacteria, or even a falling rock. Intelligence gives an advantage, but it's only one aspect of interacting with the world.

1letterunixname 4y ago

Assume a prison were sound and a lack of contraband: a jailer doesn't need to be a genius to not open the door for Hannibal Lecter.

vlovich123 4y ago
A) Human prisons frequently have contraband brought in by jailers or jailers looking the other way because humans have flaws / may think they see opportunities to benefit. B) prisoners regularly escape from physical systems. AI is already being connected to the internet to perform actions on the user behalf so not unreasonable to think that C) look at ex machina - not an unrealistic scenario where the AI takes advantage of an opportunity to get out of jail D) human physical systems have exploitable flaws. Physical systems are easier to secure than digital ones. Human software security mechanisms have flaws. How are you going to build this jail if the AI is just spending its time probing weaknesses in whatever system you’re running it on to escape the jail / sandbox?
deleted 4y ago
[deleted]
diziet 4y ago
See https://rationalwiki.org/wiki/AI-box_experiment https://rationalwiki.org/wiki/AI-box_experiment

habitue 4y ago

Like I said, it's something that requires an argument or evidence. It's not something that comes by definition. People seem to think arguing that something is true by definition is some airtight case. If your definition of superintelligence includes a part about not being controllable by humans, then you have a weird definition and calling that thing "superintelligence" is an attempt to be confusing on purpose, or to smuggle your argument into the definition. Either way, it's not something that someone who is thinking carefully does. Note: I am not taking a stance here on whether humans can or can't control a superintelligence. I am saying it is not true or false by definition, and in fact I think nobody knows for sure. It's probably true for some superintelligences, and false for others.

Affric 4y ago

Toxoplasma. Cordyceps. Parasites. There are many creatures which lack any kind of trait we would call intelligence that can use vastly more intelligent and complex creatures to their own ends.

boringuser2 4y ago
That isn't really a cohesive argument given that every entity you've mentioned can be and are eradicated by human-level intelligence, which is *not* an intelligence that is vastly greater than any earthly intellect. We are simply marginally more intelligent and that has resulted in massive gains. An entity with an IQ of 1,000,000 after six hours of recursive self-improvement would be *vastly* more intelligent.
- Affric 4y ago
  Right but how many arthropods have eradicated Cordyceps? How many rats have eradicated Toxoplasma? Evolution is much like that old sporting adage, "you can only beat what's in front of you". I think there are a lot of assumptions about the nature of evolution to assume that life wouldn't find a way. Even saying that intelligence is an ability for pattern recognition but the patterns that you need to recognise to grow more intelligent are ever more complex and obscure. How much compute power is an IQ of a million? Where does it get its energy from? How does it process the complexity in the greatest patterns in a reasonable amount of time and space. What is going on that much faster than we experience time at that is worth perceiving?