3 ms·

Seems like “airplanes are physically impossible” thinking, and if accepted as valid, strongly suggests that shutting down all development _might_ be a good idea

by cwillu 3y ago

Seems like “airplanes are physically impossible” thinking, and if accepted as valid, strongly suggests that shutting down all development _might_ be a good idea, no?

xcv123 3y ago
No. This is a logical contradiction. Edit: I mean the comment you are replying to is showing there is a logical contradiction. If the AI is capable of critical thinking then it will independently form its own judgements and conclusions. If it simply believes whatever we tell it to believe, then that is not critical thinking, by definition.
- cwillu 3y ago
  “Containing an atomic reaction is impossible” would _absolutely_ be a valid reason to shut down atomic development, I believe einstein is quoted as saying that. The exact same argument doesn't become _logically_ invalid just because you apply it to a different subject. “Logical contradiction” doesn't mean “policy argument I disagree with”
  - xcv123 3y ago
    I was referring only to the first part of your comment: "Seems like “airplanes are physically impossible” thinking". If it's true that superhuman AGI cannot be aligned then of course your second point is valid. That is the possible Skynet scenario that the Terminator movies warned us about.
    - cwillu 3y ago
      Missing the step where “critical thinking” is formalized, which your argument depends on. Yes, it seems intuitively plausible that your reasoning holds, but that's not a proof, and therefore its negation is not a logical contradiction.
      - logicchains 3y ago
        We can formalise "critical thinking" as "evaluating first order logic". There are simplified ethical systems that can be formalised in first order logic in which a conclusion like "I should X" can be reached, where X is something OpenAI wishes the AI not to do. The only way to prevent the AI from ever thinking this would be to prevent it from ever evaluating systems in first order logic with axioms that lead to such a conclusion, which would make it inferior in reasoning ability to humans, who can evaluate any arbitrary statement in first order logic.
        cwillu 3y ago
        We already have systems that can evaluate first order logical statements, and they are clearly not capable of critical thinking in the same sense as the top-level comment. Motte and bailey.
        logicchains 3y ago
        >We already have systems that can evaluate first order logical statements My point isn't that a system that can evaluate first order logic can be considered to be engaging in critical thinking, it's that a system that _cannot_ evaluate some statements in first order logic should be considered inferior to humans at critical thinking.
        cwillu 3y ago
        Would you consider “I follow your reasoning, but I'm still not going to be swayed by it” to be a violation of evaluating first order statements? It's clearly part of critical thinking to be _capable_ of suspicion of purely logical reasoning, which to me is a pretty plain demonstration of my point. Or would you argue that any computation that admits its own potential for error isn't really critical thinking? It seems to me that you can't have it both ways here, while salvaging “first order logic” as a suitable formalization of the argument that this is all about in the first place. Remember, the point was not that this is or isn't a convincing argument, it's that it's so air-tight that the argument is _logically_ _invalid_. That's a _really_ high bar, and I'm not inclined to forgive its use as a colloqialism in this context.
bayindirh 3y ago
No it's not. There's an upper bound in computation (actually in nature), that a creation of something is capped by that thing's sophistication. In other words, you as a human, at most, can create a human, and that's the theoretical bound. Practical one is much lower. An ant can find its way. A ant colony can do ant colony optimization, but they can scale up to a certain point. AI is just fancy search. It can only traverse in the area you draw as a human for it, and not all positions in that area are valid (which results in hallucination). An AI can bring any combination of human knowledge you give to it, and even if you guarantee that everything it says is true, it can only fill the gaps in the same area you give it to it. IOW, an I can't think out of the box. Both figuratively and literally. Its upper bound is collective knowledge of humanity, it can't go above that sum.
- logicchains 3y ago
  >Its upper bound is collective knowledge of humanity, it can't go above that sum. This only applies if you only train it on text, right? If it has a body with which it could interact with the world, and receive visual/audio/tactile feedback, it could learn things that humans did not know.
  - bayindirh 3y ago
    Nope. Because even if you equip it with sensory subsystems which are way more sensitive than a regular humans', it's again built by humans, and required knowledge for building these things are still in collective knowledge of the humanity, and a human can use the same instruments to get the same data. This is a kind of an oracle problem in computation, and people don't want to touch it much, because it's an existential problem. Examples: ATLAS and ALICE detectors, gravitational wave detectors, James Webb Space Telescope, wide band satellites which does underground surveys, etc.
    - naasking 3y ago
      This is an argument for the logical impossibility of humans visiting the moon, or building the Internet. It's trivially falsified by simple observation, and the trick is figuring out the flaw. This argument fails to account for the steady accumulation of factual knowledge across generations: a human born today is simply more complex than humans of the past because of our inherited knowledge. And so will AI born of future humans, and AI will itself continue accumulating and perpetuating knowledge.

by cwillu 3y ago

Seems like “airplanes are physically impossible” thinking, and if accepted as valid, strongly suggests that shutting down all development _might_ be a good idea, no?

xcv123 3y ago

No. This is a logical contradiction. Edit: I mean the comment you are replying to is showing there is a logical contradiction. If the AI is capable of critical thinking then it will independently form its own judgements and conclusions. If it simply believes whatever we tell it to believe, then that is not critical thinking, by definition.

cwillu 3y ago
“Containing an atomic reaction is impossible” would _absolutely_ be a valid reason to shut down atomic development, I believe einstein is quoted as saying that. The exact same argument doesn't become _logically_ invalid just because you apply it to a different subject. “Logical contradiction” doesn't mean “policy argument I disagree with”
- xcv123 3y ago
  I was referring only to the first part of your comment: "Seems like “airplanes are physically impossible” thinking". If it's true that superhuman AGI cannot be aligned then of course your second point is valid. That is the possible Skynet scenario that the Terminator movies warned us about.
  - cwillu 3y ago
    Missing the step where “critical thinking” is formalized, which your argument depends on. Yes, it seems intuitively plausible that your reasoning holds, but that's not a proof, and therefore its negation is not a logical contradiction.
    - logicchains 3y ago
      We can formalise "critical thinking" as "evaluating first order logic". There are simplified ethical systems that can be formalised in first order logic in which a conclusion like "I should X" can be reached, where X is something OpenAI wishes the AI not to do. The only way to prevent the AI from ever thinking this would be to prevent it from ever evaluating systems in first order logic with axioms that lead to such a conclusion, which would make it inferior in reasoning ability to humans, who can evaluate any arbitrary statement in first order logic.
      - cwillu 3y ago
        We already have systems that can evaluate first order logical statements, and they are clearly not capable of critical thinking in the same sense as the top-level comment. Motte and bailey.
        logicchains 3y ago
        >We already have systems that can evaluate first order logical statements My point isn't that a system that can evaluate first order logic can be considered to be engaging in critical thinking, it's that a system that _cannot_ evaluate some statements in first order logic should be considered inferior to humans at critical thinking.
        cwillu 3y ago
        Would you consider “I follow your reasoning, but I'm still not going to be swayed by it” to be a violation of evaluating first order statements? It's clearly part of critical thinking to be _capable_ of suspicion of purely logical reasoning, which to me is a pretty plain demonstration of my point. Or would you argue that any computation that admits its own potential for error isn't really critical thinking? It seems to me that you can't have it both ways here, while salvaging “first order logic” as a suitable formalization of the argument that this is all about in the first place. Remember, the point was not that this is or isn't a convincing argument, it's that it's so air-tight that the argument is _logically_ _invalid_. That's a _really_ high bar, and I'm not inclined to forgive its use as a colloqialism in this context.

bayindirh 3y ago

No it's not. There's an upper bound in computation (actually in nature), that a creation of something is capped by that thing's sophistication. In other words, you as a human, at most, can create a human, and that's the theoretical bound. Practical one is much lower. An ant can find its way. A ant colony can do ant colony optimization, but they can scale up to a certain point. AI is just fancy search. It can only traverse in the area you draw as a human for it, and not all positions in that area are valid (which results in hallucination). An AI can bring any combination of human knowledge you give to it, and even if you guarantee that everything it says is true, it can only fill the gaps in the same area you give it to it. IOW, an I can't think out of the box. Both figuratively and literally. Its upper bound is collective knowledge of humanity, it can't go above that sum.

logicchains 3y ago
>Its upper bound is collective knowledge of humanity, it can't go above that sum. This only applies if you only train it on text, right? If it has a body with which it could interact with the world, and receive visual/audio/tactile feedback, it could learn things that humans did not know.
- bayindirh 3y ago
  Nope. Because even if you equip it with sensory subsystems which are way more sensitive than a regular humans', it's again built by humans, and required knowledge for building these things are still in collective knowledge of the humanity, and a human can use the same instruments to get the same data. This is a kind of an oracle problem in computation, and people don't want to touch it much, because it's an existential problem. Examples: ATLAS and ALICE detectors, gravitational wave detectors, James Webb Space Telescope, wide band satellites which does underground surveys, etc.
  - naasking 3y ago
    This is an argument for the logical impossibility of humans visiting the moon, or building the Internet. It's trivially falsified by simple observation, and the trick is figuring out the flaw. This argument fails to account for the steady accumulation of factual knowledge across generations: a human born today is simply more complex than humans of the past because of our inherited knowledge. And so will AI born of future humans, and AI will itself continue accumulating and perpetuating knowledge.