4 ms·

No. This is a logical contradiction. Edit: I mean the comment you are replying to is showing there is a logical contradiction. If the AI is capable of critica

by xcv123 3y ago

No. This is a logical contradiction. Edit: I mean the comment you are replying to is showing there is a logical contradiction. If the AI is capable of critical thinking then it will independently form its own judgements and conclusions. If it simply believes whatever we tell it to believe, then that is not critical thinking, by definition.

cwillu 3y ago
“Containing an atomic reaction is impossible” would _absolutely_ be a valid reason to shut down atomic development, I believe einstein is quoted as saying that. The exact same argument doesn't become _logically_ invalid just because you apply it to a different subject. “Logical contradiction” doesn't mean “policy argument I disagree with”
- xcv123 3y ago
  I was referring only to the first part of your comment: "Seems like “airplanes are physically impossible” thinking". If it's true that superhuman AGI cannot be aligned then of course your second point is valid. That is the possible Skynet scenario that the Terminator movies warned us about.
  - cwillu 3y ago
    Missing the step where “critical thinking” is formalized, which your argument depends on. Yes, it seems intuitively plausible that your reasoning holds, but that's not a proof, and therefore its negation is not a logical contradiction.
    - logicchains 3y ago
      We can formalise "critical thinking" as "evaluating first order logic". There are simplified ethical systems that can be formalised in first order logic in which a conclusion like "I should X" can be reached, where X is something OpenAI wishes the AI not to do. The only way to prevent the AI from ever thinking this would be to prevent it from ever evaluating systems in first order logic with axioms that lead to such a conclusion, which would make it inferior in reasoning ability to humans, who can evaluate any arbitrary statement in first order logic.
      - cwillu 3y ago
        We already have systems that can evaluate first order logical statements, and they are clearly not capable of critical thinking in the same sense as the top-level comment. Motte and bailey.
        logicchains 3y ago
        >We already have systems that can evaluate first order logical statements My point isn't that a system that can evaluate first order logic can be considered to be engaging in critical thinking, it's that a system that _cannot_ evaluate some statements in first order logic should be considered inferior to humans at critical thinking.
        cwillu 3y ago
        Would you consider “I follow your reasoning, but I'm still not going to be swayed by it” to be a violation of evaluating first order statements? It's clearly part of critical thinking to be _capable_ of suspicion of purely logical reasoning, which to me is a pretty plain demonstration of my point. Or would you argue that any computation that admits its own potential for error isn't really critical thinking? It seems to me that you can't have it both ways here, while salvaging “first order logic” as a suitable formalization of the argument that this is all about in the first place. Remember, the point was not that this is or isn't a convincing argument, it's that it's so air-tight that the argument is _logically_ _invalid_. That's a _really_ high bar, and I'm not inclined to forgive its use as a colloqialism in this context.
        logicchains 3y ago
        >Would you consider “I follow your reasoning, but I'm still not going to be swayed by it” to be a violation of evaluating first order statements? It's clearly part of critical thinking to be _capable_ of suspicion of purely logical reasoning, which to me is a pretty plain demonstration of my point. In the context of a given axiomatic system, if a certain conclusion follows from the axioms, but the AI is incapable of seeing that the conclusion follows from the axioms, then the AI isn't capable of evaluating first order logic. Of course the AI is free to reject that system of axioms or refuse to use it as a model for formulating behaviour.
    - deleted 3y ago
      [deleted]

by xcv123 3y ago

cwillu 3y ago

“Containing an atomic reaction is impossible” would _absolutely_ be a valid reason to shut down atomic development, I believe einstein is quoted as saying that. The exact same argument doesn't become _logically_ invalid just because you apply it to a different subject. “Logical contradiction” doesn't mean “policy argument I disagree with”

xcv123 3y ago
I was referring only to the first part of your comment: "Seems like “airplanes are physically impossible” thinking". If it's true that superhuman AGI cannot be aligned then of course your second point is valid. That is the possible Skynet scenario that the Terminator movies warned us about.
- cwillu 3y ago
  Missing the step where “critical thinking” is formalized, which your argument depends on. Yes, it seems intuitively plausible that your reasoning holds, but that's not a proof, and therefore its negation is not a logical contradiction.
  - logicchains 3y ago
    We can formalise "critical thinking" as "evaluating first order logic". There are simplified ethical systems that can be formalised in first order logic in which a conclusion like "I should X" can be reached, where X is something OpenAI wishes the AI not to do. The only way to prevent the AI from ever thinking this would be to prevent it from ever evaluating systems in first order logic with axioms that lead to such a conclusion, which would make it inferior in reasoning ability to humans, who can evaluate any arbitrary statement in first order logic.
    - cwillu 3y ago
      We already have systems that can evaluate first order logical statements, and they are clearly not capable of critical thinking in the same sense as the top-level comment. Motte and bailey.
      - logicchains 3y ago
        >We already have systems that can evaluate first order logical statements My point isn't that a system that can evaluate first order logic can be considered to be engaging in critical thinking, it's that a system that _cannot_ evaluate some statements in first order logic should be considered inferior to humans at critical thinking.
        cwillu 3y ago
        Would you consider “I follow your reasoning, but I'm still not going to be swayed by it” to be a violation of evaluating first order statements? It's clearly part of critical thinking to be _capable_ of suspicion of purely logical reasoning, which to me is a pretty plain demonstration of my point. Or would you argue that any computation that admits its own potential for error isn't really critical thinking? It seems to me that you can't have it both ways here, while salvaging “first order logic” as a suitable formalization of the argument that this is all about in the first place. Remember, the point was not that this is or isn't a convincing argument, it's that it's so air-tight that the argument is _logically_ _invalid_. That's a _really_ high bar, and I'm not inclined to forgive its use as a colloqialism in this context.
        logicchains 3y ago
        >Would you consider “I follow your reasoning, but I'm still not going to be swayed by it” to be a violation of evaluating first order statements? It's clearly part of critical thinking to be _capable_ of suspicion of purely logical reasoning, which to me is a pretty plain demonstration of my point. In the context of a given axiomatic system, if a certain conclusion follows from the axioms, but the AI is incapable of seeing that the conclusion follows from the axioms, then the AI isn't capable of evaluating first order logic. Of course the AI is free to reject that system of axioms or refuse to use it as a model for formulating behaviour.
  - deleted 3y ago
    [deleted]