4 ms·

Thank you for this excellent article, which I almost passed by until I remembered what the original XOR problem was (so, for me at least, the title is inverted

Thank you for this excellent article, which I almost passed by until I remembered what the original XOR problem was (so, for me at least, the title is inverted click-bait!) With regard to the question of whether human thought is Turing complete: on the one hand, we clearly can solve Turing-hard problems up to the limits of our memory (a constraint which is considerably expanded through written language), while on the other hand, we can and do make mistakes while attempting such problems. Personally, I feel the former is sufficient to answer the question in the affirmative, despite of the fact of the latter. The issue of Turing completeness is central to the Penrose-Lucas argument against AGI, as discussed at length in Roger Penrose's book "The Emperor's New Mind". Here, however, the claim is that human thought goes beyond Turing completeness: Gödel-unprovable results are provable by humans. While I feel that the complete argument against AGI fails on account of it equivocating over what it means to prove something, the above claim about the scope of human thought seems to be an empirical fact. I have no idea how this observation could help in AI development, but we know that one can prompt a current-generation LLM to produce sentences that humans would recognize as expressing the sort of thoughts that would be useful in solving the given problem (though whether the LLM will go on to produce the solution that is obvious to the humans, and consistent with its earlier productions, is hit-or-miss.) This suggests to me that some sort of self-prompting capability might be necessary (but not sufficient) for a general problem-solving ability.

shawntan 4y ago
On the memory limitation: Yeah, lack of an infinite tape is a real bummer. On some level I think we're some kind of bounded-tape automaton, which still makes the current limitations of Transformers pretty severe. (Yes, yes, prompting, CoT, and all that)
YeGoblynQueenne 4y ago
(...) Gödel-unprovable results are provable by humans. Are they? Certainly humans can recognise the undedicability of a "barber's paradox", like "this sentence is false", but that doesn't mean we can actually decide its truth value. We can spot undecidability, but not resolve it. Right? So what do you mean?
- mannykannot 4y ago
  I'm just being lazy and quoting a claim from Wikipedia's entry on the Penrose-Lucas argument, allegedly referencing "The Emperor's New Mind", but without any specific location within that book. It has been a while since I read it, and I may attempt to find a more precise summary of Penrose's claim (he has written about it extensively elsewhere, and so have many other people) but I can't do that immediately.
  - benlivengood 4y ago
    I think that Penrose basically thinks it's useful that humans can heuristically step up from a logic system S to S+"statement X is true" in an almost-grounded way, but in reality we are all actually constrained by Godel, Tarski, etc. We just trick ourselves into assuming we can inductively keep stepping up from S+"X is true" indefinitely for any practical problem we encounter, but that is actually an ungrounded assumption. What seems like an even better formal solution is something like logical induction[0] that works over something like probabilities instead of truth/falsehood. It has an existence proof but not a constructive formalization, afaik, but maybe some recent progress has been made. 0: https://arxiv.org/abs/1609.03543 https://arxiv.org/abs/1609.03543
    - YeGoblynQueenne 4y ago
      Thanks for the paper, I need to read that. Much of my work is on inductive logic. >> I think that Penrose basically thinks it's useful that humans can heuristically step up from a logic system S to S+"statement X is true" in an almost-grounded way, but in reality we are all actually constrained by Godel, Tarski, etc. We just trick ourselves into assuming we can inductively keep stepping up from S+"X is true" indefinitely for any practical problem we encounter, but that is actually an ungrounded assumption. What you describe sounds to me more like logical abduction, rather than induction. Unfortunately, it's difficult to find simple explanations of what abduction is. One very rough and operational categorisation of modes of inference, as deductive, inductive or abductive, is as follows. Deduction is reasoning (as in automated theorem proving or rule-based systems), induction is learning (as in machine learning) and abduction is reasoning under uncertainty (as in probabilistic inference). There's a lot of confusion between what is abduction and what is induction, and the definitions of those in terms of "facts" and "rules" (coming from Peirce, who originally named "abduction" at least) is, for me, not very helpful. Another colloquial explanation is that abduction is a form of drawing conclusions from observations that isn't sound, so that conclusions are not necessarily true. I think that's where the association with probabilistic reasoning comes from, and that's why I think it describes better the kind of induction the linked paper is talking about. Anyway I'll have to read that, thanks.

shawntan 4y ago

On the memory limitation: Yeah, lack of an infinite tape is a real bummer. On some level I think we're some kind of bounded-tape automaton, which still makes the current limitations of Transformers pretty severe. (Yes, yes, prompting, CoT, and all that)

YeGoblynQueenne 4y ago

(...) Gödel-unprovable results are provable by humans. Are they? Certainly humans can recognise the undedicability of a "barber's paradox", like "this sentence is false", but that doesn't mean we can actually decide its truth value. We can spot undecidability, but not resolve it. Right? So what do you mean?

mannykannot 4y ago
I'm just being lazy and quoting a claim from Wikipedia's entry on the Penrose-Lucas argument, allegedly referencing "The Emperor's New Mind", but without any specific location within that book. It has been a while since I read it, and I may attempt to find a more precise summary of Penrose's claim (he has written about it extensively elsewhere, and so have many other people) but I can't do that immediately.
- benlivengood 4y ago
  I think that Penrose basically thinks it's useful that humans can heuristically step up from a logic system S to S+"statement X is true" in an almost-grounded way, but in reality we are all actually constrained by Godel, Tarski, etc. We just trick ourselves into assuming we can inductively keep stepping up from S+"X is true" indefinitely for any practical problem we encounter, but that is actually an ungrounded assumption. What seems like an even better formal solution is something like logical induction[0] that works over something like probabilities instead of truth/falsehood. It has an existence proof but not a constructive formalization, afaik, but maybe some recent progress has been made. 0: https://arxiv.org/abs/1609.03543 https://arxiv.org/abs/1609.03543
  - YeGoblynQueenne 4y ago
    Thanks for the paper, I need to read that. Much of my work is on inductive logic. >> I think that Penrose basically thinks it's useful that humans can heuristically step up from a logic system S to S+"statement X is true" in an almost-grounded way, but in reality we are all actually constrained by Godel, Tarski, etc. We just trick ourselves into assuming we can inductively keep stepping up from S+"X is true" indefinitely for any practical problem we encounter, but that is actually an ungrounded assumption. What you describe sounds to me more like logical abduction, rather than induction. Unfortunately, it's difficult to find simple explanations of what abduction is. One very rough and operational categorisation of modes of inference, as deductive, inductive or abductive, is as follows. Deduction is reasoning (as in automated theorem proving or rule-based systems), induction is learning (as in machine learning) and abduction is reasoning under uncertainty (as in probabilistic inference). There's a lot of confusion between what is abduction and what is induction, and the definitions of those in terms of "facts" and "rules" (coming from Peirce, who originally named "abduction" at least) is, for me, not very helpful. Another colloquial explanation is that abduction is a form of drawing conclusions from observations that isn't sound, so that conclusions are not necessarily true. I think that's where the association with probabilistic reasoning comes from, and that's why I think it describes better the kind of induction the linked paper is talking about. Anyway I'll have to read that, thanks.