15 ms·

The Lucas-Penrose Argument about Gödel's Theorem (2012)

64 points by lainon 9y ago

alberto_ol 9y ago
Solomon Feferman on Penrose’s Gödelian argument http://math.stanford.edu/~feferman/papers/penrose.pdf http://math.stanford.edu/~feferman/papers/penrose.pdf (PDF)
mannykannot 9y ago
Part of this discussion illustrates something about philosophy that once struck me as odd. Lucas carefully constructs a somewhat formal argument for his position, that would, if correct, rule out the computational theory of mind. One counterargument takes off from the point that Lucas' argument assumes that any computational model of a mind must be consistent, and depends on a mind being able to establish its own consistency. In response, we are told "Lucas thinks it is unlikely that an inconsistent machine could be an adequate representation of a mind." So now the careful, structured argument is reduced to a personal opinion? (The article includes some more specific variations on the theme by Lucas, but they all seem to me to be appeals to plausibility.) To me, this seems to leave Lucas with the burden of proof, and until he or someone else can counter this objection with something more than an opinion, the argument is dead in the water. I used to wonder why philosophy would hang an elaborate argument on personal opinions, but I have come to appreciate that whenever you speculate about matters where there is insufficient evidence to settle the issue, it always comes down to opinions, though this is often overlooked in the discussion.
- xamuel 9y ago
  Don't think of it as "a mind", but as "the knowledge held by a mind": one of the defining properties of knowledge is truth. (That's not intended as an endorsement of Lucas-Penrose. There are deeper flaws in the argument, like the assumption that the truth of the knowledge of the mind implies that the mind KNOWS its own truthfulness.)
  - mcguire 9y ago
    Justified true belief: https://en.m.wikipedia.org/wiki/Justified_true_belief https://en.m.wikipedia.org/wiki/Justified_true_belief The Gettier problem: https://en.m.wikipedia.org/wiki/Gettier_problem https://en.m.wikipedia.org/wiki/Gettier_problem
- Gormisdomai 9y ago
  You are doing good philosophy here. Though: I'd be careful to move from the fact that Lucas' argument comes down to a personal opinion to the fact that all philosophy comes down to personal opinions. I assume you already know about the concept of "commitment" in Philosophy, but I'll explain it briefly here in case anyone else comes across this comment. Sometimes while you can't show that a fact is categorically false (given some available evidence), you can show that people who believe that fact are committed to believing other facts. In this case Lucas' own argument has committed him to the "personal opinion" that an inconsistent machine cannot be an adequate representation of mind. But his belief isn't entirely unfalsifiable. If new evidence makes him give up this belief (e.g. a rise of paraconsistent logics) he'll have to give up his original belief along with it.
  - TheOtherHobbes 9y ago
    It's not about personal opinions. Or rather it is, but not in an obvious way. Human minds are clearly not consistent. Human minds are a loose collection of cobbled-together heuristics. Logic is one of those heuristics. It has limits because ultimately "logic" ends up being not a personal opinion, but a personal experience of the truth - or otherwise - of a proposition. Developed and trained minds have generated a heuristic of independent consensus about truths and experience, and strategies for increasing the correlation between the truth experience and the reality experience. Less developed untrained minds have a much more naive and subjective experience of "truth." But both work on the same foundation - a nebulous and indefinable subjective experience that is associated with truth, and is typically mistaken as the definition of it. It's impossible to "prove" truth, because the experience of deciding whether or not something is true is absolutely subjective. Any attempt to make an objective definition will always be filtered through that subjectivity and reduced to either having that experience or not having it. We're back to qualia, as usual. No one knows what "having an experience" really means. Until that question is answered, there are no tools for dealing with the rest. You can argue "But what about the scientific method, and mathematical proofs?" Same answer - they're practices that generate the truth experience in trained minds. They have a better predictive record and are better at finding consistent patterns than the disjointed heuristics of untrained minds. But they still don't - and can't - prove "truth" in any kind of ultimate definitive guaranteed-to-be-objective way. They certainly can't guarantee that the truth experience is never biased, contingent, limited, naive, circumstantial, or possibly even just plain wrong. The implications for AI are obvious. If a machine is better at finding consistent patterns, it will appear smarter than a human. It will still be smarter even if it makes errors that are obvious to a human, because at the same time it won't be making errors that humans do make, and which are similarly "obviously wrong" to the AI. It's the hit rate that matters, not idealised philosophical perfection.
- empath75 9y ago
  So take the argument as 'if you agree with me that an inconsistent machine is not a good representation of the mind', then the rest of my argument must follow. Think of it as analogous to all of the mathematical quasi-proofs that assume certain conjectures are true. They can often be valuable steps to an actual proof, as we saw with Fermat's theorem.
  - mannykannot 9y ago
    Thanks - that seems to be the idea I was groping for. In this particular case, though, it seems close to circularity - if the primary purpose of the argument is to refute the computational theory of mind, then ruling out a whole class of machines a priori looks to me like begging a chunk of the question - though I appreciate that it leads to a better-defined question.
- brudgers 9y ago
  Philosophy covers a lot of ground (e.g. the subject matter that is generally called "science" today was once called "natural philosophy" and much of the contemporary science of psychology has its roots in the late 19th and early 20th century American philosophical movement called "pragmatism" (or later "pargmaticism" by CS Pierce in order that the term would be so ugly as to not be misappropriated)). And there are people who argue that psychology is not really a science and usually more because people will argue about anything than out of deep objectivity that is applied to their own position. Anyway, there is a manifold in philosophic space in which proving things to win arguments matters and a manifold where torturing philosophy with the scientific method matters and a manifold in which there is doubt about the external world and one in which confusion arises from lack of rigor regarding language and one in which the goal is to prove the existence of gods and a manifold of other manifolds. We all get to take our pick because philosophy is a big tent that is big enough that a philosopher is free to wander among the lion tamers and clowns and acrobats and look at what is interesting. Radical skepticism is a useful intellectual tool, but so are Platonic idealism and deconstruction and Kantian metaphysics. Anyone who thinks they've got it all correct is probably ignoring human fallibility (but maybe not).
- naasking 9y ago
  > One counterargument takes off from the point that Lucas' argument assumes that any computational model of a mind must be consistent, and depends on a mind being able to establish its own consistency. Which would obviously be false. There are clearly true theorems we will never be able to prove. Simply because we can prove many theorems that simpler formal systems cannot, entails nothing about our logical consistency or completeness.
  - posterboy 9y ago
    There's an argument to be made that these logical constructs you mention might just nkt be meaningful in the real world. Everything has to come down to zeroth order logic, eventually, I guess.
- mcguire 9y ago
  At the risk of having my junior philosopher's secret decoder ring taken away, the true definition of philosophy is an elaborate logical structure built to decorate your previously existing, unreasoned prejudices.
  - newdayrising 9y ago
    Only someone that knows very little of philosophy would say such a thing. Oh, and I doubt you'd appreciate someone similarly pissing on your profession.
- dpierce9 9y ago
  Philosophers are concerned with two points: soundness and validity. A valid argument is one where the conclusion follows from the premises. The way to think about this is 'If I assume the premises are true, is my conclusion likely to be true'. A sound argument is a valid argument with true premises. Now valid arguments can be interesting in their own right even if their premises aren't true. Given this, the proper way to read Lucas's opinion is: I am taking it as given that the computational theory of mind requires a consistent system of inference rules + math; if that isn't true, my argument might be unsound but not invalid (and its validity is interesting); but I also think it is true for x,y,z reasons (though these additional reasons do not constitute a proof). In other words, the Lucas-Penrose argument is interesting from both a truth-of-premises and validity perspective which is far from dead in the water.
lainon 9y ago
http://www.cs.bham.ac.uk/~mmk/papers/05-KI.html http://www.cs.bham.ac.uk/~mmk/papers/05-KI.html (Why is the Lucas-Penrose Argument Invalid?)
- fiatjaf 9y ago
  > The matter is confused by the prima facie paradoxical fact that Gödel proved the truth of the sentence that "This sentence is not provable." Gödel didn't prove that. He saw that it was true.
manyoso 9y ago
Most interesting part was suggestion that Gödel himself agreed with Penrose argument against Strong AI. I think this will be highly contentious.
- manyoso 9y ago
  Here is the quote from Gödel supposedly supporting this: "So the following disjunctive conclusion is inevitable: Either mathematics is incompletable in this sense, that its evident axioms can never be comprised in a finite rule, that is to say, the human mind (even within the realm of pure mathematics) infinitely surpasses the powers of any finite machine, or else there exist absolutely unsolvable diophantine problems of the type specified" . . . (Gödel 1995: 310).
- serhei 9y ago
  Wikipedia describes Godel as a nonreligious theist who believed in the existence of an afterlife. In that context, it is not surprising that he would draw far-reaching conclusions that a human being's behaviour cannot be described by a formal system.
- badcede 9y ago
  It won't be contentious among people who know much about Gödel. He spent his later years working on things like proofs of the existence of God. Like many geniuses he went far beyond what his profession was comfortable with.
empath75 9y ago
What if the human mind is computational, but isn't one unified system. I think most neurologists today believe the mind is composed of many loosely coupled systems communicating with each other?
- serhei 9y ago
  A collection of loosely coupled computational systems still forms a computational system, not some fundamentally different entity.
serhei 9y ago
Both the Lucas-Penrose argument and the counterarguments seem to assume that consciousness is a phenomenon within the laws of physics which arises from more fundamental laws (e.g. chemistry or quantum mechanics), which is (to me) a completely non-obvious assumption. For example, I perceive my own thoughts rather than other people's. This implies that my own thoughts would have an additional physical property, that of being perceived by me, which the thoughts of other people lack. However, according to all up-to-date theories of physics and neurology, there is no difference between my own thoughts and other people's that would explain the presence of this additional property.
- bsaunder 9y ago
  Aren't your own thoughts similar to a CPU's registers or cache? One's thoughts seem to occur within the context of the brain and the billions of neighboring (to the ones inducing the current "thought") neurons. Its as obvious to me that you can't perceive another's thoughts just as surely as one CPU can't "perceive" the registers of another. The perception of ones own thoughts seems very physical, as is the equally obvious (to me) explanation that you can't directly perceive the thoughts of another.
- lucozade 9y ago
  For you to perceive your thoughts doesn't require different properties to other people's thoughts. Just different values to common properties. It's not clear why you would expect this to be inherently different from any other neurological response. For example, if I stub my toe, are you suggesting that science doesn't have an answer to why you don't feel it?
  - mcguire 9y ago
    If I turn the light on in my closet with the door closed, can anyone explain why you can't see it?
  - serhei 9y ago
    I am suggesting that the fact that I (the perceiving subject) am myself (the biological entity) rather than you (a different biological entity, but ostensibly subject to the same laws) is (a) an objective fact (b) not within the domain of science. I do not expect this fact to map to any observable physical properties.
Udo 9y ago
> Now suppose that we construct the Gödel sentence for this formal system. Since the Gödel sentence cannot be proven in the system, the machine will be unable to produce this sentence as a truth of arithmetic. However, a human can look and see that the Gödel sentence is true. In other words, there is at least one thing that a human mind can do that no machine can. Therefore, “a machine cannot be a complete and adequate model of the mind” (Lucas 1961: 113). In short, the human mind is not a machine. That is not a difference between human and machine. It's a rhetorical trick at best, and an unreasonable appeal to mysticism at worst. A human mind looking at the Gödel sentence while strictly adhering to the same formal structure could also not see that the Gödel sentence is true. Conversely, a machine that is allowed to operate outside of this postulated formalism could perfectly well make an inference that the Gödel sentence is true. The Lucas-Penrose argument boils down to a questionable attempt at a logic bomb. So by first tying the machine's hands, they argue that the machine's hand's are indeed tied, and therefore humans - who in this example have been given free rein - can do something that you explicitly disallowed machines from doing.
- scandox 9y ago
  What formalism, if any, is the human mind subject to? Isn't our current conception of a machine that it will bound to some formalism in terms of reasoning? Are we so bound?
  - naasking 9y ago
    > What formalism, if any, is the human mind subject to? There could be many that could answer some such questions, paraconsistent and inconsistent for example.
  - Udo 9y ago
    That's a loaded argument. Human biology, and by extension human minds, follow a myriad of rules that are a result of biological processes being played out. Likewise "machines" also follow the rules laid out by their physical design. There is no fundamental difference in this regard, there is no new set of information-theoretical rules that comes into play when you switch to a carbon-based machinery. > What formalism, if any, is the human mind subject to? In the context of this argument, the mind of the human mathematician would be subject to the rules of the formal system. Since Gödel we know that such a formal system cannot be proven true inside the system itself. However, an entity that views the system from the outside can potentially see it's true. There is no fundamental difference between a human mind and a hypothetical machine mind as far as the capability to come to the same conclusion is concerned. That's why Lucas took special care to specify upfront that the machine mind is prohibited from coming to that conclusion, by confining it to live within the formal system only. Yes, their argument is really that circular. > Isn't our current conception of a machine that it will bound to some formalism in terms of reasoning? Are we so bound? We are not, and neither would a generally intelligent machine. The burden of proof is on Lucas/Penrose to show that a generally intelligent machine would still be bound to that formalism, at which point it would cease to be considered generally intelligent. They seem to argue that their postulate is true because the individual components of such a machine are bound by those rules, but then again so are the individual components of our brains.
falsedan 9y ago
> When we notice an inconsistency within ourselves, we generally “eschew” it, whereas “if we really were inconsistent machines, we should remain content with our inconsistencies, and would happily affirm both halves of a contradiction” I feel that only philosophers & logicians are this black-and-white. There are many people content to occupy a midpoint on the spectrum of "This fact is true"..."This fact is false"; some even will accept a range! One fact I think we can all agree on is that philosophers are: old; white; dead; men; and wrong.
- falsedan 9y ago
  Love too get downvotes with no comments/feedback. Is this about the dead white men? I take it back. I didn't mean it. I changed my mind. I never meant for it to be taken this way. of all the possible interpretations of my words, I intended you to adopt the one where they were both witty and erudite. It's your fault you picked the wrong one. I stand by my words. I am right. Only I am right. I have seen into the heart of Man and I know its true meaning.
Chronos 9y ago
Ugh, this again. 1. Let's suppose for sake of argument that humans really can see the inherent truth of "Peano Arithmetic is consistent". That doesn't mean humans violate Gödel's Incompleteness Theorem: it could just mean that humans use axioms stronger than PA. 2. Gödel's Incompleteness Theorem only applies to systems that are perfectly logically consistent. Not sure how Penrose didn't notice, but humans... aren't. 3. When scientists proposed Quantum Mechanics as a replacement for Classical Mechanics, it was on them to explain how Quantum Mechanics simplified to Classical Mechanics in the common case. "Penrose Mechanics" is an even more radical departure — especially from a physics of computation standpoint, as Penrose Mechanics by definition would allow solving at least some of the problems in (ALL - R) in ~polynomial time. Penrose needs to explain how Penrose Mechanics reduces to Quantum Mechanics in the common case. 4. Penrose proposes that (a) there exist new physics, (b) that evolution has learned to computationally exploit the new physics via microtubules, and yet (c) that humans are the only lineage to make use of this feature of microtubules, even though microtubules are found in all eukaryotic cells (from mushrooms to amoebae). From a predator-prey standpoint alone, it would seemingly be a huge evolutionary advantage to be able to compute NP or R functions in polynomial time. (That ability is not _strictly_ implied by Penrose Mechanics, but it's a very likely consequence.) Penrose needs to explain why only humans are taking advantage of the computational power of microtubules, when microtubules have existed for billions of years and across millions of species. (TL;DR: It's the pineal gland all over again.)
- manyoso 9y ago
  #1 Please show how to program a Turing machine to use "axioms stronger than PA" #2 Please show how to program a Turing machine so that it is not logically consistent at a fundamental level? #3 Are you questioning the viability of objective collapse theories in general? To date we have no experimental evidence of said theories, but I don't think anyone has suggested they are inconsistent with QM. Are you? If so, please show how... #4 Show where Penrose has said humans are only species to make use of microtubules and your claims regarding Penrose' theory of objective collapse and having staggering new computational speedups are ... without any evidence. Please provide some. The tenor of your comment suggests you find Penrose to be an idiot and missing obvious problems. Consider that perhaps you are the one misunderstanding what he is proposing and that he is not, in fact, an idiot.
mcguire 9y ago
1. If Lucas and Penrose are right, there are statements which are unequivocally true, but which cannot be proved in any known formal system. It should be possible to exhibit a few. (Extra credit if they avoid theology.) If the only such statements are artificial Gödelian sentences, I'm afraid I'm...uninterested, in the same way Lucas is uninterested in inconsistent humans. 2. Lucas and Penrose assert that some entities are significantly more powerful than others. To my knowledge, Lucas describes no mechanism for this. (To his credit, Penrose does, but, to my knowledge, the Bell inequality among other things means quantum mechanics is fundamentally random. That's not helpful.) What could be the nature of such a mechanism? If it's material, it should be possible to include it in future machines. If it's not, we are in the realm of theology again. 3. So humans are intelligent and intelligent-seeming machines only simulate intelligence. If the difference is not common, obvious, and readily meaningful, then where is the distinction? ("So what? I can't write an opera either.")
- manyoso 9y ago
  Sigh, they don't claim that. You are also arguing with straw men. I wish people would read what Penrose actually argues instead of just assuming he is an idiot (one of the most brilliant minds of this or any century) because he argues against Strong AI. There are good refutations of Penrose' argument, but I've seen none of them here. Just a bunch of bandwagon hackers assuming Penrose doesn't know what he is talking about and then putting up lazy straw man takedowns.
  - mcguire 9y ago
    Don't claim what? That there are statements that are known to be true but which cannot be proved, or that the human mind is strictly stronger than a machine? Edit: Or perhaps that Penrose is not claiming that QM is some kind of super secret sauce that makes multicellular life intrinsically stronger than computing machines?
    - manyoso 9y ago
      "...statements which are unequivocally true, but which cannot be proved in any known formal system." That.
lisper 9y ago
Refuting Lucas is very simple: Godel sentences can only be constructed for circumscribed systems, i.e. a system where you have written down all of the axioms and inference rules. So the only way to circumscribe a human brain is to isolate it from its environment. As soon as you let it communicate with (say) another human brain, you can no longer Godelize it because that brain is not the system any more, it's the two communicating brains working together. That system might have a Godel sentence of its own, but it's not the same as the Godel sentence of individual brains. So if I can put you in an isolation tank and model your brain, then I can in fact (at least in principle) construct your Godel sentence and know for certain that you will never come to the realization that this outrageously complicated mess I've just constructed (the one that says of itself that you will never come to see its truth) is true, and I can feel smug about the fact that I can see that this is true and you never will. But here's the kicker: I can never gloat about my superiority to you, because as soon as I do you are no longer in isolation. You are communicating with me, and the sentence I've take great pains to construct is no longer your Godel sentence. It is the Godel sentence of you in isolation, but as soon as I tell you your Godel sentence -- indeed as soon as I (or anyone else) say anything to you, my construction falls apart and I have to start over. Ironically, far from proving that we are not machines, Lucas's argument actually goes a long way towards demonstrating that we are, because coming up with examples of things that we interacting brains know to be true that a particular isolated brain will (with extremely high probability) never come to realize is not difficult. The four-color theorem, for example, is probably beyond any single isolated human brain. Indeed, the very argument you are reading now is probably beyond any single human brain because the brain that's writing it would never have come up with it had Godel and Lucas (and Turing) not paved the way.
- fiatjaf 9y ago
  That was a good argument. Have you formulated it by yourself?
  - lisper 9y ago
    If I said "yes" I would be refuting my own argument :-) I composed the words in the above comment, but I don't actually know if the substance of that argument is original with me. It is heavily based on things I've learned in other places, but I don't have any specific recollection of having seen the argument being framed in exactly that way before. It's possible I invented it, but I doubt it. It's not a particularly deep insight, a minor variation on a deeper theme, that "meta-Godelization" is not algorithmic, a fact that I first learned by reading Hofstadter's "Godel, Escher, Bach." I don't actually know what the original source of that result is (I should probably go find out). But I'll happily take credit for the pedagogy if it somehow turns out I got there first.
Crito 9y ago
Franky it's irrelevant. Penrose's insistence that the human mind isn't algorithmic in the Turing sense is born from his human insecurities. But if the human mind isn't bound in such a way, than any man-made computer need not be bound either. Penrose's argument falls short of being a proper dualist argument. Furthermore Roger Penrose has not proven his claims, and they are not supported by mainstream neuroscientists. He appeals to quantum woo that he pulled out of his ass.
- manyoso 9y ago
  That you disparage Penrose and cite his mental motivation is likely much more relevant to your own insecurities than anything else.
- dang 9y ago
  > born from his human insecurities > woo that he pulled out of his ass It's not ok to argue like this on HN, so please don't. It lowers the quality of discussion in its own right and sets off a downward spiral from others. Edit: Actually, we've banned your account. Your comment history shows that you've broken HN's civility rule so egregiously, and so repeatedly, that we should have banned it long time ago. If you don't want to be banned on HN, you're welcome to email hn@ycombinator.com and promise to follow the rules scrupulously in the future. We detached this subthread from https://news.ycombinator.com/item?id=14446064 https://news.ycombinator.com/item?id=14446064 and marked it off-topic.
ccvannorman 9y ago
Site is down. Anyone have a mirror?
yongjik 9y ago
Well, I don't think our brains work by applying "logic" when we think about Gödel's theorem (or anything else). We evolved to do pattern matching, so each of us spends decades to learn more and more sophisticated pattern matching techniques, starting from mom and dad's faces to counting things to multiplication tables to, finally, understanding Gödel's theorem (for some people). In light of this, we can reinterpret "However, a human can look and see that the Gödel sentence is true" as: When you take the best pattern matching machine nature has produced, train it with distilled essentials of centuries of mathematical geniuses, and present it with the Gödel sentence, the pattern matcher strongly signals "This looks true!" When viewed this way, it looks less like an inherently human-specific achievement, but more like exactly how ML algorithms work. The only difference is that ML cannot quite pull it off, so far.
enugu 9y ago
Penrose argument, while it can be challenged, is actually much more careful than dismissals here suggest. For instance people talking about 'humans are not consistent', we are talking about effort directed into a well specified formal domain, and after long term reflection. So, of course, people make mistakes in mathematics all the time, but we are talking about consistency after intense review over long period of time. We write a program, and cant find the bug even if we look at it for a google years. To the claim, 'Godel statement is not interesting relative to other statements', the goal is that to demonstrate non-algorithmic nature of some human activity, which itself would be a big deal if true. A more qualified version of the LP claim can be expressed as follows 'One of the following is true 1) The activity of humans in some formal domain(like mathematics) is inconsistent in a deep way(not routine mistakes). 2)Humans are reflective consistent, algorithmic but will never have access to the source code(a claim about limits about brain research). 3) We are reflective consistent, can access source code, but wont be able to see why the source itself is consistent.'

64 points by lainon 9y ago

Solomon Feferman on Penrose’s Gödelian argument http://math.stanford.edu/~feferman/papers/penrose.pdf http://math.stanford.edu/~feferman/papers/penrose.pdf (PDF)

mannykannot 9y ago

Part of this discussion illustrates something about philosophy that once struck me as odd. Lucas carefully constructs a somewhat formal argument for his position, that would, if correct, rule out the computational theory of mind. One counterargument takes off from the point that Lucas' argument assumes that any computational model of a mind must be consistent, and depends on a mind being able to establish its own consistency. In response, we are told "Lucas thinks it is unlikely that an inconsistent machine could be an adequate representation of a mind." So now the careful, structured argument is reduced to a personal opinion? (The article includes some more specific variations on the theme by Lucas, but they all seem to me to be appeals to plausibility.) To me, this seems to leave Lucas with the burden of proof, and until he or someone else can counter this objection with something more than an opinion, the argument is dead in the water. I used to wonder why philosophy would hang an elaborate argument on personal opinions, but I have come to appreciate that whenever you speculate about matters where there is insufficient evidence to settle the issue, it always comes down to opinions, though this is often overlooked in the discussion.

xamuel 9y ago
Don't think of it as "a mind", but as "the knowledge held by a mind": one of the defining properties of knowledge is truth. (That's not intended as an endorsement of Lucas-Penrose. There are deeper flaws in the argument, like the assumption that the truth of the knowledge of the mind implies that the mind KNOWS its own truthfulness.)
- mcguire 9y ago
  Justified true belief: https://en.m.wikipedia.org/wiki/Justified_true_belief https://en.m.wikipedia.org/wiki/Justified_true_belief The Gettier problem: https://en.m.wikipedia.org/wiki/Gettier_problem https://en.m.wikipedia.org/wiki/Gettier_problem
Gormisdomai 9y ago
You are doing good philosophy here. Though: I'd be careful to move from the fact that Lucas' argument comes down to a personal opinion to the fact that all philosophy comes down to personal opinions. I assume you already know about the concept of "commitment" in Philosophy, but I'll explain it briefly here in case anyone else comes across this comment. Sometimes while you can't show that a fact is categorically false (given some available evidence), you can show that people who believe that fact are committed to believing other facts. In this case Lucas' own argument has committed him to the "personal opinion" that an inconsistent machine cannot be an adequate representation of mind. But his belief isn't entirely unfalsifiable. If new evidence makes him give up this belief (e.g. a rise of paraconsistent logics) he'll have to give up his original belief along with it.
- TheOtherHobbes 9y ago
  It's not about personal opinions. Or rather it is, but not in an obvious way. Human minds are clearly not consistent. Human minds are a loose collection of cobbled-together heuristics. Logic is one of those heuristics. It has limits because ultimately "logic" ends up being not a personal opinion, but a personal experience of the truth - or otherwise - of a proposition. Developed and trained minds have generated a heuristic of independent consensus about truths and experience, and strategies for increasing the correlation between the truth experience and the reality experience. Less developed untrained minds have a much more naive and subjective experience of "truth." But both work on the same foundation - a nebulous and indefinable subjective experience that is associated with truth, and is typically mistaken as the definition of it. It's impossible to "prove" truth, because the experience of deciding whether or not something is true is absolutely subjective. Any attempt to make an objective definition will always be filtered through that subjectivity and reduced to either having that experience or not having it. We're back to qualia, as usual. No one knows what "having an experience" really means. Until that question is answered, there are no tools for dealing with the rest. You can argue "But what about the scientific method, and mathematical proofs?" Same answer - they're practices that generate the truth experience in trained minds. They have a better predictive record and are better at finding consistent patterns than the disjointed heuristics of untrained minds. But they still don't - and can't - prove "truth" in any kind of ultimate definitive guaranteed-to-be-objective way. They certainly can't guarantee that the truth experience is never biased, contingent, limited, naive, circumstantial, or possibly even just plain wrong. The implications for AI are obvious. If a machine is better at finding consistent patterns, it will appear smarter than a human. It will still be smarter even if it makes errors that are obvious to a human, because at the same time it won't be making errors that humans do make, and which are similarly "obviously wrong" to the AI. It's the hit rate that matters, not idealised philosophical perfection.
empath75 9y ago
So take the argument as 'if you agree with me that an inconsistent machine is not a good representation of the mind', then the rest of my argument must follow. Think of it as analogous to all of the mathematical quasi-proofs that assume certain conjectures are true. They can often be valuable steps to an actual proof, as we saw with Fermat's theorem.
- mannykannot 9y ago
  Thanks - that seems to be the idea I was groping for. In this particular case, though, it seems close to circularity - if the primary purpose of the argument is to refute the computational theory of mind, then ruling out a whole class of machines a priori looks to me like begging a chunk of the question - though I appreciate that it leads to a better-defined question.
brudgers 9y ago
Philosophy covers a lot of ground (e.g. the subject matter that is generally called "science" today was once called "natural philosophy" and much of the contemporary science of psychology has its roots in the late 19th and early 20th century American philosophical movement called "pragmatism" (or later "pargmaticism" by CS Pierce in order that the term would be so ugly as to not be misappropriated)). And there are people who argue that psychology is not really a science and usually more because people will argue about anything than out of deep objectivity that is applied to their own position. Anyway, there is a manifold in philosophic space in which proving things to win arguments matters and a manifold where torturing philosophy with the scientific method matters and a manifold in which there is doubt about the external world and one in which confusion arises from lack of rigor regarding language and one in which the goal is to prove the existence of gods and a manifold of other manifolds. We all get to take our pick because philosophy is a big tent that is big enough that a philosopher is free to wander among the lion tamers and clowns and acrobats and look at what is interesting. Radical skepticism is a useful intellectual tool, but so are Platonic idealism and deconstruction and Kantian metaphysics. Anyone who thinks they've got it all correct is probably ignoring human fallibility (but maybe not).
naasking 9y ago
> One counterargument takes off from the point that Lucas' argument assumes that any computational model of a mind must be consistent, and depends on a mind being able to establish its own consistency. Which would obviously be false. There are clearly true theorems we will never be able to prove. Simply because we can prove many theorems that simpler formal systems cannot, entails nothing about our logical consistency or completeness.
- posterboy 9y ago
  There's an argument to be made that these logical constructs you mention might just nkt be meaningful in the real world. Everything has to come down to zeroth order logic, eventually, I guess.
mcguire 9y ago
At the risk of having my junior philosopher's secret decoder ring taken away, the true definition of philosophy is an elaborate logical structure built to decorate your previously existing, unreasoned prejudices.
- newdayrising 9y ago
  Only someone that knows very little of philosophy would say such a thing. Oh, and I doubt you'd appreciate someone similarly pissing on your profession.
dpierce9 9y ago
Philosophers are concerned with two points: soundness and validity. A valid argument is one where the conclusion follows from the premises. The way to think about this is 'If I assume the premises are true, is my conclusion likely to be true'. A sound argument is a valid argument with true premises. Now valid arguments can be interesting in their own right even if their premises aren't true. Given this, the proper way to read Lucas's opinion is: I am taking it as given that the computational theory of mind requires a consistent system of inference rules + math; if that isn't true, my argument might be unsound but not invalid (and its validity is interesting); but I also think it is true for x,y,z reasons (though these additional reasons do not constitute a proof). In other words, the Lucas-Penrose argument is interesting from both a truth-of-premises and validity perspective which is far from dead in the water.

lainon 9y ago

http://www.cs.bham.ac.uk/~mmk/papers/05-KI.html http://www.cs.bham.ac.uk/~mmk/papers/05-KI.html (Why is the Lucas-Penrose Argument Invalid?)

fiatjaf 9y ago
> The matter is confused by the prima facie paradoxical fact that Gödel proved the truth of the sentence that "This sentence is not provable." Gödel didn't prove that. He saw that it was true.

manyoso 9y ago

Most interesting part was suggestion that Gödel himself agreed with Penrose argument against Strong AI. I think this will be highly contentious.

manyoso 9y ago
Here is the quote from Gödel supposedly supporting this: "So the following disjunctive conclusion is inevitable: Either mathematics is incompletable in this sense, that its evident axioms can never be comprised in a finite rule, that is to say, the human mind (even within the realm of pure mathematics) infinitely surpasses the powers of any finite machine, or else there exist absolutely unsolvable diophantine problems of the type specified" . . . (Gödel 1995: 310).
serhei 9y ago
Wikipedia describes Godel as a nonreligious theist who believed in the existence of an afterlife. In that context, it is not surprising that he would draw far-reaching conclusions that a human being's behaviour cannot be described by a formal system.
badcede 9y ago
It won't be contentious among people who know much about Gödel. He spent his later years working on things like proofs of the existence of God. Like many geniuses he went far beyond what his profession was comfortable with.

empath75 9y ago

What if the human mind is computational, but isn't one unified system. I think most neurologists today believe the mind is composed of many loosely coupled systems communicating with each other?

serhei 9y ago
A collection of loosely coupled computational systems still forms a computational system, not some fundamentally different entity.

serhei 9y ago

Both the Lucas-Penrose argument and the counterarguments seem to assume that consciousness is a phenomenon within the laws of physics which arises from more fundamental laws (e.g. chemistry or quantum mechanics), which is (to me) a completely non-obvious assumption. For example, I perceive my own thoughts rather than other people's. This implies that my own thoughts would have an additional physical property, that of being perceived by me, which the thoughts of other people lack. However, according to all up-to-date theories of physics and neurology, there is no difference between my own thoughts and other people's that would explain the presence of this additional property.

bsaunder 9y ago
Aren't your own thoughts similar to a CPU's registers or cache? One's thoughts seem to occur within the context of the brain and the billions of neighboring (to the ones inducing the current "thought") neurons. Its as obvious to me that you can't perceive another's thoughts just as surely as one CPU can't "perceive" the registers of another. The perception of ones own thoughts seems very physical, as is the equally obvious (to me) explanation that you can't directly perceive the thoughts of another.
lucozade 9y ago
For you to perceive your thoughts doesn't require different properties to other people's thoughts. Just different values to common properties. It's not clear why you would expect this to be inherently different from any other neurological response. For example, if I stub my toe, are you suggesting that science doesn't have an answer to why you don't feel it?
- mcguire 9y ago
  If I turn the light on in my closet with the door closed, can anyone explain why you can't see it?
- serhei 9y ago
  I am suggesting that the fact that I (the perceiving subject) am myself (the biological entity) rather than you (a different biological entity, but ostensibly subject to the same laws) is (a) an objective fact (b) not within the domain of science. I do not expect this fact to map to any observable physical properties.

Udo 9y ago

> Now suppose that we construct the Gödel sentence for this formal system. Since the Gödel sentence cannot be proven in the system, the machine will be unable to produce this sentence as a truth of arithmetic. However, a human can look and see that the Gödel sentence is true. In other words, there is at least one thing that a human mind can do that no machine can. Therefore, “a machine cannot be a complete and adequate model of the mind” (Lucas 1961: 113). In short, the human mind is not a machine. That is not a difference between human and machine. It's a rhetorical trick at best, and an unreasonable appeal to mysticism at worst. A human mind looking at the Gödel sentence while strictly adhering to the same formal structure could also not see that the Gödel sentence is true. Conversely, a machine that is allowed to operate outside of this postulated formalism could perfectly well make an inference that the Gödel sentence is true. The Lucas-Penrose argument boils down to a questionable attempt at a logic bomb. So by first tying the machine's hands, they argue that the machine's hand's are indeed tied, and therefore humans - who in this example have been given free rein - can do something that you explicitly disallowed machines from doing.

scandox 9y ago
What formalism, if any, is the human mind subject to? Isn't our current conception of a machine that it will bound to some formalism in terms of reasoning? Are we so bound?
- naasking 9y ago
  > What formalism, if any, is the human mind subject to? There could be many that could answer some such questions, paraconsistent and inconsistent for example.
- Udo 9y ago
  That's a loaded argument. Human biology, and by extension human minds, follow a myriad of rules that are a result of biological processes being played out. Likewise "machines" also follow the rules laid out by their physical design. There is no fundamental difference in this regard, there is no new set of information-theoretical rules that comes into play when you switch to a carbon-based machinery. > What formalism, if any, is the human mind subject to? In the context of this argument, the mind of the human mathematician would be subject to the rules of the formal system. Since Gödel we know that such a formal system cannot be proven true inside the system itself. However, an entity that views the system from the outside can potentially see it's true. There is no fundamental difference between a human mind and a hypothetical machine mind as far as the capability to come to the same conclusion is concerned. That's why Lucas took special care to specify upfront that the machine mind is prohibited from coming to that conclusion, by confining it to live within the formal system only. Yes, their argument is really that circular. > Isn't our current conception of a machine that it will bound to some formalism in terms of reasoning? Are we so bound? We are not, and neither would a generally intelligent machine. The burden of proof is on Lucas/Penrose to show that a generally intelligent machine would still be bound to that formalism, at which point it would cease to be considered generally intelligent. They seem to argue that their postulate is true because the individual components of such a machine are bound by those rules, but then again so are the individual components of our brains.

falsedan 9y ago

> When we notice an inconsistency within ourselves, we generally “eschew” it, whereas “if we really were inconsistent machines, we should remain content with our inconsistencies, and would happily affirm both halves of a contradiction” I feel that only philosophers & logicians are this black-and-white. There are many people content to occupy a midpoint on the spectrum of "This fact is true"..."This fact is false"; some even will accept a range! One fact I think we can all agree on is that philosophers are: old; white; dead; men; and wrong.

falsedan 9y ago
Love too get downvotes with no comments/feedback. Is this about the dead white men? I take it back. I didn't mean it. I changed my mind. I never meant for it to be taken this way. of all the possible interpretations of my words, I intended you to adopt the one where they were both witty and erudite. It's your fault you picked the wrong one. I stand by my words. I am right. Only I am right. I have seen into the heart of Man and I know its true meaning.

Chronos 9y ago

Ugh, this again. 1. Let's suppose for sake of argument that humans really can see the inherent truth of "Peano Arithmetic is consistent". That doesn't mean humans violate Gödel's Incompleteness Theorem: it could just mean that humans use axioms stronger than PA. 2. Gödel's Incompleteness Theorem only applies to systems that are perfectly logically consistent. Not sure how Penrose didn't notice, but humans... aren't. 3. When scientists proposed Quantum Mechanics as a replacement for Classical Mechanics, it was on them to explain how Quantum Mechanics simplified to Classical Mechanics in the common case. "Penrose Mechanics" is an even more radical departure — especially from a physics of computation standpoint, as Penrose Mechanics by definition would allow solving at least some of the problems in (ALL - R) in ~polynomial time. Penrose needs to explain how Penrose Mechanics reduces to Quantum Mechanics in the common case. 4. Penrose proposes that (a) there exist new physics, (b) that evolution has learned to computationally exploit the new physics via microtubules, and yet (c) that humans are the only lineage to make use of this feature of microtubules, even though microtubules are found in all eukaryotic cells (from mushrooms to amoebae). From a predator-prey standpoint alone, it would seemingly be a huge evolutionary advantage to be able to compute NP or R functions in polynomial time. (That ability is not _strictly_ implied by Penrose Mechanics, but it's a very likely consequence.) Penrose needs to explain why only humans are taking advantage of the computational power of microtubules, when microtubules have existed for billions of years and across millions of species. (TL;DR: It's the pineal gland all over again.)

manyoso 9y ago
#1 Please show how to program a Turing machine to use "axioms stronger than PA" #2 Please show how to program a Turing machine so that it is not logically consistent at a fundamental level? #3 Are you questioning the viability of objective collapse theories in general? To date we have no experimental evidence of said theories, but I don't think anyone has suggested they are inconsistent with QM. Are you? If so, please show how... #4 Show where Penrose has said humans are only species to make use of microtubules and your claims regarding Penrose' theory of objective collapse and having staggering new computational speedups are ... without any evidence. Please provide some. The tenor of your comment suggests you find Penrose to be an idiot and missing obvious problems. Consider that perhaps you are the one misunderstanding what he is proposing and that he is not, in fact, an idiot.

mcguire 9y ago

1. If Lucas and Penrose are right, there are statements which are unequivocally true, but which cannot be proved in any known formal system. It should be possible to exhibit a few. (Extra credit if they avoid theology.) If the only such statements are artificial Gödelian sentences, I'm afraid I'm...uninterested, in the same way Lucas is uninterested in inconsistent humans. 2. Lucas and Penrose assert that some entities are significantly more powerful than others. To my knowledge, Lucas describes no mechanism for this. (To his credit, Penrose does, but, to my knowledge, the Bell inequality among other things means quantum mechanics is fundamentally random. That's not helpful.) What could be the nature of such a mechanism? If it's material, it should be possible to include it in future machines. If it's not, we are in the realm of theology again. 3. So humans are intelligent and intelligent-seeming machines only simulate intelligence. If the difference is not common, obvious, and readily meaningful, then where is the distinction? ("So what? I can't write an opera either.")

manyoso 9y ago
Sigh, they don't claim that. You are also arguing with straw men. I wish people would read what Penrose actually argues instead of just assuming he is an idiot (one of the most brilliant minds of this or any century) because he argues against Strong AI. There are good refutations of Penrose' argument, but I've seen none of them here. Just a bunch of bandwagon hackers assuming Penrose doesn't know what he is talking about and then putting up lazy straw man takedowns.
- mcguire 9y ago
  Don't claim what? That there are statements that are known to be true but which cannot be proved, or that the human mind is strictly stronger than a machine? Edit: Or perhaps that Penrose is not claiming that QM is some kind of super secret sauce that makes multicellular life intrinsically stronger than computing machines?
  - manyoso 9y ago
    "...statements which are unequivocally true, but which cannot be proved in any known formal system." That.

lisper 9y ago

Refuting Lucas is very simple: Godel sentences can only be constructed for circumscribed systems, i.e. a system where you have written down all of the axioms and inference rules. So the only way to circumscribe a human brain is to isolate it from its environment. As soon as you let it communicate with (say) another human brain, you can no longer Godelize it because that brain is not the system any more, it's the two communicating brains working together. That system might have a Godel sentence of its own, but it's not the same as the Godel sentence of individual brains. So if I can put you in an isolation tank and model your brain, then I can in fact (at least in principle) construct your Godel sentence and know for certain that you will never come to the realization that this outrageously complicated mess I've just constructed (the one that says of itself that you will never come to see its truth) is true, and I can feel smug about the fact that I can see that this is true and you never will. But here's the kicker: I can never gloat about my superiority to you, because as soon as I do you are no longer in isolation. You are communicating with me, and the sentence I've take great pains to construct is no longer your Godel sentence. It is the Godel sentence of you in isolation, but as soon as I tell you your Godel sentence -- indeed as soon as I (or anyone else) say anything to you, my construction falls apart and I have to start over. Ironically, far from proving that we are not machines, Lucas's argument actually goes a long way towards demonstrating that we are, because coming up with examples of things that we interacting brains know to be true that a particular isolated brain will (with extremely high probability) never come to realize is not difficult. The four-color theorem, for example, is probably beyond any single isolated human brain. Indeed, the very argument you are reading now is probably beyond any single human brain because the brain that's writing it would never have come up with it had Godel and Lucas (and Turing) not paved the way.

fiatjaf 9y ago
That was a good argument. Have you formulated it by yourself?
- lisper 9y ago
  If I said "yes" I would be refuting my own argument :-) I composed the words in the above comment, but I don't actually know if the substance of that argument is original with me. It is heavily based on things I've learned in other places, but I don't have any specific recollection of having seen the argument being framed in exactly that way before. It's possible I invented it, but I doubt it. It's not a particularly deep insight, a minor variation on a deeper theme, that "meta-Godelization" is not algorithmic, a fact that I first learned by reading Hofstadter's "Godel, Escher, Bach." I don't actually know what the original source of that result is (I should probably go find out). But I'll happily take credit for the pedagogy if it somehow turns out I got there first.

Crito 9y ago

Franky it's irrelevant. Penrose's insistence that the human mind isn't algorithmic in the Turing sense is born from his human insecurities. But if the human mind isn't bound in such a way, than any man-made computer need not be bound either. Penrose's argument falls short of being a proper dualist argument. Furthermore Roger Penrose has not proven his claims, and they are not supported by mainstream neuroscientists. He appeals to quantum woo that he pulled out of his ass.

manyoso 9y ago
That you disparage Penrose and cite his mental motivation is likely much more relevant to your own insecurities than anything else.
dang 9y ago
> born from his human insecurities > woo that he pulled out of his ass It's not ok to argue like this on HN, so please don't. It lowers the quality of discussion in its own right and sets off a downward spiral from others. Edit: Actually, we've banned your account. Your comment history shows that you've broken HN's civility rule so egregiously, and so repeatedly, that we should have banned it long time ago. If you don't want to be banned on HN, you're welcome to email hn@ycombinator.com and promise to follow the rules scrupulously in the future. We detached this subthread from https://news.ycombinator.com/item?id=14446064 https://news.ycombinator.com/item?id=14446064 and marked it off-topic.

ccvannorman 9y ago

Site is down. Anyone have a mirror?

yongjik 9y ago

Well, I don't think our brains work by applying "logic" when we think about Gödel's theorem (or anything else). We evolved to do pattern matching, so each of us spends decades to learn more and more sophisticated pattern matching techniques, starting from mom and dad's faces to counting things to multiplication tables to, finally, understanding Gödel's theorem (for some people). In light of this, we can reinterpret "However, a human can look and see that the Gödel sentence is true" as: When you take the best pattern matching machine nature has produced, train it with distilled essentials of centuries of mathematical geniuses, and present it with the Gödel sentence, the pattern matcher strongly signals "This looks true!" When viewed this way, it looks less like an inherently human-specific achievement, but more like exactly how ML algorithms work. The only difference is that ML cannot quite pull it off, so far.

enugu 9y ago

Penrose argument, while it can be challenged, is actually much more careful than dismissals here suggest. For instance people talking about 'humans are not consistent', we are talking about effort directed into a well specified formal domain, and after long term reflection. So, of course, people make mistakes in mathematics all the time, but we are talking about consistency after intense review over long period of time. We write a program, and cant find the bug even if we look at it for a google years. To the claim, 'Godel statement is not interesting relative to other statements', the goal is that to demonstrate non-algorithmic nature of some human activity, which itself would be a big deal if true. A more qualified version of the LP claim can be expressed as follows 'One of the following is true 1) The activity of humans in some formal domain(like mathematics) is inconsistent in a deep way(not routine mistakes). 2)Humans are reflective consistent, algorithmic but will never have access to the source code(a claim about limits about brain research). 3) We are reflective consistent, can access source code, but wont be able to see why the source itself is consistent.'