14 ms·

I'm not sure about it makes sense to apply Gödel's theorem to AI. Personally, I prefer to think about it in terms of basic computability theory: We think, that

I'm not sure about it makes sense to apply Gödel's theorem to AI. Personally, I prefer to think about it in terms of basic computability theory: We think, that is a fact. Therefore, there is a function capable of transforming information into "thinked information", or what we usually call reasoning. We know that function exists, because we ourselves are an example of such function. Now, the question is: can we create a smaller function capable of performing the same feat? If we assume that that function is computable in the Turing sense then, kinda yes, there are an infinite number of turing machines that given enough time will be able to produce the expected results. Basically we need to find something between our own brain and the Kolmogorov complexity limit. That lower bound is not computable, but given that my cats understands when we are discussing to take them to the vet then... maybe we don't really need a full sized human brain for language understanding. We can run Turing machines ourselves, so we are at least Turing equivalent machines. Now, the question is: are we at most just Turing machines or something else? If we are something else, then our own CoT won't be computable, no matter how much scale we throw at it. But if we are then it is just matter of time until we can replicate ourselves.

thrance 2y ago
Penrose is a dualist, he does not believe that function can be computed in our physical universe. He believes the mind comes from another realm and "pilots" us through quantum phenomenons in the brain.
- falcor84 2y ago
  Interesting. Does that fit with the simulation hypothesis? That the world's physics are simulated on one computer, but us characters are simulated on different machines, with some latency involved?
  - mcgee21 2y ago
    Its all pop pseudoscience. Things exist. Anything that exists has an identity. Physics exists and other things (simulations, computing, etc.) that exist are subject to those physics. To say that it happens the other way around is poor logic and/or lacks falsifiability.
    - deleted 2y ago
      [deleted]
    - throwawaymaths 2y ago
      eh, its possible that weird shit is happening in physics. however, there is no evidence that this is the case. its just vibes, really.
    - fsckboy 2y ago
      >Things exist oh, cogito existo sum! checkmate theists! dude, the simulation hypothesis does not mean things don't exist, it means they don't necessarily exist in the way you have, rather unimaginatively, imagined, and you have no way to tell. and Occam's Razor does not solve the problem.
      - mcgee21 2y ago
        > … and you have no way to tell. This is exactly my point. If we have no way to tell, what experiment could you possibly use to test whether we’re in a simulation or not? The simulation hypothesis lacks falsifiability and is pseudoscience.
        fsckboy 2y ago
        the "it's not a simulation" hypothesis lacks falsifiability. also, falsifiability lacks falsifiability. don't make up complex axioms and then believe in them on faith. Occam's razor: if the universe can be simulated, then whatever form the simulation takes is the simpler explanation for the turtles all the way down. if the universe can't be simulated, explain how you know that.
- bbor 2y ago
  Which is—to use the latest philosophy lingo—dumb. To be fair to Penrose, the “Gödel’s theory about formal systems proves that souls exist” is an extremely common take; anyone following LLM discussions has likely seen it rediscovered at least once or twice. To pull from the relevant part of Hofstadter’s incredible I am a Strange Loop (a book also happens to more rigorously invoke Gödel for cognitive science): And this is our central quandary. Either we believe in a nonmaterial soul that lives outside the laws of physics, which amounts to a nonscientific belief in magic, or we reject that idea, in which case the eternally beckoning question "What could ever make a mere physical pattern be me?” After all, a phrase like "physical system" or "physical substrate" brings to mind for most people… an intricate structure consisting of vast numbers of interlocked wheels, gears, rods, tubes, balls, pendula, and so forth, even if they are tiny, invisible, perfectly silent, and possibly even probabilistic. Such an array of interacting inanimate stuff seems to most people as unconscious and devoid of inner light as a flush toilet, an automobile transmission, a fancy Swiss watch (mechanical or electronic), a cog railway, an ocean liner, or an oil refinery. Such a system is not just probably unconscious, it is *necessarily* so, as they see it. This is the kind of single-level intuition so skillfully exploited by John Searle in his attempts to convince people that computers could never be conscious, no matter what abstract patterns might reside in them, and could never mean anything at all by whatever long chains of lexical items they might string together. Highly recommend it for anyone who liked Gödel, Escher, Bach, but wants more explicit scientific theses! He basically wrote it to clarify the more artsy/rhetorical points made in the former book.
  - jfengel 2y ago
    It feels really weird to say that Roger Penrose is being dumb. It's accurate. But it feels really weird. It's not uncommon for great scientists to be totally out of their depth even in nearby fields, and not realize it. But this isn't the hard part of either computability or philosophy of mind.
    - thrance 2y ago
      We really ought to stop mythicizing people into superhuman heroes/geniuses. Penrose is an authority in some fields of theoretical physics, but that doesn't give any value to what he has to say on consciousness or AI. On that topic, he has clearly adopted an unscientific approach: he wants to believe the soul exists and is immaterial and seeks evidence for it.
- lupusreal 2y ago
  I could be wrong but I don't think he's a dualist. He rejects human consciousness being computable on classical computers, like Turing machines, but thinks that quantum stuff is necessary. Quantum brains/computers are or would be computing in our physical universe, making him a materialist not a dualist. His big thing is using this quantum stuff to free his own mind from the confines of Gödel's theorems (I think for egotistical reasons.) Simultaneously, actual dualists flock to his theories because they think association with Penrose lends credibility to their religious stuff.
northern-lights 2y ago
> there is a function capable of transforming information into "thinked information", or what we usually call reasoning. We know that function exists, because we ourselves are an example of such function. We mistakenly assume, they are true because perhaps we want them to be true. But we have no proof that either of these are true.
- genewitch 2y ago
  Descartes disagrees.
  - s_m_t 2y ago
    And? Mind-body dualism as Descartes imagined has been practically disproven on almost every front.
    - bigstrat2003 2y ago
      Mind-body dualism has nothing to do with this. The point is that, as Descartes observed, the fact that I myself am thinking proves that I exist. This goes directly against what northern-lights said, when he said that we have no proof that reasoning exists or that we do it.
      - riwsky 2y ago
        “Cogito ergo cogito”?
      - viccis 2y ago
        Kant addressed this Cartesian duality in the "The paralogisms of pure reason" section of the Transcendental Dialectic within his Critique of Pure Reason. He points out that the "I" in "I think, therefore I am" is a different "I" in the subject part vs the object part of that phrase. Quick context: His view of what constitutes a subject, which is to say a thinking person in this case, is one which over time (and time is very important here) observes manifold partial aspects about objects through perception, then through apprehension (the building of understanding through successive sensibilities over time) the subject schematizes information about the object. Through logical judgments, from which Kant derives his categories, we can understand the object and use synthetic a priori reasoning about the object. So for him, the statement "I am" means simply that you are a subject who performs this perception and reasoning process, as one's "existence" is mediated and predicated on doing such a process over time. So then "I think, therefore I am" becomes a tautology. Assuming that the "I" in "I am" exists as an object, which is to say a thing of substance, one which other thinking subjects could reason about, becomes what he calls "transcendental illusion", which is the application of transcendental reasoning not rooted in sensibility. He calls this metaphysics, and he focuses on the soul (the topic at hand here), the cosmos, and God as the three topics of metaphysics in his Transcendental Dialectic. I think that in general, discussion about epistemology with regard to AI would be better if people started at least from Kant (either building on his ideas or critical of them), as his CPR really shaped a lot of the post-Enlightenment views on epistemology that a lot of us carry with us without knowing. In my opinion, AI is vulnerable to a criticism that empiricists like Hume applied to people (viewing people as "bundles of experience" and critiquing the idea that we can create new ideas independent of our experience). I do think that AI suffers from this problem, as estimating a generative probability distribution over data means that no new information can be created that is not simply a logically ungrounded combination of previous information. I have not read any discussion of how Kant's view of our ability to make new information (application of categories grounded by our perception) might influence a way to make an actual thinking machine. It would be fascinating to see an approach that combines new AI approaches as the way the machine perceives information and then combines it with old AI approaches that build on logic systems to "reason" in a way that's grounded in truth. The problem with old AI is that it's impossible to model everything with logic (the failure of logical posivitism should have warned them), however it IS possible to combine logic with perception like Kant proposed. I hope this makes sense. I've noticed a lack of philosophical rigor around the discussion of AI epistemology, and it feels like a lot of American philosophy research, being rooted in modern analytical tradition that IMO can't adapt easily to an ontological shift from human to machine as the subject, hasn't really risen to the challenge yet.
bwoj 2y ago
It is a big mistake to think that most computability theory applies to AI, including Gödel’s Theorem. People start off wrong by talking about AI “algorithms.” The term applies more correctly to concepts like gradient descent. But the inferences of the resulting neural nets is not an algorithm. It is not a defined sequence of operations that produces a defined result. It is better described as a heuristic, a procedure that approximates a correct result but provides no mathematical guarantees. Other aspects of ANN that show that Gödel doesn’t apply is that they are not formal systems. Formal system is a collection of defined operations. The building blocks of ANN could perhaps be built into a formal system. Petri nets have been demonstrated to be computationally equivalent to Turing machines. But this is really an indictment on the implementation. It’s the same as using your PC, implementing a formal system like its instruction set to run a heuristic computation. Formal system can implement informal systems. I don’t think you have to look at humans very hard to see that humans don’t implement any kind of formal system and are not equivalent to Turing machines.
- PeterWhittaker 2y ago
  But surely any limits on formal systems apply to informal systems? By this, I am more or less suggesting that formal systems are the best we can do, the best possible representations of knowledge, computability, etc., and that informal systems cannot be "better" (a loaded term herein, for sure) than formal systems. So if Gödel tells us that either formal systems will be consistent and make statements they cannot prove XOR be inconsistent and therefore unreliable, at least to some degree, then surely informal systems will, at best, be the same, and, at worst, be much worse?
  - zja 2y ago
    Yeah I don’t know why GP would think computability theory doesn’t apply to AI. Is there a single example of a problem that isn’t computable by a Turing machine that can be computed by AI?
    - bwoj 2y ago
      It does apply to AI in terms of the computers we compute neural networks on may be equivalent to Turning machines but the ANN networks are not. If you did reduce the ANN down to a formal system, you will likely find that in terms of Godels theorem that it would be sufficiently powerful to prove a falsehood. Thus not meeting the consistency property we would like in a system used to prove things.
btilly 2y ago
With sufficient compute capacity, a complete physical simulation of a human should be possible. This means that, even though we are fallible, there is nothing that we do which can't be simulated on a Turing machine.
- chromanoid 2y ago
  May still only yield a philosophical zombie. You can simulate gravity but never move something with its simulation.
  - genewitch 2y ago
    If I have a 9-DOF sensor in meatspace and am feeding that to a "simulation" that helps a PID coalesce faster then my simulation can move something. When I tell my computer to simulate blackbody radiation... What you said sounds good, but I don't think it's philosophically robust.
    - chromanoid 2y ago
      I think you misunderstood my point. A simulation is never the actual simulated phenomenon. When you understand consciousness as a "physical" phenomenon (e.g. as in most forms of panprotopsychism), believing in being able to create it by computation is like believing in being able to generate gravity by computation.
      - genewitch 2y ago
        we're out of my wheelhouse but it feels to me that entropy and gravity are fundamentally linked and a quick search shows i'm not alone: https://en.wikipedia.org/wiki/Entropic_gravity https://en.wikipedia.org/wiki/Entropic_gravity read this as: literally creating gravity by simulating it hard enough.
        chromanoid 2y ago
        I don't see how computation itself can be a plausible cause here. The physical representation of the computation might be the cause, but the computation itself is not substrate-independent in its possible effect. That is again my point.
skywhopper 2y ago
Not every fact is computable. We are not Turing machines.
- cornel_io 2y ago
  We don't know every fact, either, so I don't know how you can use that idea to say that we're not Turing machines. Apart, of course, from the trivial fact that we are far more limited than a Turing machine...
cowl 2y ago
he starts with "consciousnes is not computable". You can not just ignore it as a central argument withouth explaining why your preference to think on it as basic computability theory makes more sence than his. What's more, whatever you like to call the transoforming of information into thinked information by definition can not be a (mathematical) function, because it would require all people to process the same information in the same way and this is plainly false
- amon_raw 2y ago
  >> What's more, whatever you like to call the transoforming of information into thinked information by definition can not be a (mathematical) function, because it would require all people to process the same information in the same way and this is plainly false No this isn't the checkmate you think it is. It could still be a mathematical function. But every person transforming information into "thinked information" could have a different implementation of this function. Which would be expected as no person is made of the same code (DNA).
  - cb321 2y ago
    Not sure there aren't 10 other "lines of discussion/disagreement" on this, but the one I think might be most salient is Davidson's https://en.wikipedia.org/wiki/Anomalous_monism https://en.wikipedia.org/wiki/Anomalous_monism (Davidson was a student of Quine who people here might know more from his work in logic/kitchy programs-that-print-themselves games than as the bigtime analytic philosophy Word & Object, naturalized epistemology type work). Note that using words like "function" and "mathematical" are more the biases of computer science/Penrose while philosophy of the mind & psychology has more typically used slightly different ideas of "attitudes" and "events" to guide the discussion. I don't think this really radically shifts many central disputes, except (which Penrose might view as a critical "except") for all the funny business quantum mechanics / quantum computation can potentially bring in and the (undisputed) physicality of our brains and (also undisputed) lack of understanding of many details of biological computation.
katabasis 2y ago
Many philosophical traditions which incorporate a meditation practice emphasize that your consciousness is distinct from the contents of your thoughts. Meditation (even practiced casually) can provide a direct experience of this. When it comes to the various kinds of thought-processes that humans engage in (linguistic thinking, logic, math, etc) I agree that you can describe things in terms of functions that have definite inputs and outputs. So human thinking is probably computable, and I think that LLMs can be said to be ”think” in ways that are analogous to what we do. But human consciousness produces an experience (the experience of being conscious) as opposed to some definite output. I do not think it is computable in the same way. I don’t necessarily think that you need to subscribe to dualism or religious beliefs to explain consciousness - it seems entirely possible (maybe even likely) that what we experience as consciousness is some kind of illusory side-effect of biological processes as opposed to something autonomous and “real”. But I do think it’s still important to maintain a distinction between “thinking” (computable, we do it, AIs do it as well) and “consciousness” (we experience it, probably many animals experience it also, but it’s orthogonal to the linguistic or logical reasoning processes that AIs are currently capable of). At some point this vague experience of awareness may be all that differentiates us from the machines, so we shouldn’t dismiss it.
- mont_tag 2y ago
  > I think that LLMs can be said to be ”think” in ways that are analogous to what we do. ... But human consciousness produces an experience (the experience of being conscious) as opposed to some definite output. I do not think it is computable in the same way. "We've all been dancing around the basic issue: does Data have a soul?" -- Captain Louvois. https://memory-alpha.fandom.com/wiki/The_Measure_Of_A_Man_(episode) https://memory-alpha.fandom.com/wiki/The_Measure_Of_A_Man_(e...
  - xpe 2y ago
    I for one (along with many thinkers) define intelligence as the extent to which an agent can solve a particular task. I choose the definition to separate it from issues involving consciousness. Both matter of course.
dartos 2y ago
All of this is a fine thought experiment, but in practice there are physical limitations to digital processors that don’t seem to manifest in our brains (energy use in the ability to think vs running discrete commands) It’s possible that we haven’t found a way to express your thinking function digitally, which I think is true, but I have a feeling that the complexity of thought requires the analog-ness of our brains.
- kaiwen1 2y ago
  If human-like cognition isn't possible on digital computers, it's certainly is on quantum ones. The Deutsch-Church-Turing principle shows that a quantum Turing machine can efficiently simulate any physically realizable computational process.
  - dartos 2y ago
    > certainly is on quantum ones I don’t think that’s a certainty.
    - riwsky 2y ago
      In 50% of possible worlds, it’s 100% certain!
      - dartos 2y ago
        Does that math still check out if math works differently in other worlds?
        riwsky 2y ago
        Probably!
        dartos 2y ago
        In about 50% of the worlds, there’s 100% chance, would you say?
esperent 2y ago
> If we assume that that function is computable in the Turing sense This is a big assumption. I'm not saying it's wrong, but I am saying it's not reasonable to just handwave and assume that it's right.
dinkumthinkum 2y ago
No, I mean, it's nice but I don't think any of that works. You say "Therefore, there is a function capable ..." that is a non-sequitur. But, let's set that aside, I think the key point here is about Turing machines and computability. Do you really think your mind and thought-process is a Turing machine? How many watts of power did it take to write your comment? I think it is an absolute certainty that human intelligence is not like a Turing machine at all. Do you find it much more troublesome to think about continuous problems or is ironically more troublesome to discretize continuous problems in order to work with them?
- donkeybeer 2y ago
  I think it's much more difficult to fix a broken down human than a broken down car. Does it mean, by your logic, we are closer to cars?
- int_19h 2y ago
  What makes it an "absolute certainty" for you? FWIW human brain does indeed consume a lot of energy, accounting for over 20% of our body metabolism. We don't know how to attribute specific watts consumed to specific thoughts because we don't understand the functioning of the brain enough, but there's no obvious reason why it shouldn't be possible.
lanstin 2y ago
Worth pointing out that we aren't Turing equivalent machines - infinite storage is not a computability class that is realizable in the universe, so far as we know (and such a claim would require extraordinary evidence). As well, perhaps, worth noting that because a subset of the observable universe is performing some function, then it is an assumption that there is some finite or digital mathematical function equivalent to that function; a reasonable assumption but still an assumption. Most models of the quantum universe involve continuously variable values, not digital values. Is there a Turing machine that can output all the free parameters of the standard model?
- eru 2y ago
  > Is there a Turing machine that can output all the free parameters of the standard model? Sure, just hard code them. > As well, perhaps, worth noting that because a subset of the observable universe is performing some function, then it is an assumption that there is some finite or digital mathematical function equivalent to that function; a reasonable assumption but still an assumption. Most models of the quantum universe involve continuously variable values, not digital values. Things seem to be quantised at a low enough level. Also: interestingly enough quantum mechanics is both completely deterministic and linear. That means even if it was continuous, you could simulate it to an arbitrary precision without errors building up chaotically. (Figuring out how chaos, as famously observed in the weather, arises in the real world is left as an exercise to the reader. Also a note: the Copenhagen interpretation introduces non-determinism to _interpret_ quantum mechanics but that's not part of the underlying theory, and there are interpretations that have no need for this crutch.)
  - tsimionescu 2y ago
    > Things seem to be quantised at a low enough level. Some things. But others are very much not: in particular, space and time are not quantized, and in fact are not even quantizable. A theory in which there is some discrete minimal unit of space (or of spacetime) is trivially incompatible with special relativity, so it is incompatible with quantum mechanics (QFT, specifically). This is easy to see from the nature of the Lorenz transformation: if two objects are at a distance D = n*min for some observer, they will be at a distance D' = n*min*gamma for some other observer, where gamma is always < 1 for an o server moving at a higher speed in a direction aligned with the two objects. So the distance for that second observer will be a non-integer multiple of the minimum distance, so your theory is no longer quantized.
kazinator 2y ago
[dead]
mxkopy 2y ago
I think the complication here is that brains are probabilistic, which admits the possibility that they can’t be directly related to non probabilistic computability classes. I think there’s a paper I forget the name of that says quantum computers can decide the halting problem with some probability (which makes sense because you could always just flip a coin and decide it with some probability) - maybe brains are similar
PaulDavisThe1st 2y ago
> We think, that is a fact. It likely is a fact, but we don't really know what we mean by "think". LLMs have illuminated this point from a relatively new direction: we do not know if their mechanism(s) for language generation are similar to our own, or not. We don't really understand the relationship between "reasoning" and "thinking". We don't really understand the difference between Kahneman's "fast" and "slow" thinking. Something happens, probably in our brains, that we experience and that seems causally prior to some of our behavior. We call it thinking, but we don't know much about what it actually is.
- jay_kyburz 2y ago
  I don't think its useful or even interesting to talk about AI in relation to how humans think, or whether or not they will be "conscious" whatever that might mean. AIs are not going to be like humans because they will have perfect recall of a massive database of facts, and be able to do math well beyond any human brain. The interesting question to me is, when will we be able to give AI very large tasks, and when will it to be able to break the tasks down into smaller and smaller tasks and complete them. When will it be able to set its own goals, and know when it has achieved them? When will it be able to recognize that it doesn't know something and do the work to fill in the blanks. I get the impression that LLMs don't really know what they are saying at the moment, so don't have any way to test what they are saying is true or not.
  - unparagoned 2y ago
    There was a study where they trained a model to lie. When they looked at what was happening internally they could see it knew the truth but just switched things at the outer nodes to lie.
- agarren 2y ago
  I think we have a pretty good idea that we are not stochastic parrots - sophisticated or not. Anyone suggesting that we’re running billion parameter models in order to bang out a snarky comment is probably trying to sell you something (and crypto’s likely involved.) I think you’re right, LLMs have demonstrated that relatively sophisticated mathematics involving billions of params and an internet full of training data is capable of some truly, truly, remarkable things. But as Penrose is saying, there are provable limits to computation. If we’re going to assume that intelligence as we experience it is computable, then Gödel’s theorem (and, frankly, the field of mathematics) seems to present a problem.
agarren 2y ago
Gödel’s incompleteness theorem, and, say, the halting problem seem to fall squarely into the bucket of “basic computability theory” in precisely the way that “we think, that is a fact”, does not (D.A. hat tip) You’re arguing that we know artificial reasoning exists because we are capable of reasoning. This presupposes that reasoning is computable and that we ourselves reason by computation. But that’s exactly what Penrose is saying isn’t the case - you’re saying we’re walking Turing machines, we’re intelligent, so we must be able to effectively create copies of that intelligence. Penrose is saying that intelligence is poorly defined, that it requires consciousness which is poorly understood, and that we are not meat-based computers. Your last question misses the point completely. “If we are something else, then out CoT won’t be computable…” it’s like you’re almost there but you can’t let go of “we are meat-machines, everything boils down to computation, we can cook up clones”. Except, “basic computability theory” says that’s not even wrong.
godelski 2y ago
> Personally, I prefer to think about it in terms of basic computability theory: Gödel's incompleteness theorem applies to computing. I'm sure you're familiar with the Halting Problem. Gödel's applies to any axiomatic system. The trouble is, it's very hard to make a system without axioms. They are sneaky and it's different than any logic you're probably familiar with. And don't forget Church-Turing, Gödel Numbers, and all the other stuff. Programming is math and Gödel did essential work on the theory of computation. It would be weird NOT to include his work in this conversation. > are we at most just Turing machines or something else? But this is a great question. Many believe no. Personally I'm unsure, but lean no. Penrose is a clear no but he has some wacky ideas. Problem is, it's hard to tell a bad wacky idea from a good wacky idea. Rephrasing Clarke's Second Law: Genius is nearly indistinguishable from insanity. The only way to tell is with time. But look into things like NARS and Super Turing machines (Hypercomputation). There's a whole world of important things that are not often discussed when it comes to the discussion of AGI. But for those that don't want to dig deep into the math, pick up some Sci-Fi and suspend your disbelief. Star Trek, The Orville and the like have holographic simulations and I doubt anyone would think they're conscious, despite being very realistic. But The Doctor in Voyager or Isaac in The Orville are good examples of the contrary. The Doctor is an entity you see become conscious. It's fiction, but that doesn't mean there aren't deep philosophical questions. Even if they're marked by easy to digest entertainment. Good stories are like good horror, they get under your skin, infect you, and creep in Edit: I'll leave you with another question. Regardless of our Turing or Super-Turing status; is a Turing machine sufficient for consciousness to arise?
- red75prime 2y ago
  > Regardless of our Turing or Super-Turing status; is a Turing machine sufficient for consciousness to arise? Another question. How do you go about detecting whether consciousness has arisen?
  - godelski 2y ago
    It's a great question. Best we got is ~~RLHF~~ Potter's definition: I know it when I see it
- 2y ago
milesrout 2y ago
>Therefore, there is a function capable of transforming information into "thinked information", or what we usually call reasoning. We know that function exists, because we ourselves are an example of such function. "Thinked information" is a colour not an inherent property of information. The fact that information has been thought is like the fact it is copyrighted. It is not something inherent to the information, but a property of its history. https://ansuz.sooke.bc.ca/entry/23 https://ansuz.sooke.bc.ca/entry/23
eru 2y ago
> Now, the question is: can we create a smaller function capable of performing the same feat? Where does this question come from? Especially where does the 'smaller' requirement come from?
AndyNemmity 2y ago
You're arguing that submarines swim.

thrance 2y ago

Penrose is a dualist, he does not believe that function can be computed in our physical universe. He believes the mind comes from another realm and "pilots" us through quantum phenomenons in the brain.

falcor84 2y ago
Interesting. Does that fit with the simulation hypothesis? That the world's physics are simulated on one computer, but us characters are simulated on different machines, with some latency involved?
- mcgee21 2y ago
  Its all pop pseudoscience. Things exist. Anything that exists has an identity. Physics exists and other things (simulations, computing, etc.) that exist are subject to those physics. To say that it happens the other way around is poor logic and/or lacks falsifiability.
  - deleted 2y ago
    [deleted]
  - throwawaymaths 2y ago
    eh, its possible that weird shit is happening in physics. however, there is no evidence that this is the case. its just vibes, really.
  - fsckboy 2y ago
    >Things exist oh, cogito existo sum! checkmate theists! dude, the simulation hypothesis does not mean things don't exist, it means they don't necessarily exist in the way you have, rather unimaginatively, imagined, and you have no way to tell. and Occam's Razor does not solve the problem.
    - mcgee21 2y ago
      > … and you have no way to tell. This is exactly my point. If we have no way to tell, what experiment could you possibly use to test whether we’re in a simulation or not? The simulation hypothesis lacks falsifiability and is pseudoscience.
      - fsckboy 2y ago
        the "it's not a simulation" hypothesis lacks falsifiability. also, falsifiability lacks falsifiability. don't make up complex axioms and then believe in them on faith. Occam's razor: if the universe can be simulated, then whatever form the simulation takes is the simpler explanation for the turtles all the way down. if the universe can't be simulated, explain how you know that.
bbor 2y ago
Which is—to use the latest philosophy lingo—dumb. To be fair to Penrose, the “Gödel’s theory about formal systems proves that souls exist” is an extremely common take; anyone following LLM discussions has likely seen it rediscovered at least once or twice. To pull from the relevant part of Hofstadter’s incredible I am a Strange Loop (a book also happens to more rigorously invoke Gödel for cognitive science): And this is our central quandary. Either we believe in a nonmaterial soul that lives outside the laws of physics, which amounts to a nonscientific belief in magic, or we reject that idea, in which case the eternally beckoning question "What could ever make a mere physical pattern be me?” After all, a phrase like "physical system" or "physical substrate" brings to mind for most people… an intricate structure consisting of vast numbers of interlocked wheels, gears, rods, tubes, balls, pendula, and so forth, even if they are tiny, invisible, perfectly silent, and possibly even probabilistic. Such an array of interacting inanimate stuff seems to most people as unconscious and devoid of inner light as a flush toilet, an automobile transmission, a fancy Swiss watch (mechanical or electronic), a cog railway, an ocean liner, or an oil refinery. Such a system is not just probably unconscious, it is *necessarily* so, as they see it. This is the kind of single-level intuition so skillfully exploited by John Searle in his attempts to convince people that computers could never be conscious, no matter what abstract patterns might reside in them, and could never mean anything at all by whatever long chains of lexical items they might string together. Highly recommend it for anyone who liked Gödel, Escher, Bach, but wants more explicit scientific theses! He basically wrote it to clarify the more artsy/rhetorical points made in the former book.
- jfengel 2y ago
  It feels really weird to say that Roger Penrose is being dumb. It's accurate. But it feels really weird. It's not uncommon for great scientists to be totally out of their depth even in nearby fields, and not realize it. But this isn't the hard part of either computability or philosophy of mind.
  - thrance 2y ago
    We really ought to stop mythicizing people into superhuman heroes/geniuses. Penrose is an authority in some fields of theoretical physics, but that doesn't give any value to what he has to say on consciousness or AI. On that topic, he has clearly adopted an unscientific approach: he wants to believe the soul exists and is immaterial and seeks evidence for it.
lupusreal 2y ago
I could be wrong but I don't think he's a dualist. He rejects human consciousness being computable on classical computers, like Turing machines, but thinks that quantum stuff is necessary. Quantum brains/computers are or would be computing in our physical universe, making him a materialist not a dualist. His big thing is using this quantum stuff to free his own mind from the confines of Gödel's theorems (I think for egotistical reasons.) Simultaneously, actual dualists flock to his theories because they think association with Penrose lends credibility to their religious stuff.

northern-lights 2y ago

> there is a function capable of transforming information into "thinked information", or what we usually call reasoning. We know that function exists, because we ourselves are an example of such function. We mistakenly assume, they are true because perhaps we want them to be true. But we have no proof that either of these are true.

genewitch 2y ago
Descartes disagrees.
- s_m_t 2y ago
  And? Mind-body dualism as Descartes imagined has been practically disproven on almost every front.
  - bigstrat2003 2y ago
    Mind-body dualism has nothing to do with this. The point is that, as Descartes observed, the fact that I myself am thinking proves that I exist. This goes directly against what northern-lights said, when he said that we have no proof that reasoning exists or that we do it.
    - riwsky 2y ago
      “Cogito ergo cogito”?
    - viccis 2y ago
      Kant addressed this Cartesian duality in the "The paralogisms of pure reason" section of the Transcendental Dialectic within his Critique of Pure Reason. He points out that the "I" in "I think, therefore I am" is a different "I" in the subject part vs the object part of that phrase. Quick context: His view of what constitutes a subject, which is to say a thinking person in this case, is one which over time (and time is very important here) observes manifold partial aspects about objects through perception, then through apprehension (the building of understanding through successive sensibilities over time) the subject schematizes information about the object. Through logical judgments, from which Kant derives his categories, we can understand the object and use synthetic a priori reasoning about the object. So for him, the statement "I am" means simply that you are a subject who performs this perception and reasoning process, as one's "existence" is mediated and predicated on doing such a process over time. So then "I think, therefore I am" becomes a tautology. Assuming that the "I" in "I am" exists as an object, which is to say a thing of substance, one which other thinking subjects could reason about, becomes what he calls "transcendental illusion", which is the application of transcendental reasoning not rooted in sensibility. He calls this metaphysics, and he focuses on the soul (the topic at hand here), the cosmos, and God as the three topics of metaphysics in his Transcendental Dialectic. I think that in general, discussion about epistemology with regard to AI would be better if people started at least from Kant (either building on his ideas or critical of them), as his CPR really shaped a lot of the post-Enlightenment views on epistemology that a lot of us carry with us without knowing. In my opinion, AI is vulnerable to a criticism that empiricists like Hume applied to people (viewing people as "bundles of experience" and critiquing the idea that we can create new ideas independent of our experience). I do think that AI suffers from this problem, as estimating a generative probability distribution over data means that no new information can be created that is not simply a logically ungrounded combination of previous information. I have not read any discussion of how Kant's view of our ability to make new information (application of categories grounded by our perception) might influence a way to make an actual thinking machine. It would be fascinating to see an approach that combines new AI approaches as the way the machine perceives information and then combines it with old AI approaches that build on logic systems to "reason" in a way that's grounded in truth. The problem with old AI is that it's impossible to model everything with logic (the failure of logical posivitism should have warned them), however it IS possible to combine logic with perception like Kant proposed. I hope this makes sense. I've noticed a lack of philosophical rigor around the discussion of AI epistemology, and it feels like a lot of American philosophy research, being rooted in modern analytical tradition that IMO can't adapt easily to an ontological shift from human to machine as the subject, hasn't really risen to the challenge yet.

bwoj 2y ago

It is a big mistake to think that most computability theory applies to AI, including Gödel’s Theorem. People start off wrong by talking about AI “algorithms.” The term applies more correctly to concepts like gradient descent. But the inferences of the resulting neural nets is not an algorithm. It is not a defined sequence of operations that produces a defined result. It is better described as a heuristic, a procedure that approximates a correct result but provides no mathematical guarantees. Other aspects of ANN that show that Gödel doesn’t apply is that they are not formal systems. Formal system is a collection of defined operations. The building blocks of ANN could perhaps be built into a formal system. Petri nets have been demonstrated to be computationally equivalent to Turing machines. But this is really an indictment on the implementation. It’s the same as using your PC, implementing a formal system like its instruction set to run a heuristic computation. Formal system can implement informal systems. I don’t think you have to look at humans very hard to see that humans don’t implement any kind of formal system and are not equivalent to Turing machines.

PeterWhittaker 2y ago
But surely any limits on formal systems apply to informal systems? By this, I am more or less suggesting that formal systems are the best we can do, the best possible representations of knowledge, computability, etc., and that informal systems cannot be "better" (a loaded term herein, for sure) than formal systems. So if Gödel tells us that either formal systems will be consistent and make statements they cannot prove XOR be inconsistent and therefore unreliable, at least to some degree, then surely informal systems will, at best, be the same, and, at worst, be much worse?
- zja 2y ago
  Yeah I don’t know why GP would think computability theory doesn’t apply to AI. Is there a single example of a problem that isn’t computable by a Turing machine that can be computed by AI?
  - bwoj 2y ago
    It does apply to AI in terms of the computers we compute neural networks on may be equivalent to Turning machines but the ANN networks are not. If you did reduce the ANN down to a formal system, you will likely find that in terms of Godels theorem that it would be sufficiently powerful to prove a falsehood. Thus not meeting the consistency property we would like in a system used to prove things.

btilly 2y ago

With sufficient compute capacity, a complete physical simulation of a human should be possible. This means that, even though we are fallible, there is nothing that we do which can't be simulated on a Turing machine.

chromanoid 2y ago
May still only yield a philosophical zombie. You can simulate gravity but never move something with its simulation.
- genewitch 2y ago
  If I have a 9-DOF sensor in meatspace and am feeding that to a "simulation" that helps a PID coalesce faster then my simulation can move something. When I tell my computer to simulate blackbody radiation... What you said sounds good, but I don't think it's philosophically robust.
  - chromanoid 2y ago
    I think you misunderstood my point. A simulation is never the actual simulated phenomenon. When you understand consciousness as a "physical" phenomenon (e.g. as in most forms of panprotopsychism), believing in being able to create it by computation is like believing in being able to generate gravity by computation.
    - genewitch 2y ago
      we're out of my wheelhouse but it feels to me that entropy and gravity are fundamentally linked and a quick search shows i'm not alone: https://en.wikipedia.org/wiki/Entropic_gravity https://en.wikipedia.org/wiki/Entropic_gravity read this as: literally creating gravity by simulating it hard enough.
      - chromanoid 2y ago
        I don't see how computation itself can be a plausible cause here. The physical representation of the computation might be the cause, but the computation itself is not substrate-independent in its possible effect. That is again my point.

skywhopper 2y ago

Not every fact is computable. We are not Turing machines.

cornel_io 2y ago
We don't know every fact, either, so I don't know how you can use that idea to say that we're not Turing machines. Apart, of course, from the trivial fact that we are far more limited than a Turing machine...

cowl 2y ago

he starts with "consciousnes is not computable". You can not just ignore it as a central argument withouth explaining why your preference to think on it as basic computability theory makes more sence than his. What's more, whatever you like to call the transoforming of information into thinked information by definition can not be a (mathematical) function, because it would require all people to process the same information in the same way and this is plainly false

amon_raw 2y ago
>> What's more, whatever you like to call the transoforming of information into thinked information by definition can not be a (mathematical) function, because it would require all people to process the same information in the same way and this is plainly false No this isn't the checkmate you think it is. It could still be a mathematical function. But every person transforming information into "thinked information" could have a different implementation of this function. Which would be expected as no person is made of the same code (DNA).
- cb321 2y ago
  Not sure there aren't 10 other "lines of discussion/disagreement" on this, but the one I think might be most salient is Davidson's https://en.wikipedia.org/wiki/Anomalous_monism https://en.wikipedia.org/wiki/Anomalous_monism (Davidson was a student of Quine who people here might know more from his work in logic/kitchy programs-that-print-themselves games than as the bigtime analytic philosophy Word & Object, naturalized epistemology type work). Note that using words like "function" and "mathematical" are more the biases of computer science/Penrose while philosophy of the mind & psychology has more typically used slightly different ideas of "attitudes" and "events" to guide the discussion. I don't think this really radically shifts many central disputes, except (which Penrose might view as a critical "except") for all the funny business quantum mechanics / quantum computation can potentially bring in and the (undisputed) physicality of our brains and (also undisputed) lack of understanding of many details of biological computation.

katabasis 2y ago

Many philosophical traditions which incorporate a meditation practice emphasize that your consciousness is distinct from the contents of your thoughts. Meditation (even practiced casually) can provide a direct experience of this. When it comes to the various kinds of thought-processes that humans engage in (linguistic thinking, logic, math, etc) I agree that you can describe things in terms of functions that have definite inputs and outputs. So human thinking is probably computable, and I think that LLMs can be said to be ”think” in ways that are analogous to what we do. But human consciousness produces an experience (the experience of being conscious) as opposed to some definite output. I do not think it is computable in the same way. I don’t necessarily think that you need to subscribe to dualism or religious beliefs to explain consciousness - it seems entirely possible (maybe even likely) that what we experience as consciousness is some kind of illusory side-effect of biological processes as opposed to something autonomous and “real”. But I do think it’s still important to maintain a distinction between “thinking” (computable, we do it, AIs do it as well) and “consciousness” (we experience it, probably many animals experience it also, but it’s orthogonal to the linguistic or logical reasoning processes that AIs are currently capable of). At some point this vague experience of awareness may be all that differentiates us from the machines, so we shouldn’t dismiss it.

mont_tag 2y ago
> I think that LLMs can be said to be ”think” in ways that are analogous to what we do. ... But human consciousness produces an experience (the experience of being conscious) as opposed to some definite output. I do not think it is computable in the same way. "We've all been dancing around the basic issue: does Data have a soul?" -- Captain Louvois. https://memory-alpha.fandom.com/wiki/The_Measure_Of_A_Man_(episode) https://memory-alpha.fandom.com/wiki/The_Measure_Of_A_Man_(e...
- xpe 2y ago
  I for one (along with many thinkers) define intelligence as the extent to which an agent can solve a particular task. I choose the definition to separate it from issues involving consciousness. Both matter of course.

dartos 2y ago

All of this is a fine thought experiment, but in practice there are physical limitations to digital processors that don’t seem to manifest in our brains (energy use in the ability to think vs running discrete commands) It’s possible that we haven’t found a way to express your thinking function digitally, which I think is true, but I have a feeling that the complexity of thought requires the analog-ness of our brains.

kaiwen1 2y ago
If human-like cognition isn't possible on digital computers, it's certainly is on quantum ones. The Deutsch-Church-Turing principle shows that a quantum Turing machine can efficiently simulate any physically realizable computational process.
- dartos 2y ago
  > certainly is on quantum ones I don’t think that’s a certainty.
  - riwsky 2y ago
    In 50% of possible worlds, it’s 100% certain!
    - dartos 2y ago
      Does that math still check out if math works differently in other worlds?
      - riwsky 2y ago
        Probably!
        dartos 2y ago
        In about 50% of the worlds, there’s 100% chance, would you say?

esperent 2y ago

> If we assume that that function is computable in the Turing sense This is a big assumption. I'm not saying it's wrong, but I am saying it's not reasonable to just handwave and assume that it's right.

dinkumthinkum 2y ago

No, I mean, it's nice but I don't think any of that works. You say "Therefore, there is a function capable ..." that is a non-sequitur. But, let's set that aside, I think the key point here is about Turing machines and computability. Do you really think your mind and thought-process is a Turing machine? How many watts of power did it take to write your comment? I think it is an absolute certainty that human intelligence is not like a Turing machine at all. Do you find it much more troublesome to think about continuous problems or is ironically more troublesome to discretize continuous problems in order to work with them?

donkeybeer 2y ago
I think it's much more difficult to fix a broken down human than a broken down car. Does it mean, by your logic, we are closer to cars?
int_19h 2y ago
What makes it an "absolute certainty" for you? FWIW human brain does indeed consume a lot of energy, accounting for over 20% of our body metabolism. We don't know how to attribute specific watts consumed to specific thoughts because we don't understand the functioning of the brain enough, but there's no obvious reason why it shouldn't be possible.

lanstin 2y ago

Worth pointing out that we aren't Turing equivalent machines - infinite storage is not a computability class that is realizable in the universe, so far as we know (and such a claim would require extraordinary evidence). As well, perhaps, worth noting that because a subset of the observable universe is performing some function, then it is an assumption that there is some finite or digital mathematical function equivalent to that function; a reasonable assumption but still an assumption. Most models of the quantum universe involve continuously variable values, not digital values. Is there a Turing machine that can output all the free parameters of the standard model?

eru 2y ago
> Is there a Turing machine that can output all the free parameters of the standard model? Sure, just hard code them. > As well, perhaps, worth noting that because a subset of the observable universe is performing some function, then it is an assumption that there is some finite or digital mathematical function equivalent to that function; a reasonable assumption but still an assumption. Most models of the quantum universe involve continuously variable values, not digital values. Things seem to be quantised at a low enough level. Also: interestingly enough quantum mechanics is both completely deterministic and linear. That means even if it was continuous, you could simulate it to an arbitrary precision without errors building up chaotically. (Figuring out how chaos, as famously observed in the weather, arises in the real world is left as an exercise to the reader. Also a note: the Copenhagen interpretation introduces non-determinism to _interpret_ quantum mechanics but that's not part of the underlying theory, and there are interpretations that have no need for this crutch.)
- tsimionescu 2y ago
  > Things seem to be quantised at a low enough level. Some things. But others are very much not: in particular, space and time are not quantized, and in fact are not even quantizable. A theory in which there is some discrete minimal unit of space (or of spacetime) is trivially incompatible with special relativity, so it is incompatible with quantum mechanics (QFT, specifically). This is easy to see from the nature of the Lorenz transformation: if two objects are at a distance D = n*min for some observer, they will be at a distance D' = n*min*gamma for some other observer, where gamma is always < 1 for an o server moving at a higher speed in a direction aligned with the two objects. So the distance for that second observer will be a non-integer multiple of the minimum distance, so your theory is no longer quantized.

kazinator 2y ago

[dead]

mxkopy 2y ago

I think the complication here is that brains are probabilistic, which admits the possibility that they can’t be directly related to non probabilistic computability classes. I think there’s a paper I forget the name of that says quantum computers can decide the halting problem with some probability (which makes sense because you could always just flip a coin and decide it with some probability) - maybe brains are similar

PaulDavisThe1st 2y ago

> We think, that is a fact. It likely is a fact, but we don't really know what we mean by "think". LLMs have illuminated this point from a relatively new direction: we do not know if their mechanism(s) for language generation are similar to our own, or not. We don't really understand the relationship between "reasoning" and "thinking". We don't really understand the difference between Kahneman's "fast" and "slow" thinking. Something happens, probably in our brains, that we experience and that seems causally prior to some of our behavior. We call it thinking, but we don't know much about what it actually is.

jay_kyburz 2y ago
I don't think its useful or even interesting to talk about AI in relation to how humans think, or whether or not they will be "conscious" whatever that might mean. AIs are not going to be like humans because they will have perfect recall of a massive database of facts, and be able to do math well beyond any human brain. The interesting question to me is, when will we be able to give AI very large tasks, and when will it to be able to break the tasks down into smaller and smaller tasks and complete them. When will it be able to set its own goals, and know when it has achieved them? When will it be able to recognize that it doesn't know something and do the work to fill in the blanks. I get the impression that LLMs don't really know what they are saying at the moment, so don't have any way to test what they are saying is true or not.
- unparagoned 2y ago
  There was a study where they trained a model to lie. When they looked at what was happening internally they could see it knew the truth but just switched things at the outer nodes to lie.
agarren 2y ago
I think we have a pretty good idea that we are not stochastic parrots - sophisticated or not. Anyone suggesting that we’re running billion parameter models in order to bang out a snarky comment is probably trying to sell you something (and crypto’s likely involved.) I think you’re right, LLMs have demonstrated that relatively sophisticated mathematics involving billions of params and an internet full of training data is capable of some truly, truly, remarkable things. But as Penrose is saying, there are provable limits to computation. If we’re going to assume that intelligence as we experience it is computable, then Gödel’s theorem (and, frankly, the field of mathematics) seems to present a problem.

agarren 2y ago

Gödel’s incompleteness theorem, and, say, the halting problem seem to fall squarely into the bucket of “basic computability theory” in precisely the way that “we think, that is a fact”, does not (D.A. hat tip) You’re arguing that we know artificial reasoning exists because we are capable of reasoning. This presupposes that reasoning is computable and that we ourselves reason by computation. But that’s exactly what Penrose is saying isn’t the case - you’re saying we’re walking Turing machines, we’re intelligent, so we must be able to effectively create copies of that intelligence. Penrose is saying that intelligence is poorly defined, that it requires consciousness which is poorly understood, and that we are not meat-based computers. Your last question misses the point completely. “If we are something else, then out CoT won’t be computable…” it’s like you’re almost there but you can’t let go of “we are meat-machines, everything boils down to computation, we can cook up clones”. Except, “basic computability theory” says that’s not even wrong.

godelski 2y ago

> Personally, I prefer to think about it in terms of basic computability theory: Gödel's incompleteness theorem applies to computing. I'm sure you're familiar with the Halting Problem. Gödel's applies to any axiomatic system. The trouble is, it's very hard to make a system without axioms. They are sneaky and it's different than any logic you're probably familiar with. And don't forget Church-Turing, Gödel Numbers, and all the other stuff. Programming is math and Gödel did essential work on the theory of computation. It would be weird NOT to include his work in this conversation. > are we at most just Turing machines or something else? But this is a great question. Many believe no. Personally I'm unsure, but lean no. Penrose is a clear no but he has some wacky ideas. Problem is, it's hard to tell a bad wacky idea from a good wacky idea. Rephrasing Clarke's Second Law: Genius is nearly indistinguishable from insanity. The only way to tell is with time. But look into things like NARS and Super Turing machines (Hypercomputation). There's a whole world of important things that are not often discussed when it comes to the discussion of AGI. But for those that don't want to dig deep into the math, pick up some Sci-Fi and suspend your disbelief. Star Trek, The Orville and the like have holographic simulations and I doubt anyone would think they're conscious, despite being very realistic. But The Doctor in Voyager or Isaac in The Orville are good examples of the contrary. The Doctor is an entity you see become conscious. It's fiction, but that doesn't mean there aren't deep philosophical questions. Even if they're marked by easy to digest entertainment. Good stories are like good horror, they get under your skin, infect you, and creep in Edit: I'll leave you with another question. Regardless of our Turing or Super-Turing status; is a Turing machine sufficient for consciousness to arise?

red75prime 2y ago
> Regardless of our Turing or Super-Turing status; is a Turing machine sufficient for consciousness to arise? Another question. How do you go about detecting whether consciousness has arisen?
- godelski 2y ago
  It's a great question. Best we got is ~~RLHF~~ Potter's definition: I know it when I see it
2y ago

milesrout 2y ago

>Therefore, there is a function capable of transforming information into "thinked information", or what we usually call reasoning. We know that function exists, because we ourselves are an example of such function. "Thinked information" is a colour not an inherent property of information. The fact that information has been thought is like the fact it is copyrighted. It is not something inherent to the information, but a property of its history. https://ansuz.sooke.bc.ca/entry/23 https://ansuz.sooke.bc.ca/entry/23

eru 2y ago

> Now, the question is: can we create a smaller function capable of performing the same feat? Where does this question come from? Especially where does the 'smaller' requirement come from?

AndyNemmity 2y ago

You're arguing that submarines swim.