9 ms·
Does Reasoning Emerge? Probabilities of Causation in Large Language Models
- doe_eyes 2y agoThis is proposed as a way to measure "true" reasoning by asking a certain type of trick questions, but I don't quite see how this could be a basis of a sustainable benchmark. If this gets attention, the next generation of LLMs will be trained on this paper, and then fine-tuned by using this exact form of questions to appear strong on this benchmark, and... we're back to square one.
- jcoc611 2y agoFrom an external perspective, is there a way to distinguish between simulation of consciousness and the real thing? If the answer is no, could you make an argument that they are the same?
- chx 2y agoThrowing all the paintings made prior 1937 into an LLM would never get Guernica out of it. As long as it's an LLM this stands, not just today but all the way to the future. This empty sophistry of presuming automated bullshit generators somehow can mimic a human brain is laughable. Please please read https://aeon.co/essays/your-brain-does-not-process-information-and-it-is-not-a-computer https://aeon.co/essays/your-brain-does-not-process-informati...
- deleted 2y ago[deleted]
- altruios 2y agoThe author fails to provide any argument other than one of incredulity and some bad reasoning with bad faith examples. The dollar bill copying example is a faulty metaphor. His claim of humans not being information processors and he tries to demonstrate this by having a human process information (drawing from reference is processing an image and giving an output)... His argument sounds like one from 'it's always sunny'. As if metaphors never improve or get more accurate over time, and that this latest metaphor isn't the most accurate metaphor we have. It is. When we have something better: we'll all start talking about the brain in that frame of reference. This is an idiot that can write in a way that masks some deep bigotries (in favor of the mythical 'human spirit'). I do not take this person seriously. I'm glossing over all casual incorrectness of his statements - a good number of them just aren't true. the ones I just scrolled to statements like... 'the brain keeps functioning or we disappear' or 'This might sound complicated, but it is actually incredibly simple, and completely free of computations, representations and algorithms' in the description of the 'linear optical trajectory' ALGORTHIM (a set of simple steps to follow - in this case - visual pattern matching). Where is the sense in what I just read?
- chx 2y agoAI bros never take anything seriously aside from the bullshit they are drunk on https://disconnect.blog/what-comes-after-the-ai-crash/ https://disconnect.blog/what-comes-after-the-ai-crash/ this is your future
- __loam 2y agoWhy are you bringing up metaphysics when the concern is that the student has seen the exam, so to speak.
- aflukasz 2y agoI like this observation. And it fascinates me each time I see some self proclaimed conscious entity arguing that this just simply cannot be.
- cscurmudgeon 2y agoBecause you don’t see X is not a proof that X doesn’t exist. Here X may or not exist. X = difference between simulated and real consciousness Black holes were posited before they were detected empirically. We don't declare them to be non-existent when their theory came out just because we couldn't detect them.
- altruios 2y ago> self proclaimed conscious entity Well, I do not proclaim consciousness: only the subjective feeling of consciousness. I really 'feel' conscious: but I can't prove or 'know' that in fact I am 'conscious' and making choices... to be conscious is to 'make choices'... Instead of just obeying the rules of chemistry and physics... which YOU HAVE TO BREAK in order to be conscious at all (how can you make a choice at all if you are fully obeying the rules of chemistry {which have no choice}). A choice does not apply to chemistry or physics: from where does choice come from - I suspect from our fantasies and nothing from objective reality (for I do not see humans consistently breaking the way chemistry works in their brains) - it probably comes from nowhere. If you can explain the lack of choice available in chemistry first (and how that doesn't interfere with us being able to make a choice): then I'll entertain the idea that we are conscious creatures. But if choice doesn't exist at the chemical level, it can't magically emerge from following deterministic rules. And chemistry is deterministic not probabilistic (h2 + o doesn't magically make neon ever, or 2 water molecules instead of one).
- jrflowers 2y agoYou could make the argument that two things that we don’t understand are the same thing because we’re equally ignorant of both in the same way that you could make the argument that Jimmy Hoffa and Genghis Khan are probably buried in the same place, since we have equal knowledge of their locations.
- gorjusborg 2y agoLike the original Mechanical Turk. Clearly there is a difference between a small person hidden within playing chess and a fully mechanical chess automaton, but as the observer we might not be able to tell the difference. The observer's perception of the facts doesn't change the actual facts, and the implications of those facts.
- mannykannot 2y agoThe Mechanical Turk, however, was not a simulation of human consciousness, reasoning, chess-playing or any other human ability: it was the real thing, somewhat artfully dressed-up as to appear otherwise. Is it meaningful to say that Alphago Zero does not play Go, it just simulates something that does?
- mensetmanusman 2y agoThere might not be an external perspective, just someone else’s internal perspective of the external.
- layer8 2y agoConsciousness and reasoning are orthogonal to each other.
- deleted 2y ago[deleted]
- ben_w 2y agoI suspect that depends on which of the 200 definitions of "consciousness" you're using. And some other broad range of definitions of "reasoning".
- andrewla 2y agoThere's an interesting paper [1] that discusses this very possibility. [1] https://academic.oup.com/mind/article/LIX/236/433/986238?login=false https://academic.oup.com/mind/article/LIX/236/433/986238?log...
- imtringued 2y agoYou can shuffle the choices in multiple choice based benchmarks and models that memorize the benchmark tend to do really badly, almost worse than random guessing.
- slashdave 2y agoEasily overcome by training models on randomized questions. Trivial to implement too.
- tsukikage 2y agoIf the model works correctly on way more questions than there is room to store a giant list of recorded answers for, some kind of deduction of generalised rules must have taken place. "there is room to represent recorded answers for" is doing a lot of work, of course; it might e.g. have invented compression mechanisms better than known ones instead.
- slashdave 2y agoYou cannot create generalized questions using randomization alone
- Jensson 2y agoIt does system 1 thinking, it doesn't do system 2 thinking. That makes it really dumb, but can still answer a very wide range of questions it hasn't seen exact matches of since system 1 thinking can pattern match in complex ways. > it might e.g. have invented compression mechanisms better than known ones instead. You mean humans? Humans invented the transformer architecture, that is what compresses the human text to this form where semantics of text gets encoded instead of the raw words.
- altruios 2y agoMaybe: there is no measurable difference between 'real' reasoning, and 'fake' reasoning. And if there is no measurable difference... we can't measure 'realness', we just have to measure something different (and more useful) 'soundness'. Regardless of it is reasoning or not internally, if it produces a sound and logical argument: who cares? I agree: I don't think any measure tested linguistically can prove it is internally reasoning... in the same way we haven't truly proven other sentient people aren't in fact zombies (we just politely assume the most likely case that they aren't).
- doe_eyes 2y ago> Maybe: there is no measurable difference between 'real' reasoning, and 'fake' reasoning. The point is, it's easier to teach an LLM to fake it than to make it - for example, they get good at answering questions that overlap with their training data set long before they start generalizing. So on some epistemological level, your point is worth pondering; but more simply, it actually matters if an LLM has learned to game a benchmark vs approximate human cognition. If it's the former, it might fail in weird ways when we least expect it.
- visarga 2y agoIt's like students learning for the test, not really understanding. Or like regular people who don't always understand, just follow the memorized steps. How many times do we really really understand and how many do we just imitate? I have a suspicion that humans often use abstractions or methods they don't understand. We frequently rely on heuristics, mental shortcuts, and received wisdom without grasping the underlying principles. To understand has many meanings: to predict, control, use, explain, discover, model and generalize. Some also add "to feel". In one extreme we could say only a PhD in their area of expertise really understands, the rest of us just fumble concepts. I am sure rigorous causal reasoning is only possible by extended education, it is not the natural mode of operation of the brain.
- Jensson 2y ago> I am sure rigorous causal reasoning is only possible by extended education, it is not the natural mode of operation of the brain. I'd say the other way around, education teaches you to not reason and instead just follow the patterns you learned in the book. Most people do reason a ton before they go to school, but then school beats that out of them.
- mannykannot 2y agoI don't think we can just assume that training with this exact form of questioning will lead to a strong performance on such questions. For one thing, given the LLM propensity for hallucinating, I do not think we can be confident that an LLM, after this training, will reliably employ the correct model to answer a given question.
- slashdave 2y agoThere seems to be the implicit (and unspoken) assumption that these probability terms (PN,PS) are all independent. However, clearly they are not.
- croes 2y agoRelated https://news.ycombinator.com/item?id=41233206 https://news.ycombinator.com/item?id=41233206
- layer8 2y agoMy impression is that LLMs “pattern-match” on a less abstract level than general-purpose reasoning requires. They capture a large number of typical reasoning patterns through their training, but it is not sufficiently decoupled, or generalized, from what the reasoning is about in each of the concrete instances that occur in the training data. As a result, the apparent reasoning capability that LLMs exhibit significantly depends on what they are asked to reason about, and even depends on representational aspects like the sentence patterns used in the query. LLMs seem to be largely unable to symbolically abstract (as opposed to interpolate) from what is exemplified in the training data.
- arketyp 2y agoI agree. Although I wonder sometimes how much "about" is involved in my own abstract reasoning.
- astromaniak 2y agoFor some reasons LLMs get a lot of attention. But.. while simplicity is great it has limits. To make model reason you have to put it in a loop with fallbacks. It has to try possibilities and fallback from false branches. Which can be done on a level higher. This can be either algorithm, another model, or another thread in the same model. To some degree it can be done by prompting in the same thread. Like asking LLMs to first print high level algorithm and then do it step by step.
- layer8 2y agoIteration is important, but I don’t think that it can substantively compensate for the abstraction limitations outlined in the GP comment.
- refulgentis 2y ago> To make model reason you have to put it in a loop with fallbacks Source? TFA, i.e. the thing we're commenting on, tried to, and seems to, show the opposite
- 2y ago
- layer8 2y agoRegarding AI reasoning and abstraction capabilities, the ARC Prize competition is an interesting project: https://arcprize.org/ https://arcprize.org/
- w10-1 2y agoTheir hypothesis is a good one: - A form of reasoning is to connect cause and effect via probability of necessity (PN) and the probability of sufficiency (PS). - You can identify when the natural language inputs can support PN and PS inference based on LLM modeling That would mean you can engineer in more causal reasoning based on data input and model architecture. They define causal functions, project accuracy measures (false positives/negatives) onto factual and counter-factual assertion tests, and measure LLM performance wrt this accuracy. They establish surprisingly low tolerance for counterfactual error rate, and suggest it might indicate an upper limit for reasoning based on current LLM architectures. Their findings are limited by how constrained their approach is (short simple boolean chains). It's hard to see how this approach could be extended to more complex reasoning. Conversely, if/since LLM's can't get this right, it's hard to see them progressing at the rates hoped, unless this approach somehow misses a dynamic of a larger model. It seems like this would be a very useful starting point for LLM quality engineering, at least for simple inference.
- latentnumber 2y ago> It seems like this would be a very useful starting point for LLM quality engineering, at least for simple inference. Interesting. Can you elaborate on this? You mean this test can function as a metric or is it just an evaluation for applications?
- heyjamesknight 2y agoLLMs have access to the space of collective semantic understanding. I don't understand why people expect cognitive faculties that are clearly extra-semantic to just fall out of them eventually. The reason they sometimes appear to reason is because there's a lot of reasoning in the corpus of human text activity. But that's just a semantic artifact of a non-semantic process. Human cognition is much more than just our ability to string sentences together.
- matt-attack 2y agoI don’t know if you’re correct. I don’t think you know that our brains are that different? We too need to train ourselves on massive amounts of data. I feel like the kids of reasoning and understanding I’ve seen ChatGPT do are soooo far beyond something like just processing language.
- Minor49er 2y agoWhat reasoning have you seen coming from ChatGPT?
- thuuuomas 2y agoWe do represent much of our cognition in language. Sometime I feel like LLMs might be “dancing skeletons” - pulleys & wire giving motion to the bones of cognition.
- andai 2y agoBut why stop there? All matter and all life is just increasingly fancy machines. https://en.wikipedia.org/wiki/Philosophical_zombie https://en.wikipedia.org/wiki/Philosophical_zombie Editor's note: I do not promote such a worldview -- my intention is precisely the opposite.
- sgt101 2y agoOur brains have effects that proceeded language. Look at lions for an example. We are much more (and a little less) than lions in terms of mind.
- abcde777666 2y agoThere are many pillars of our own intelligence that we tend to gloss over. For instance - awareness and the ability to direct attention. Or something as simple as lifting your hand and moving some fingers at will. Those things impress me far more than the noises we produce with our mouths!
- kantapproves 2y agoI believe Hume (and Kant) have some things to say about this. The connection might need some fleshing out, but I believe, and I might be wrong here, it was decided a few centuries ago that probabilities alone cannot explain causality. It would be a hoot, wouldn’t it? Perhaps AI just need some a priori synthetics to spruce it up.
- sweeter 2y agoprobability? None imo.
- rockskon 2y agoSimple answer to the question posed by the headline: No. As much as Google, Microsoft, OpenAI, and every other company that's poured billions into this technology want to think otherwise - more training data will not turn your AI model into AGI. Any argument to the contrary is copium.
- IgorPartola 2y agoSo the best way I can describe how humans abstract our thinking is that “a thought about a thought is itself a thought”. I am not an expert but I don’t believe LLMs can arbitrarily abstract their current “thought”, put it down for later contemplation, trace back to a previous thought or a random thought, and out of these individual thoughts form an understanding. I would expect that a higher level algorithm would be required to string together thoughts into understandings. Then again, I wonder if what we are going to see is fundamentally different kinds of intelligences that just do not necessarily think like humans. Chimps cannot tell you about last Tuesday since their memory seems a lot more associative than recall based. But they have situational awareness that even our superheroes in our comics do not generally posses (flash some numbers in front of a chimp for one second and he will remember all their positions and order even if you distract him immediately after). Maybe LLMs cannot be human intelligent but you could argue that they are a kind of intelligence.