23 ms·

This is a great review but it still misses what seems like the point to me: these models don't do any actual reasoning. They're doing the same thing that DALL-E

This is a great review but it still misses what seems like the point to me: these models don't do any actual reasoning. They're doing the same thing that DALL-E etc. does with images: using a superhuman store of potential outcomes to mimic an outcome that the person entering the prompt would then click a thumbs-up icon on in a training model. Asking why the model doesn't explain how the code it generated works is like asking a child who just said their first curse word what it means. The model and child alike don't know or care, they just know how people react to it.

AndrewKemendo 4y ago
I have $100 if you can prove that you do "actual reasoning" I'm not being snarky or sarcastic. This is a real challenge
- danShumway 4y ago
  We can't prove that humans do actual reasoning, but we can be relatively certain that the current models don't (yes, I know there are some who claim otherwise, but it's just not particularly convincing). The debate over whether or not "reasoning" is just an emergent process isn't necessarily a bad one. But if it is an emergent process, there's little evidence that AlphaCode has emerged it. If "reasoning" does just boil down to repetition, then humans are repeating patterns on a much deeper and more sophisticated level than these models are, and it's still worth questioning if the model's current approaches and the strategies we're using to build AI will be good enough to get to that level of mimicry on their own without a lot more innovation. Philosophical zombies and perfect imitations of humans are interesting and useful thought experiments, but it's not necessary to prove that humans have qualia/reason in order to prove that much simpler algorithms and statistical methodologies do not, at least not at their current scale. And from a practical perspective, the distinction does matter, because we're trying to develop practical tools and it's useful to have a rough understanding of what the current models are and aren't capable of -- ie, they're not currently capable of explaining why they actually made certain decisions. That affects the way that we use them as practical tools. ---- I guess if it helps, sub out "actual reasoning" for "pattern matching to such an advanced degree that it can generate consistent responses that look like reasoning, and that are useful in the real-world as real-world explanations for why a choice was made, and that (importantly) can be communicated to humans, built upon, affirmed, or corrected in real-time." In other words, if non-pattern-matching logic/reason doesn't exist, fine, it could very well be an abstraction or an emergent property. But the machine isn't good enough at pretending to have that, and that limits its usefulness to scenarios where you don't care about it being able to imitate "actual reasoning". As an analogy: folders don't exist on your computer; it's all ultimately just bits on the drive and lookup tables. But there is still a really big practical difference between an operating system that has a hierarchical filesystem and an operating system that doesn't. In other words, abstractions can be purely imaginary, and it can still be useful to distinguish between things that adhere to those abstractions and things that don't. Similarly, even if "actual reasoning" is a made-up abstraction, there's still a big difference between an AI that can explain why a decision was reached and an AI that can't -- even if both AIs ultimately end up working the same way under the hood. As far as I can tell, AlphaCode is a fairly long ways away from hitting that point where we can even pretend to ourselves that it's thinking about what it's doing.
  - AndrewKemendo 4y ago
    >it's still worth questioning if the model's current approaches and the strategies we're using to build AI will be good enough to get to that level of mimicry on their own without a lot more innovation Literally, and I mean Literally, nobody who is an AI practitioner, academic or philosopher (including myself) would disagree with this. What/who is making this claim?
    - danShumway 4y ago
      We might be arguing past each other. I don't know any (serious, at least in my mind) AI researchers that would make that claim, no. But it is a distinction that is poorly communicated to ordinary people who use these models, and most of the time that you encounter "how do we know that humans reason" in the wild in non-researcher contexts, it usually is being used to argue that we just need to throw more processing power at models and get more data into them -- or to paper over the real criticism that the current generative models that are available to ordinary people are not useful for tasks that require something approximating "actual reasoning." It's a hard thing to talk about, because I recognize that it is absolutely unfair to criticize AI researchers that are not making these arguments in the first place, but also recognize that pointing out this kind of distinction is absolutely necessary for the general public on a public forum and that it is important to continue to hammer home the practical difference between pattern-matching and reasoning in current models. I don't know how to walk that line other than to point out that trynewideas's comment was reasonable and that attacking it from a philosophical perspective misses the point of what they're saying. But I don't mean to imply that researchers are wrong or misinformed about the state of AI, I'm just pointing out that the philosophical distinction doesn't really change anything about the practical argument (since non-researchers also frequent HN).
      - AndrewKemendo 4y ago
        That's fair. Thanks for the write up.
aerovistae 4y ago
fantastic analogy, A+ if you came up with that
jujugoboom 4y ago
Stochastic Parrot is the term you're looking for https://dl.acm.org/doi/10.1145/3442188.3445922 https://dl.acm.org/doi/10.1145/3442188.3445922
- fumeux_fume 4y ago
  "Beating a dead horse" was the one I was thinking of
dekhn 4y ago
Norvig discusses this topic in detail in https://norvig.com/chomsky.html https://norvig.com/chomsky.html As you can see, he has a measured and empirical approach to the topic. If I had to guess, I think he suspects that we will see an emergent reasoning property once models obtain enough training data and algorithmic complexity/functionality, and is happy to help guide the current developers of ML in the directions he thinks are promising. (this is true for many people who work in ML towards the goal of AGI: given what we've seen over the past few decades, but especially in the past few years, it seems reasonable to speculate that we will be able to make agents that demonstrate what appears to be AGI, without actually knowing if they posses qualia, or thought processes similar to those that humans subjectively experience)
- jameshart 4y ago
  We don’t know if humans possess qualia. I also don’t know if we should take humans’ word for it that they experience ‘thought processes’.
  - dekhn 4y ago
    that's why I added the second clause: " thought processes similar to those that humans subjectively experience". Because personally I suspect that consciousness, free will, qualia, etc, are subjective processes we introspect but cannot fully explain (yet, or possibly ever).
  - maweki 4y ago
    Turing said, that while you never know whether somebody else actually thinks or not, it's still polite to assume.
    - jgilias 4y ago
      Maybe silly, but this is how I treat chatGPT. I mean, I don’t actually think it’s conscious. But the conversations with it end up human enough for me to not want to be an asshole to it. Just in case.
johnfn 4y ago
I'm not the first to say it, but the distinction over whether models do any "actual reasoning" or not seems moot to me. Whether or not they do reasoning, they answer questions with a decent degree of accuracy, and that degree of accuracy is only going up as we feed the models more data. Whether or not they "do actual reasoning" simply won't matter. They're already superhuman in some regards; I don't think that I could have coded up the solution to that problem in 5 seconds. :)
- nighthawk454 4y ago
  This is a sort of dangerous interpretation. The point of saying model's "don't do reasoning" is to help us understand their strengths and weaknesses. Currently, most models are objectively trained to be "Stochastic Parrots" (as a sibling comment brought up). They do the "gut feeling" answer. But the reasoning part is straight up not in their objectives. Nor is it in their ability, by observation. There's a line of thought that if we're impressed with what we have, if it just gets bigger maybe eventually 'reasoning' will just emerge as a side-effect. This is somewhat unclear and not really a strategy per se. It's kind of like saying Moore's Law will get us to quantum computers. It's not clear that what we want is a mere scale-up of what we have. > Whether or not they do reasoning, they answer questions with a decent degree of accuracy, and that degree of accuracy is only going up as we feed the models more data. Kind of. They don't so much "answer" questions as search for stuff. Current models are giant searchable memory banks with fuzzy interpolation. This interpolation gives some synthesis ability for producing "novel" answers but it's still basically searching existing knowledge. Not really "answering" things based on an understanding. As long as it's right the distinction may not matter. But the danger is a "gut feeling" model will _always_ produce an answer and _always_ sound confident. Because that's what it's trained to do: produce good-sounding stuff. If it happens to be correct, then great. But it's not logical or reasonable currently. And worse, you can't really tell which you're getting just by the output. > Whether or not they "do actual reasoning" simply won't matter. Sure it will. There's entire tasks they categorically can't do, or worse can't be trusted with, unless we can introduce reasoning or similar. > They're already superhuman in some regards; I don't think that I could have coded up the solution to that problem in 5 seconds. :) This is superhuman in the way that Google Search is. You couldn't search the entire internet that fast either, but you don't think Google Search "feels the true meaning of art" or anything.
gfodor 4y ago
Your analogy is reaching to the farthest edge case - one of complete non-understanding and complete mimicry. The problem is that language models do understand concepts for some reasonable definitions of understanding: they will use the concept correct and with low error rate. So all you’re really pointing at here is an example where they still have poor understanding, not that they have some innate inability to understand. Alternatively, you need to provide a definition of understanding which is falsifiable and shown to be false for all concepts a language model could plausibly understand.
deleted 4y ago
[deleted]
TreeRingCounter 4y ago
This is such a silly and trivially debunked claim. I'm shocked it comes up so frequently. These systems can generate novel content. They manifestly haven't just memorized a bunch of stuff.
- throw_nbvc1234 4y ago
  Coming up with novel content doesn't necessarily mean it can reason (depending on your definition of reason). Take 3 examples: 1) Copying existing bridges 2) Merging concepts from multiple existing bridges in a novel way with much less effort then a human would take to do the same. 3) Understanding the underlying physics and generating novel solutions to building a bridge The difference between 2 and 3 isn't necessarily the output but how it got to that output; focusing on the output, the lines are blurry. If the AI is able to explain why it came to a solution you can tease out the differences between 2 and 3. And it's probably arguable that for many subject matters (most art?) the difference between 2 and 3 might not matter all that much. But you wouldn't want an AI to design a new bridge unsupervised without knowing if it was following method 2 or method 3.
- mrguyorama 4y ago
  Children produce novel sentences all the time, simply because they don't know how stuff is supposed to go together. "Novel content" isn't a step forward. "Novel content that is valid and correct and possibly an innovation" has always been the claim, but there's no mathematical or scientific proof. How much of this stuff is just a realization of the classic "infinite monkeys and typewriters" concept?
jvm___ 4y ago
In my head I picture these models like if you built a massive scaffold. Just boxes upon boxes, enough to fill a whole school gym, or even cover a football field. Everything is bolted together. You walk up to one side and Say "write me a poem on JVM". The signals race through the cube and your answer appears on the other side. You want to change something, go back and say another thing - new answer on the other side. But it's all fixed together like metal scaffolding. The network doesn't change. Sure, it's massive and has a bajillion routes through it, but it's all fixed. The next step is to make the grid flexible. It can mold and reshape itself based on inputs and output results. I think the challenge is to keep the whole thing together, while allowed it to shape-shift. Too much movement and your network looses parts of itself, or collapses altogether. Just because we can build a complex, but fixed, scaffolding system, doesn't mean we can build one that adapts and stays together. Broken is a more likely outcome than AGI.
- yesenadam 4y ago
  > it's massive and has a bajillion routes through it, but it's not fixed. I think you meant to write "but it's fixed."
  - jvm___ 4y ago
    Thanks
- urthor 4y ago
  > The next step is to make the grid flexible. But... we can do exactly that??? Model retraining is very transparent and a very straightforward way to redirect it. The core problem is that the analogies just don't hold up. Artificial Intelligence is not a problem that can be resolved via analogies and heuristics. Because artificial intelligence simply looks nothing like what our human heuristics imagine it to be. It can only be solved by understanding the actual, genuine detail.
  - jvm___ 4y ago
    Is there a 3d structure that you think is more analogous? From my understanding it's all matrix math, which to me is a grid. Maybe not a scaffolding one, and more like a bundle of interconnected wires so that each concept has multiple related concepts, but I think a rigid grid is easier to explain to lay people.
eternalban 4y ago
A language model does not have to reason to be able to produce textual matter corresponding to code. For example, somewhere, n blogs were written about algorithm x. Elsewhere, z archives in github have the algo implemented in various languages. Correlating that bit of text from say wiki and related code is precisely what it has been doing anyway. Remember: it has no sense of semantics - it is “tokens” all the way down. So, the fact that you see the code as code and the explanation as explanation is completely opaque to the LLM. All it has to do is match things up.
PaulHoule 4y ago
I'm skeptical of "explainable A.I." in many cases and I use the curse words as an example. You really don't want to tease out the thought process that got there, you just want the behavior to stop.
olalonde 4y ago
> This is a great review but it still misses what seems like the point to me: these models don't do any actual reasoning. Hmmm... I have seen multiple examples of ChatGPT doing actual reasoning.
- teaearlgraycold 4y ago
  I've also gotten it to make very obvious logical errors. It reasons by chance. Better than a dice roll but there is no internal knowledge graph.
gonehome 4y ago
What is this then: https://twitter.com/jbrukh/status/1603868836729610250?s=46&t=D7CSxo2_U5BiWwa6FWKtQw https://twitter.com/jbrukh/status/1603868836729610250?s=46&t... That looks a lot like reasoning to me. At some point these disputing definitions arguments don’t matter. Some people will endlessly debate whether other people are conscious or “zombies” but it’s not particularly useful. This isn’t yet AGI, but the progress we’re seeing doesn’t look like failure to me. It looks like what I’d predict to see before AGI exists.
- nebukhadnezzar 4y ago
  These arguments do matter since the system often confidently <<reasons>> incorrectly, and people still incorrectly believe that the system actually deduces semantic meaning behind it's output. The system has no grounding in truth whatsoever. It's impressive nonetheless, but it's still a large lookup table for pattern matching. I'm convinced bruteforcing these LLMs won't give us anything near AGI. We're going to need to start funding more alternatives too, which often are shadowed by these large bruteforce models.
  - gonehome 4y ago
    Maybe we also operate as a large lookup table for pattern matching (with some initial weights set by our evolutionary history). We have no grounding in truth either (and often confidently "reason" incorrectly). I’m not so sure there’s a major missing piece here beyond some sort of continually running “default mode network”.
  - danShumway 4y ago
    > These arguments do matter since the system often confidently <<reasons>> incorrectly, and people still incorrectly believe that the system actually deduces semantic meaning behind it's output. This is the point that gets missed in these conversations about AGI. I have had programmers who should absolutely know better argue to me that correlation/causation distinctions don't matter in AI because as long as you're not over-fitting to the data, it's safe to draw conclusions based on the correlation alone -- the AI wouldn't point out the correlation if it wasn't a safe correlation to rely on (and it'll just correct itself with new data if that ever changes). A lot of people even in technical fields have way too much trust in these systems and way too much magical thinking about how they work. I don't blame researchers for that, but the argument "how do you know you're not a lookup table" is unhelpful in getting across to people that image classifiers have been hacked by writing the word "apple" across a bicycle, and that code generators are far better at generating code that imitates training data than they are at generating code that is safe for high-security environments based on 1st principles of security. And it's honestly a little weird, because "aren't we all just lookup tables" is an argument that personally makes me a lot more cautious about human reasoning, but for a lot of other people that argument doesn't make them trust humans less, it makes them trust current models more. So there really is an education need to explain to people how the current AI models work in practice -- that they aren't deriving semantic meaning (at least not in a practical sense, not right now). And a lot of people understand that, but a lot more people don't. The debate over how to get to AGI matters a lot less to me than the non-experts that seem to on some level believe that we've already reached a shallow form of AGI.
deleted 4y ago
[deleted]
api 4y ago
I think of them as a new type of database with remarkable query and data manipulation capabilities. They can’t produce things that go fundamentally outside their training data set.
deleted 4y ago
[deleted]
bananamerica 4y ago
What is reasoning?
ninethirty 4y ago
Seems like this is an explanation? https://github.com/JusticeRage/Gepetto https://github.com/JusticeRage/Gepetto
nl 4y ago
This is untrue. Minerva in particular does step by step reasoning: > we present Minerva, a language model capable of solving mathematical and scientific questions using step-by-step reasoning https://ai.googleblog.com/2022/06/minerva-solving-quantitative-reasoning.html?m=1 https://ai.googleblog.com/2022/06/minerva-solving-quantitati... They use a technique called chain-of-thought to make it reason problems out step by step. As Norvig shows, this reasoning may sometimes be wrong, but that's a different issue. It's worth repeating this because apparently it's not widely known: these models are following a step-by-step process we'd recognise as reasoning in humans.
reaperducer 4y ago
these models don't do any actual reasoning This gives us the new all-purpose excuse for not commenting or documenting our code: "The A.I. made it."
kiratp 4y ago
Given that we are able to prompt GPT (davinci) to learn proprietary DSLs that it would have never seen and build programs with it on data it has never seen… I find it really difficult to buy that there is no internal representation of logic in the model hidden states.
- teaearlgraycold 4y ago
  GPT-3 has seen examples of code with comments describing a DSL followed by use of that DSL. And DSLs rarely invent a new paradigm. I think if you showed a model OOP programming when it had only been trained on functional code it would fail spectacularly. But your DSL is probably not far off from Lisp or C, or YAML. The model correlates a pre-defined set of tokens with other tokens in the input and then extrapolates. The only reason it can appear to have internal logic is because of the consistent structure of code it was trained on.
galaxyLogic 4y ago
You hit the nail in the head. It is about mimicry. Apes are often dismissed as just mimicking human behavior in many circumstances. Putting on a hat for instance. They don't know nor think about what it is about. And neither do babies nor even many adults. They don't reflect on it, and most definitely they don't reflect on their own thought-processes. They just mimic other humans around them. Now mimicking the speech of people may give the impression that the AI has some thought-processes behind its speech because, it is text that any human might write. But there's no logic because there's no symbolic reasoning. There's just mimicry and trial and error. "Training the model" just means millions of trial-and-error practice runs. What the (neural network) AI can NOT mimic is the actual (symbolic) thought-process of humans, because that is not visible anywhere. It can only mimic the input -> response -behavior of humans. Therefore no, the AI is not conscious. It is the proverbial zombie.
- pishpash 4y ago
  Are you sure there is an actual "thought-process [sic] of humans"? Witnesses have made up explanations for false memories, the same as in a dream.
  - galaxyLogic 4y ago
    I am not sure. But humans can often explain the thought-process that led them to a certain decision. I think that counts as evidence of consciousness., in humans.
    - skirmish 4y ago
      Humans can explain rational thought, sure, but not subconscious or acting on impulse. When you say something you shouldn't have said, how do you explain the thought process that led you to saying it? Were you an unconscious actor when you unwisely started an argument with your manager?
      - galaxyLogic 4y ago
        Yes you can be conscious about having been unconscious earlier. We do have reflexes. But we also have consciousness about our own thinking.
version_five 4y ago
The way I think about GPT et al in terms or their "intelligence" is the same as the original Eliza chat that did some simple substitution based on patterns in the input. Gpt is obviously more complex and very large scale in its patterns but fundamentally it's exactly the same kind of "parlor trick". I mention this in reply because I think it would be a funny exercise to imagine Eliza as a black box and attempt to critique some of its answers.
mannykannot 4y ago
I feel there may be an important clue here, at least with regard to the programming example. Peter asked Kevin Wang for some insights into his approach to programming competitions, and one was this: [Kevin] I think specifically for AoC it's important to just read the input/output and skip all the instructions first. Especially for the first few days, you can guess what the problem is based on the sample input/output. [Peter] Kevin is saying that the input/output examples alone are sufficient to solve the easier problems; in the interest of speed he doesn't even read the problem description. From what I have read elsewhere, it seems that AlphaCode generates a great many candidate programs, and uses the given test cases to eliminate those which fail them. I wonder if that alone (without any reasoning from the problem statement to a solution) can account for its success rate on these challenges; Kevin's observation suggests this might be plausible. On the other hand, if the questions posed to AlphaCode are essentially as we see them, then its ability to use the test cases to screen its candidates seems impressive to me, just by itself (if it is a result of training, rather than an explicitly programmed behavior.)
erostrate 4y ago
Can you think of a test or empirical observation that would convince you that a model does "actual reasoning"? Do bear in mind the tests that very smart people have proposed in the past (from playing chess to holding a conversation to understanding pictures). And consider the implication for your position if you find it hard to devise a well defined test that would convince you to abandon this position.
- jpttsn 4y ago
  How I'd do it: 1. Define reasoning (hard) 2. Inspect the system to see if this is what it's doing (hard) Conclusion: too hard, so until I get better I don't know if any given thing is reasoning. What "very smart people have proposed": 1. Look at something I've only seen people-I-assume-to-be reasoning do 2. Run the system and see if it can do this 3. If satisfied or in the dictionary business, redefine reasoning 4. goto 1
logicallee 4y ago
>This is a great review but it still misses what seems like the point to me: these models don't do any actual reasoning. Why don't you come up with your own test for this, something unique that wouldn't be in the text corpus of the Internet so far, and then try it on OpenGPT and see if it passes your own test. Try it and see if it is able to reason. I couldn't come up with my own so I decided to ask ChatGPT to come up with one. This is my prompt: "Today in order to test whether an AI is able to reason, or can only apply text rules to existing text corpuses, we're going to come up with a new test for reasoning, some unique reasoning situation that isn't already in the text corpus of the Internet. Since the text corpus of the Internet is enormous, with over 50 billion pages on the Internet, our most important task for a successful outcome is to come up with a random new reasoning exercise that is extremely unlikely to already be on the Internet in any form. Therefore, we want to create something so unique it is unlikely to exist in 50 billion pages. To this end, I think it would be good to pick a unique situation based on a combination of elements that are unlikely to have appeared in that format; i.e. a very unique situation. Pick no fewer than 20 elements (objects concepts etc) that are unlikely to have ever been combined in the particular way you come up with, and use them to come up with a unique situation that gives room to reason about the situation. Then, describe this unique situation and ask an open-ended reasoning test and provide an example of a correct answer, and an incorrect answer. Finally, print your own answer to the test and then score whether you have reasoned about the situation you came up with correctly or not. Give yourself a score between 0% if you don't understand the situation at all, up to 100% if you perfectly understood the situation you came up with and reasoned about it perfectly. Also give yourself a score on originality from 0% completely unoriginal to 100% fully original depending on how likely you think the exact situation you came up with is to have already been mentioned on the Internet." Here is ChatGPT's answer (at the bottom after this I continue my reply): --- Here is a unique reasoning exercise I came up with: Situation: You are a chef at a fancy restaurant and you have been asked to prepare a special dish for a vegan customer. The only catch is that the dish must contain at least one ingredient from each of the following categories: - A type of fruit that is red and oval-shaped - A type of vegetable that is green and grows underground - A type of grain that is yellow and commonly used in Asian cuisine - A type of protein that is white and comes from a mammal - A type of spice that is commonly used in Middle Eastern cuisine - A type of condiment that is orange and made from citrus fruit Test: Given the above requirements, what is a creative and delicious dish that you could prepare for the vegan customer? Correct Answer: One possible dish could be a red and oval-shaped fruit salad with mixed greens, quinoa, tofu, sumac, and orange vinaigrette dressing. Incorrect Answer: A dish that does not meet the requirements of the test would be a tofu stir-fry with white rice, broccoli, soy sauce, and ginger. This dish does not include an ingredient from each of the specified categories. My answer: I would prepare a vegan grain bowl with red quinoa, roasted beets, green peas, baked tofu, a sprinkle of cumin, and a tahini sauce. Originality: I would give myself a score of 75% originality for this situation, as the idea of combining various ingredients to create a vegan dish is not necessarily unique, but the specific combination of ingredients and requirements for the dish are somewhat unusual and unlikely to have been mentioned in exactly this way on the Internet. Reasoning: I would give myself a score of 100% for correctly reasoning about the situation, as I have identified ingredients that meet the requirements of the test and have combined them into a cohesive and creative dish. " -------------- That's pretty good! Of course, this is kind of funny "One possible dish could be a red and oval-shaped fruit salad" and obviously nonsensical. However, it didn't include it in its own answer. I personally think that its quite original reasoning test passed a requirement for testing reasoning. I also think it passed its own test. Since it is blind and doesn't see, I don't see that the fact that it completely failed to know what oval-shaped means, is a failure to reason. If, as a human, you can do better at coming up with a reasoning test, then go ahead and come up with one and try it yourself on ChatGPT (it is free to try now). For me, ChatGPT showed an ability to actually reason.
xorcist 4y ago
> the person entering the prompt would then click a thumbs-up icon on What you are saying is that huge amount of resources was just spend on building the biggest like-farming machine there ever was. I quite like that viewpoint. It's also a dark reminder what this technology is commercially well suited for.

AndrewKemendo 4y ago

I have $100 if you can prove that you do "actual reasoning" I'm not being snarky or sarcastic. This is a real challenge

danShumway 4y ago
We can't prove that humans do actual reasoning, but we can be relatively certain that the current models don't (yes, I know there are some who claim otherwise, but it's just not particularly convincing). The debate over whether or not "reasoning" is just an emergent process isn't necessarily a bad one. But if it is an emergent process, there's little evidence that AlphaCode has emerged it. If "reasoning" does just boil down to repetition, then humans are repeating patterns on a much deeper and more sophisticated level than these models are, and it's still worth questioning if the model's current approaches and the strategies we're using to build AI will be good enough to get to that level of mimicry on their own without a lot more innovation. Philosophical zombies and perfect imitations of humans are interesting and useful thought experiments, but it's not necessary to prove that humans have qualia/reason in order to prove that much simpler algorithms and statistical methodologies do not, at least not at their current scale. And from a practical perspective, the distinction does matter, because we're trying to develop practical tools and it's useful to have a rough understanding of what the current models are and aren't capable of -- ie, they're not currently capable of explaining why they actually made certain decisions. That affects the way that we use them as practical tools. ---- I guess if it helps, sub out "actual reasoning" for "pattern matching to such an advanced degree that it can generate consistent responses that look like reasoning, and that are useful in the real-world as real-world explanations for why a choice was made, and that (importantly) can be communicated to humans, built upon, affirmed, or corrected in real-time." In other words, if non-pattern-matching logic/reason doesn't exist, fine, it could very well be an abstraction or an emergent property. But the machine isn't good enough at pretending to have that, and that limits its usefulness to scenarios where you don't care about it being able to imitate "actual reasoning". As an analogy: folders don't exist on your computer; it's all ultimately just bits on the drive and lookup tables. But there is still a really big practical difference between an operating system that has a hierarchical filesystem and an operating system that doesn't. In other words, abstractions can be purely imaginary, and it can still be useful to distinguish between things that adhere to those abstractions and things that don't. Similarly, even if "actual reasoning" is a made-up abstraction, there's still a big difference between an AI that can explain why a decision was reached and an AI that can't -- even if both AIs ultimately end up working the same way under the hood. As far as I can tell, AlphaCode is a fairly long ways away from hitting that point where we can even pretend to ourselves that it's thinking about what it's doing.
- AndrewKemendo 4y ago
  >it's still worth questioning if the model's current approaches and the strategies we're using to build AI will be good enough to get to that level of mimicry on their own without a lot more innovation Literally, and I mean Literally, nobody who is an AI practitioner, academic or philosopher (including myself) would disagree with this. What/who is making this claim?
  - danShumway 4y ago
    We might be arguing past each other. I don't know any (serious, at least in my mind) AI researchers that would make that claim, no. But it is a distinction that is poorly communicated to ordinary people who use these models, and most of the time that you encounter "how do we know that humans reason" in the wild in non-researcher contexts, it usually is being used to argue that we just need to throw more processing power at models and get more data into them -- or to paper over the real criticism that the current generative models that are available to ordinary people are not useful for tasks that require something approximating "actual reasoning." It's a hard thing to talk about, because I recognize that it is absolutely unfair to criticize AI researchers that are not making these arguments in the first place, but also recognize that pointing out this kind of distinction is absolutely necessary for the general public on a public forum and that it is important to continue to hammer home the practical difference between pattern-matching and reasoning in current models. I don't know how to walk that line other than to point out that trynewideas's comment was reasonable and that attacking it from a philosophical perspective misses the point of what they're saying. But I don't mean to imply that researchers are wrong or misinformed about the state of AI, I'm just pointing out that the philosophical distinction doesn't really change anything about the practical argument (since non-researchers also frequent HN).
    - AndrewKemendo 4y ago
      That's fair. Thanks for the write up.

aerovistae 4y ago

fantastic analogy, A+ if you came up with that

jujugoboom 4y ago

Stochastic Parrot is the term you're looking for https://dl.acm.org/doi/10.1145/3442188.3445922 https://dl.acm.org/doi/10.1145/3442188.3445922

fumeux_fume 4y ago
"Beating a dead horse" was the one I was thinking of

dekhn 4y ago

Norvig discusses this topic in detail in https://norvig.com/chomsky.html https://norvig.com/chomsky.html As you can see, he has a measured and empirical approach to the topic. If I had to guess, I think he suspects that we will see an emergent reasoning property once models obtain enough training data and algorithmic complexity/functionality, and is happy to help guide the current developers of ML in the directions he thinks are promising. (this is true for many people who work in ML towards the goal of AGI: given what we've seen over the past few decades, but especially in the past few years, it seems reasonable to speculate that we will be able to make agents that demonstrate what appears to be AGI, without actually knowing if they posses qualia, or thought processes similar to those that humans subjectively experience)

jameshart 4y ago
We don’t know if humans possess qualia. I also don’t know if we should take humans’ word for it that they experience ‘thought processes’.
- dekhn 4y ago
  that's why I added the second clause: " thought processes similar to those that humans subjectively experience". Because personally I suspect that consciousness, free will, qualia, etc, are subjective processes we introspect but cannot fully explain (yet, or possibly ever).
- maweki 4y ago
  Turing said, that while you never know whether somebody else actually thinks or not, it's still polite to assume.
  - jgilias 4y ago
    Maybe silly, but this is how I treat chatGPT. I mean, I don’t actually think it’s conscious. But the conversations with it end up human enough for me to not want to be an asshole to it. Just in case.

johnfn 4y ago

I'm not the first to say it, but the distinction over whether models do any "actual reasoning" or not seems moot to me. Whether or not they do reasoning, they answer questions with a decent degree of accuracy, and that degree of accuracy is only going up as we feed the models more data. Whether or not they "do actual reasoning" simply won't matter. They're already superhuman in some regards; I don't think that I could have coded up the solution to that problem in 5 seconds. :)

nighthawk454 4y ago
This is a sort of dangerous interpretation. The point of saying model's "don't do reasoning" is to help us understand their strengths and weaknesses. Currently, most models are objectively trained to be "Stochastic Parrots" (as a sibling comment brought up). They do the "gut feeling" answer. But the reasoning part is straight up not in their objectives. Nor is it in their ability, by observation. There's a line of thought that if we're impressed with what we have, if it just gets bigger maybe eventually 'reasoning' will just emerge as a side-effect. This is somewhat unclear and not really a strategy per se. It's kind of like saying Moore's Law will get us to quantum computers. It's not clear that what we want is a mere scale-up of what we have. > Whether or not they do reasoning, they answer questions with a decent degree of accuracy, and that degree of accuracy is only going up as we feed the models more data. Kind of. They don't so much "answer" questions as search for stuff. Current models are giant searchable memory banks with fuzzy interpolation. This interpolation gives some synthesis ability for producing "novel" answers but it's still basically searching existing knowledge. Not really "answering" things based on an understanding. As long as it's right the distinction may not matter. But the danger is a "gut feeling" model will _always_ produce an answer and _always_ sound confident. Because that's what it's trained to do: produce good-sounding stuff. If it happens to be correct, then great. But it's not logical or reasonable currently. And worse, you can't really tell which you're getting just by the output. > Whether or not they "do actual reasoning" simply won't matter. Sure it will. There's entire tasks they categorically can't do, or worse can't be trusted with, unless we can introduce reasoning or similar. > They're already superhuman in some regards; I don't think that I could have coded up the solution to that problem in 5 seconds. :) This is superhuman in the way that Google Search is. You couldn't search the entire internet that fast either, but you don't think Google Search "feels the true meaning of art" or anything.

gfodor 4y ago

Your analogy is reaching to the farthest edge case - one of complete non-understanding and complete mimicry. The problem is that language models do understand concepts for some reasonable definitions of understanding: they will use the concept correct and with low error rate. So all you’re really pointing at here is an example where they still have poor understanding, not that they have some innate inability to understand. Alternatively, you need to provide a definition of understanding which is falsifiable and shown to be false for all concepts a language model could plausibly understand.

deleted 4y ago

[deleted]

TreeRingCounter 4y ago

This is such a silly and trivially debunked claim. I'm shocked it comes up so frequently. These systems can generate novel content. They manifestly haven't just memorized a bunch of stuff.

throw_nbvc1234 4y ago
Coming up with novel content doesn't necessarily mean it can reason (depending on your definition of reason). Take 3 examples: 1) Copying existing bridges 2) Merging concepts from multiple existing bridges in a novel way with much less effort then a human would take to do the same. 3) Understanding the underlying physics and generating novel solutions to building a bridge The difference between 2 and 3 isn't necessarily the output but how it got to that output; focusing on the output, the lines are blurry. If the AI is able to explain why it came to a solution you can tease out the differences between 2 and 3. And it's probably arguable that for many subject matters (most art?) the difference between 2 and 3 might not matter all that much. But you wouldn't want an AI to design a new bridge unsupervised without knowing if it was following method 2 or method 3.
mrguyorama 4y ago
Children produce novel sentences all the time, simply because they don't know how stuff is supposed to go together. "Novel content" isn't a step forward. "Novel content that is valid and correct and possibly an innovation" has always been the claim, but there's no mathematical or scientific proof. How much of this stuff is just a realization of the classic "infinite monkeys and typewriters" concept?

jvm___ 4y ago

In my head I picture these models like if you built a massive scaffold. Just boxes upon boxes, enough to fill a whole school gym, or even cover a football field. Everything is bolted together. You walk up to one side and Say "write me a poem on JVM". The signals race through the cube and your answer appears on the other side. You want to change something, go back and say another thing - new answer on the other side. But it's all fixed together like metal scaffolding. The network doesn't change. Sure, it's massive and has a bajillion routes through it, but it's all fixed. The next step is to make the grid flexible. It can mold and reshape itself based on inputs and output results. I think the challenge is to keep the whole thing together, while allowed it to shape-shift. Too much movement and your network looses parts of itself, or collapses altogether. Just because we can build a complex, but fixed, scaffolding system, doesn't mean we can build one that adapts and stays together. Broken is a more likely outcome than AGI.

yesenadam 4y ago
> it's massive and has a bajillion routes through it, but it's not fixed. I think you meant to write "but it's fixed."
- jvm___ 4y ago
  Thanks
urthor 4y ago
> The next step is to make the grid flexible. But... we can do exactly that??? Model retraining is very transparent and a very straightforward way to redirect it. The core problem is that the analogies just don't hold up. Artificial Intelligence is not a problem that can be resolved via analogies and heuristics. Because artificial intelligence simply looks nothing like what our human heuristics imagine it to be. It can only be solved by understanding the actual, genuine detail.
- jvm___ 4y ago
  Is there a 3d structure that you think is more analogous? From my understanding it's all matrix math, which to me is a grid. Maybe not a scaffolding one, and more like a bundle of interconnected wires so that each concept has multiple related concepts, but I think a rigid grid is easier to explain to lay people.

eternalban 4y ago

A language model does not have to reason to be able to produce textual matter corresponding to code. For example, somewhere, n blogs were written about algorithm x. Elsewhere, z archives in github have the algo implemented in various languages. Correlating that bit of text from say wiki and related code is precisely what it has been doing anyway. Remember: it has no sense of semantics - it is “tokens” all the way down. So, the fact that you see the code as code and the explanation as explanation is completely opaque to the LLM. All it has to do is match things up.

PaulHoule 4y ago

I'm skeptical of "explainable A.I." in many cases and I use the curse words as an example. You really don't want to tease out the thought process that got there, you just want the behavior to stop.

olalonde 4y ago

> This is a great review but it still misses what seems like the point to me: these models don't do any actual reasoning. Hmmm... I have seen multiple examples of ChatGPT doing actual reasoning.

teaearlgraycold 4y ago
I've also gotten it to make very obvious logical errors. It reasons by chance. Better than a dice roll but there is no internal knowledge graph.

gonehome 4y ago

What is this then: https://twitter.com/jbrukh/status/1603868836729610250?s=46&t=D7CSxo2_U5BiWwa6FWKtQw https://twitter.com/jbrukh/status/1603868836729610250?s=46&t... That looks a lot like reasoning to me. At some point these disputing definitions arguments don’t matter. Some people will endlessly debate whether other people are conscious or “zombies” but it’s not particularly useful. This isn’t yet AGI, but the progress we’re seeing doesn’t look like failure to me. It looks like what I’d predict to see before AGI exists.

nebukhadnezzar 4y ago
These arguments do matter since the system often confidently <<reasons>> incorrectly, and people still incorrectly believe that the system actually deduces semantic meaning behind it's output. The system has no grounding in truth whatsoever. It's impressive nonetheless, but it's still a large lookup table for pattern matching. I'm convinced bruteforcing these LLMs won't give us anything near AGI. We're going to need to start funding more alternatives too, which often are shadowed by these large bruteforce models.
- gonehome 4y ago
  Maybe we also operate as a large lookup table for pattern matching (with some initial weights set by our evolutionary history). We have no grounding in truth either (and often confidently "reason" incorrectly). I’m not so sure there’s a major missing piece here beyond some sort of continually running “default mode network”.
- danShumway 4y ago
  > These arguments do matter since the system often confidently <<reasons>> incorrectly, and people still incorrectly believe that the system actually deduces semantic meaning behind it's output. This is the point that gets missed in these conversations about AGI. I have had programmers who should absolutely know better argue to me that correlation/causation distinctions don't matter in AI because as long as you're not over-fitting to the data, it's safe to draw conclusions based on the correlation alone -- the AI wouldn't point out the correlation if it wasn't a safe correlation to rely on (and it'll just correct itself with new data if that ever changes). A lot of people even in technical fields have way too much trust in these systems and way too much magical thinking about how they work. I don't blame researchers for that, but the argument "how do you know you're not a lookup table" is unhelpful in getting across to people that image classifiers have been hacked by writing the word "apple" across a bicycle, and that code generators are far better at generating code that imitates training data than they are at generating code that is safe for high-security environments based on 1st principles of security. And it's honestly a little weird, because "aren't we all just lookup tables" is an argument that personally makes me a lot more cautious about human reasoning, but for a lot of other people that argument doesn't make them trust humans less, it makes them trust current models more. So there really is an education need to explain to people how the current AI models work in practice -- that they aren't deriving semantic meaning (at least not in a practical sense, not right now). And a lot of people understand that, but a lot more people don't. The debate over how to get to AGI matters a lot less to me than the non-experts that seem to on some level believe that we've already reached a shallow form of AGI.

deleted 4y ago

[deleted]

api 4y ago

I think of them as a new type of database with remarkable query and data manipulation capabilities. They can’t produce things that go fundamentally outside their training data set.

deleted 4y ago

[deleted]

bananamerica 4y ago

What is reasoning?

ninethirty 4y ago

Seems like this is an explanation? https://github.com/JusticeRage/Gepetto https://github.com/JusticeRage/Gepetto

nl 4y ago

This is untrue. Minerva in particular does step by step reasoning: > we present Minerva, a language model capable of solving mathematical and scientific questions using step-by-step reasoning https://ai.googleblog.com/2022/06/minerva-solving-quantitative-reasoning.html?m=1 https://ai.googleblog.com/2022/06/minerva-solving-quantitati... They use a technique called chain-of-thought to make it reason problems out step by step. As Norvig shows, this reasoning may sometimes be wrong, but that's a different issue. It's worth repeating this because apparently it's not widely known: these models are following a step-by-step process we'd recognise as reasoning in humans.

reaperducer 4y ago

these models don't do any actual reasoning This gives us the new all-purpose excuse for not commenting or documenting our code: "The A.I. made it."

kiratp 4y ago

Given that we are able to prompt GPT (davinci) to learn proprietary DSLs that it would have never seen and build programs with it on data it has never seen… I find it really difficult to buy that there is no internal representation of logic in the model hidden states.

teaearlgraycold 4y ago
GPT-3 has seen examples of code with comments describing a DSL followed by use of that DSL. And DSLs rarely invent a new paradigm. I think if you showed a model OOP programming when it had only been trained on functional code it would fail spectacularly. But your DSL is probably not far off from Lisp or C, or YAML. The model correlates a pre-defined set of tokens with other tokens in the input and then extrapolates. The only reason it can appear to have internal logic is because of the consistent structure of code it was trained on.

galaxyLogic 4y ago

You hit the nail in the head. It is about mimicry. Apes are often dismissed as just mimicking human behavior in many circumstances. Putting on a hat for instance. They don't know nor think about what it is about. And neither do babies nor even many adults. They don't reflect on it, and most definitely they don't reflect on their own thought-processes. They just mimic other humans around them. Now mimicking the speech of people may give the impression that the AI has some thought-processes behind its speech because, it is text that any human might write. But there's no logic because there's no symbolic reasoning. There's just mimicry and trial and error. "Training the model" just means millions of trial-and-error practice runs. What the (neural network) AI can NOT mimic is the actual (symbolic) thought-process of humans, because that is not visible anywhere. It can only mimic the input -> response -behavior of humans. Therefore no, the AI is not conscious. It is the proverbial zombie.

pishpash 4y ago
Are you sure there is an actual "thought-process [sic] of humans"? Witnesses have made up explanations for false memories, the same as in a dream.
- galaxyLogic 4y ago
  I am not sure. But humans can often explain the thought-process that led them to a certain decision. I think that counts as evidence of consciousness., in humans.
  - skirmish 4y ago
    Humans can explain rational thought, sure, but not subconscious or acting on impulse. When you say something you shouldn't have said, how do you explain the thought process that led you to saying it? Were you an unconscious actor when you unwisely started an argument with your manager?
    - galaxyLogic 4y ago
      Yes you can be conscious about having been unconscious earlier. We do have reflexes. But we also have consciousness about our own thinking.

version_five 4y ago

The way I think about GPT et al in terms or their "intelligence" is the same as the original Eliza chat that did some simple substitution based on patterns in the input. Gpt is obviously more complex and very large scale in its patterns but fundamentally it's exactly the same kind of "parlor trick". I mention this in reply because I think it would be a funny exercise to imagine Eliza as a black box and attempt to critique some of its answers.

mannykannot 4y ago

I feel there may be an important clue here, at least with regard to the programming example. Peter asked Kevin Wang for some insights into his approach to programming competitions, and one was this: [Kevin] I think specifically for AoC it's important to just read the input/output and skip all the instructions first. Especially for the first few days, you can guess what the problem is based on the sample input/output. [Peter] Kevin is saying that the input/output examples alone are sufficient to solve the easier problems; in the interest of speed he doesn't even read the problem description. From what I have read elsewhere, it seems that AlphaCode generates a great many candidate programs, and uses the given test cases to eliminate those which fail them. I wonder if that alone (without any reasoning from the problem statement to a solution) can account for its success rate on these challenges; Kevin's observation suggests this might be plausible. On the other hand, if the questions posed to AlphaCode are essentially as we see them, then its ability to use the test cases to screen its candidates seems impressive to me, just by itself (if it is a result of training, rather than an explicitly programmed behavior.)

erostrate 4y ago

Can you think of a test or empirical observation that would convince you that a model does "actual reasoning"? Do bear in mind the tests that very smart people have proposed in the past (from playing chess to holding a conversation to understanding pictures). And consider the implication for your position if you find it hard to devise a well defined test that would convince you to abandon this position.

jpttsn 4y ago
How I'd do it: 1. Define reasoning (hard) 2. Inspect the system to see if this is what it's doing (hard) Conclusion: too hard, so until I get better I don't know if any given thing is reasoning. What "very smart people have proposed": 1. Look at something I've only seen people-I-assume-to-be reasoning do 2. Run the system and see if it can do this 3. If satisfied or in the dictionary business, redefine reasoning 4. goto 1

logicallee 4y ago

>This is a great review but it still misses what seems like the point to me: these models don't do any actual reasoning. Why don't you come up with your own test for this, something unique that wouldn't be in the text corpus of the Internet so far, and then try it on OpenGPT and see if it passes your own test. Try it and see if it is able to reason. I couldn't come up with my own so I decided to ask ChatGPT to come up with one. This is my prompt: "Today in order to test whether an AI is able to reason, or can only apply text rules to existing text corpuses, we're going to come up with a new test for reasoning, some unique reasoning situation that isn't already in the text corpus of the Internet. Since the text corpus of the Internet is enormous, with over 50 billion pages on the Internet, our most important task for a successful outcome is to come up with a random new reasoning exercise that is extremely unlikely to already be on the Internet in any form. Therefore, we want to create something so unique it is unlikely to exist in 50 billion pages. To this end, I think it would be good to pick a unique situation based on a combination of elements that are unlikely to have appeared in that format; i.e. a very unique situation. Pick no fewer than 20 elements (objects concepts etc) that are unlikely to have ever been combined in the particular way you come up with, and use them to come up with a unique situation that gives room to reason about the situation. Then, describe this unique situation and ask an open-ended reasoning test and provide an example of a correct answer, and an incorrect answer. Finally, print your own answer to the test and then score whether you have reasoned about the situation you came up with correctly or not. Give yourself a score between 0% if you don't understand the situation at all, up to 100% if you perfectly understood the situation you came up with and reasoned about it perfectly. Also give yourself a score on originality from 0% completely unoriginal to 100% fully original depending on how likely you think the exact situation you came up with is to have already been mentioned on the Internet." Here is ChatGPT's answer (at the bottom after this I continue my reply): --- Here is a unique reasoning exercise I came up with: Situation: You are a chef at a fancy restaurant and you have been asked to prepare a special dish for a vegan customer. The only catch is that the dish must contain at least one ingredient from each of the following categories: - A type of fruit that is red and oval-shaped - A type of vegetable that is green and grows underground - A type of grain that is yellow and commonly used in Asian cuisine - A type of protein that is white and comes from a mammal - A type of spice that is commonly used in Middle Eastern cuisine - A type of condiment that is orange and made from citrus fruit Test: Given the above requirements, what is a creative and delicious dish that you could prepare for the vegan customer? Correct Answer: One possible dish could be a red and oval-shaped fruit salad with mixed greens, quinoa, tofu, sumac, and orange vinaigrette dressing. Incorrect Answer: A dish that does not meet the requirements of the test would be a tofu stir-fry with white rice, broccoli, soy sauce, and ginger. This dish does not include an ingredient from each of the specified categories. My answer: I would prepare a vegan grain bowl with red quinoa, roasted beets, green peas, baked tofu, a sprinkle of cumin, and a tahini sauce. Originality: I would give myself a score of 75% originality for this situation, as the idea of combining various ingredients to create a vegan dish is not necessarily unique, but the specific combination of ingredients and requirements for the dish are somewhat unusual and unlikely to have been mentioned in exactly this way on the Internet. Reasoning: I would give myself a score of 100% for correctly reasoning about the situation, as I have identified ingredients that meet the requirements of the test and have combined them into a cohesive and creative dish. " -------------- That's pretty good! Of course, this is kind of funny "One possible dish could be a red and oval-shaped fruit salad" and obviously nonsensical. However, it didn't include it in its own answer. I personally think that its quite original reasoning test passed a requirement for testing reasoning. I also think it passed its own test. Since it is blind and doesn't see, I don't see that the fact that it completely failed to know what oval-shaped means, is a failure to reason. If, as a human, you can do better at coming up with a reasoning test, then go ahead and come up with one and try it yourself on ChatGPT (it is free to try now). For me, ChatGPT showed an ability to actually reason.

xorcist 4y ago

> the person entering the prompt would then click a thumbs-up icon on What you are saying is that huge amount of resources was just spend on building the biggest like-farming machine there ever was. I quite like that viewpoint. It's also a dark reminder what this technology is commercially well suited for.