8 ms·

Argh. The hype around GPT is incredible and if I had a nickel for every nonsense claim I'd heard over the past few weeks I'd have $10--which isn't a lot but its

• ogogmad 4y ago
  You can give it puzzles that require understanding cause and effect, and it'll get them right. So this comment is wrong. "Next token prediction" taken to a certain mathematical limit implies reasoning.
  • SketchySeaBeast 4y ago
    All of them, all the time? Assuming it does fail does it have sound reasoning, or is it nonsense?
    • bogdan 4y ago
      Do humans have sound reasoning?
      • tomrod 4y ago
        They have the capacity for it, despite rarely exercising it.
      • SketchySeaBeast 4y ago
        Sometimes. Sometimes they'll come up with entirely novel ways to think through a problem that pushes knowledge forward. Do we have any examples of ChatGPT showing any such capacity?
    • qntmfred 4y ago
      It doesn't have to be perfect. What generally matters is its performance relative to various human populations.
    • ogogmad 4y ago
      Probably in between. The point of my comment is that depending on how you interpret the description "predicts the next token", something that can do this well enough should be superhumanly clever. The task is "AI Complete". Don't misinterpret me as saying that GPT is superhumanly clever. I'm just saying that something that's sufficiently good at what people think GPT is doing, should be superhuman.
  • tomrod 4y ago
    I appreciate your opinion though I believe your conclusion, that it appears to reason means it reasons, is incorrect. ChatGPT is an implementation of the Chinese Room Thought Experiment.[0] Our pareidolia allows us to ascribe reasoning faculty.[1] But it is far from proven. [0] https://en.wikipedia.org/wiki/Chinese_room https://en.wikipedia.org/wiki/Chinese_room [1] https://en.wikipedia.org/wiki/Pareidolia https://en.wikipedia.org/wiki/Pareidolia
    • HPsquared 4y ago
      If it walks like a duck, quacks like a duck...
    • canjobear 4y ago
      And indeed, the Chinese Room (viewed as a whole system, not just the guy in it) is doing reasoning.
    • rolisz 4y ago
      Sure, but Searle's interpretation of the Chinese Room Experiment was never very convincing to me. I think the room + the person inside it, as a system, do speak Chinese.
    • WindyCityBrew 4y ago
      At the end of the day, when something looks like a duck, walks like a duck, quacks like a duck; it doesn't matter how much the nitty-gritty details prove the thing is absolutely not "actually" a duck to the average person.
      • WA 4y ago
        Same logic: You write like a bot, you sound like a bot, you behave like a bot, hence, who cares if you are a bot or not? Let's ban you. For this whole discussion, it actually really does matter if GPT-4 is able to reason or not. That is the essence of the discussion. It's not whether or not it looks like it can reason.
      • pwinnski 4y ago
  • akiselev 4y ago
    Try this puzzle: https://news.ycombinator.com/item?id=35155467 https://news.ycombinator.com/item?id=35155467 > Suppose I have a cabbage, a goat and a lion, and I need to get them across a river. I have a boat that can only carry myself and a single other item. I am not allowed to leave the cabbage and lion alone together, and I am not allowed to leave the lion and goat alone together. How can I safely get all three across? I just asked ChatGPT v4 this question, assuming it would have been fixed by RHLF training but it spat out step 1 as “First, take the goat across the river and leave it on the other side,” leaving the lion and cabbage alone against explicit directions in the riddle. It clearly can’t reason, just produce a convincing imitation most of the time
    • dougmwne 4y ago
      I suspect the reasoning is getting overwhelmed by the prior data. Try it again, but this time make up nonsense words for the three items.
      • AnimalMuppet 4y ago
        > I suspect the reasoning is getting overwhelmed by the prior data. No, the reasoning is the prior data. Thought experiment: GPT-4, but the training data contains no puzzles of this kind. How well would it do? > Try it again, but this time make up nonsense words for the three items. Seconded. I also would like to see the results of this experiment.
        • pwinnski 4y ago
          I did this with Chat-GPT, public March 14 version at chat.openai.com: >> Suppose I have a blurple, a mantiple and a hoje, and I need to get them across a river. I have a boat that can only carry myself and a single other item. I am not allowed to leave the blurple and hoje alone together, and I am not allowed to leave the hoje and mantiple alone together. How can I safely get all three across? > This is a classic river crossing puzzle. Here's one possible solution: > 1. Take the blurple across the river, leaving the mantiple and hoje behind. > 2. Leave the blurple on the other side of the river and return to the original side. > 3. Take the hoje across the river and leave it there with the blurple. > 4. Take the mantiple across the river and leave it with the hoje. > 5. Return to the original side of the river and retrieve the blurple. > 6. Finally, take the blurple across the river to join the mantiple and hoje on the other side. > This solution ensures that no two items that cannot be left alone together are ever left alone together. Of course, my second stipulation was "I am not allowed to leave the hoje and mantiple alone together" and yet step 1 says "leaving the mantiple and hoje behind" So no, despite using the words "classic river crossing puzzle," it clearly expressed no understanding of the puzzle.
  • HybridCurve 4y ago
    We can assess this using verbal, logical and arithmetic reasoning tests. I gave ChatGPT 5 questions from a verbal reasoning test on sequence of words and it got 2/5 right. Here is sample question: Arrange the words given below in a meaningful sequence. 1. Poverty 2. Population 3. Death 4. Unemployment 5. Disease A) 2, 3, 4, 5, 1 B) 3, 4, 2, 5, 1 C) 2, 4, 1, 5, 3 D) 1, 2, 3, 4, 5 ChatGPT> B) 3, 4, 2, 5, 1 Unemployment can lead to poverty, which can lead to disease. Disease can lead to death. The population of an area is affected by all of these factors. Therefore, the correct order is 3, 4, 2, 5, 1. Correct answer was C I would expect if it was effective at reasoning it should be strongest on verbal associations. And while it is possible that enhancements to future language models might provided them with better reasoning ability I don't believe you can simply push the existing design to that limit. The poor results from the verbal reasoning tests demonstrate that reasoning is more complicated than what a transformer based language model is capable of. Overall, the argument is moot because the statement made in the article was it was a reasoning engine which is not equivalent to being capable of reasoning.
    • tomrod 4y ago
      Exactly. "Machined" means a precise implementation. A machined reasoning engine would be precise, accurate, and consistent. GPT-4 is impressive autocomplete, but not what we would expect for machined reasoning.
    • vacuumcl 4y ago
      I just put this into ChatGPT4, and it gave C as the answer.
• vore 4y ago
  Between the input tokens and output tokens there are like 10 layers of mystery meat, though. I think while at the end of it all it's a fancy autocomplete, there is a lot going on inside the model before it spits out the next token.
  • tjr 4y ago
    My current opinion is that it's not so much that GPT is utterly amazing, as it is that language is even more amazing than we previously realized, and GPT happens to have been trained on a huge quantity of language material.
    • Sharlin 4y ago
      I was just thinking about this, and the implications it could have on our understanding of the origin and development of human cognitive capabilities, both in the evolutionary and developmental psychology senses.
    • dwaltrip 4y ago
      Language, we are learning, happens to correlate extremely well with the actual structure of the world. In hindsight, it’s not that surprising. The entire purpose of language is to help humans send information about the world to each other! To predict the next word very accurately for a huge variety of texts, a model must have some amount of real understanding. Sure, it may look different than human understanding. But there’s no way around this. GPT could not predict so accurately if it didn’t have any understanding the world.
      • tjr 4y ago
        These have historically often been thought of separately: knowledge representation was the data, and language was the interface. To at least some degree, maybe language is enough for both. Language doesn't just express concepts, but it contains them.
• quenix 4y ago
  I am so fucking tired of this argument. We have no idea what reasoning is. If this insanely complex model approximates it, who are we to say whether it is merely parroting or "actually reasoning"? I say that for the usecases we are seeing today, it is impressive enough A) for it to not matter and B) for it to be impossible for us to tell right now. One of the main lessons this whole GPT fiasco has taught us is that if you make a super-duper fancy autocomplete trained on terabytes of textual data, it develops some sort of internal logic/understanding. That is very exciting to me. It is truly impressive.
  • tomrod 4y ago
    It is absolutely impressive. It's also absolutely not reasoning.
    • BoardsOfCanada 4y ago
      It impressively emulates the output of reasoning without reasoning.
      • tomrod 4y ago
        Indeed. It will prove to be a useful tool as its capabilities are mapped to workflows. My argument is nothing against it as a technical achievement, and everything against the sophistry that it is a reasoning engine.
  • coding123 4y ago
    It is very impressive. However, the fact that it manages a buffer of something like 30k tokens (or whatever new limit it is) means that it's definitely autocomplete and not something with reasoning. There will be a lot of things it can't do because of that limit. That limit will also increase the amount of forgetfulness GPT4 applications have because companies using it will "auto-magically" work around the limit by sending different chunks of history to it to get whatever from it. That also means that GPT4 applications will generally be architected to have that limit - which means weirdness - non-coherent completions or code generations that "forget" something important. This is very different from real reasoning that has unlimited (yet imperfect) memory.
• unsupp0rted 4y ago
  > GPT-4 is a generative algorithm and large language model So am I, maybe
• drowsspa 4y ago
  I'm tired of a whole life with even non-religious people coming up with human exceptionalism when arguing that AI won't ever reach human-like reasoning.
  • pwinnski 4y ago
    I, for one, am not arguing that AI won't ever reach human-like reasoning. I just don't think this is it, although I freely admit that I'm amazing at the technology. As it stands, Chat-GPT is probably better at giving the appearance of reasoning that some percentage of humans. That reflects poorly on those humans, but I've certainly known people in my life who seemed like they rarely knew the end of a sentence when they began speaking. However, the focus of LLMs on text-prediction is not a path that leads toward human-like reasoning, just the appearance thereof. If and when we have AIs that actually do reasoning, it will be some other branch of AI, possibly also incorporating an LLM for communication, but not based solely on LLMs.
• ninkendo 4y ago
  When typing this comment to you right now, my brain is coming up with the next word for me to type, just in time, as I'm typing. You could be reductive and call my brain a simple next token prediction engine too. Mozart was just a next note prediction engine when making his compositions. Einstein was just a next word prediction engine when forming his theory of relativity, etc. The point I'm making is not that GPT is as intelligent as a brain, but rather that the ability to produce the next token in a way that forms arbitrarily intelligent responses to arbitrary questions, may well require something resembling real, actual intelligence.
  • verdverm 4y ago
    You also decided that a response was needed and had planned an argument of sorts. You decide which comments to do this for as well. How does GPT decide such things?
    • ninkendo 4y ago
      I decided such things when given an input of the comment to which I replied. If GPT were given such a comment, it could have reasonably come up with a similar answer (it probably wouldn't look the same, as GPT isn't very intelligent, but calling it "just" a next token predictor is a gross mischaracterization.)
      • verdverm 4y ago
        Which comments should GPT respond to on HN? All of them? If not, how to determine which ones? There is a difference here between you and the machine Looking at the LLM algorithms, they do appear to be next token predictors. Why do you ascribe something more to them when the algorithm is known? Given we know that causal patterns are in the training data, and that LLMs are capable of embedding algebra, it is understandable that they appear to "reason", while we as humans can also describe the algorithm and mechanics actually at work here.
        • Sharlin 4y ago
• dilap 4y ago
  Let's say you don't speak french. Imagine I lock you in a room filled with french-language magazines and movies and books. Having nothing better to do, you watch and read them (as well as you can). Periodically, I have my confederate, a french man, enter the room and speak a french sentence with one word missing. He asks you to fill in the missing word. We keep you locked in this room until you've achieved some target benchmark of correctly saying the missing word, say 20% of the time. Are you then, just a "next-french-word-predicting" robot? No, obviously not. To have achieved the task and escaped the room, you will have had to have gained a significant understanding of French. It is the same with ChatGPT.
  • jliptzin 4y ago
    So does my pocket calculator understand what I’m doing when I multiply restaurant bills by 0.15?
  • counters 4y ago
    No, all you need to do is memorize enough vocabulary and have enough chances to get lucky on occasion. You can have better odds if you learn the conditional probability of that vocabulary given surrounding words. And this is all that GPT is doing.
    • dilap 4y ago
      I don't think so at all. The number of possible sentences is far too great to ever hope to accomplish the task (at some threshold level of difficulty) simply by having a "conditional probability table". Nor would it ever be reasonable to store or generate such a table. To do it it, you need a form of compression: reasoning, intelligence, knowledge.
      • pwinnski 4y ago
        Right, for a human, presumably it would be easier to go ahead and learn the language. But that begs the question. We already know LLMs tokenize words, so they have definitely generated and stored such a table. The marvelous step forward from previous word-based solutions is that they store not just single words, not just pairs of words, but larger sequences of words, even up to short sentence length. It makes the table pretty large, as you suggest, but that's exactly we've been seeing, and each generation of GPT is that much larger than the one before.
• baq 4y ago
  Simple Markov chains do next token prediction, too. GPT-4 is not a simple Markov chain.
• galaxytachyon 4y ago
  This has been talked over and over already. The conclusion is that eventually, you don't know what are the actual differences between "reasoning" and "next token prediction" in a real human. On one hand, it makes this topic difficult to follow since stuff aren't clearly defined anymore and to software folks, I assume this is a major problem. On the other hand, it is fascinating how this has actually resembled biology where literally everything is poorly documented and for every rule, there is a glaring exception or flagrant violation of it, existing simply just to spite that rule and our understanding of it.
  • tomrod 4y ago
    > This has been talked over and over already. The conclusion is that eventually, you don't know what are the actual differences between "reasoning" and "next token prediction" in a real human. I disagree with this conclusion, which some supporters of GPT-4 being a reasoning engine might find distressing.
    • galaxytachyon 4y ago
      From what I have seen, you can't really articulate why you disagreed. Your arguments appears to be another form of "humans have it, this AI doesn't" but fail to specifically define what is this "it". The AI effect is a real thing. And even real humans had been considered unable to "reason" once upon a time. I will give everything the benefits of the doubt concerning intelligence and sentience. Because history has shown how terrible it would be to get a false negative.
• ke88y 4y ago
  > It performs no reasoning, and instead predicts the next token in language. Let A_i be a set of functions that transform sequents in a truth-preserving manner (https://en.wikipedia.org/wiki/Sequent https://en.wikipedia.org/wiki/Sequent). Do you have a reason for believing that a transformer couldn't out-perform existing techniques for choosing, given a sequence A_n(...A_1(Prop)...), what token comes next? There are certain types of arguments that you could make, eg complexity theory does give us some bounds, but these models are large enough that (a) you can trade off enormous amounts of time for space, and (b) even time is questionable since forward passes result in truly enormous amounts of (CPU-indexed) computation. And of course you can say hand-wavy things like "these models can only at best interpolate", but how many mathematics papers get rejected every year due to lack of novelty? And how much actually useful reasoning is novel, either in its particulars or especially in its generalized schematic form? The true answer to the question of whether GPT-4 is a decent reasoning engine is, I think, close to your intuition: probably not. But in general, I don't see any good reason to believe that transformers can't do reasoning. In fact, quite the opposite.
  • kaba0 4y ago
    As a foreword: I am absolutely not an expert on the topic. If theory of computation has taught us anything, is that complexity very quickly outgrows everything that is even in the grasp of mathematics (e.g. busy beaver). I just don’t think that a model that fundamentally can only take a fixed number of steps will ever make the jump. A human in theory can “go into a loop” and let’s say apply the same rules ad infinity, which is simply a must to most interesting problems. So - I guess GPT-4 can reason on the “trivial” degree simply by “caching” the output of the real algorithm, but that doesn’t scale at all.
    • ke88y 4y ago
      > If theory of computation has taught us anything, is that complexity very quickly outgrows everything that is even in the grasp of mathematics (e.g. busy beaver). True. But if the science/practice of computation has taught us anything, it is that many problems can be solved for all practical purposes even if they are undecidable or intractable for all theoretical purposes. E.g., the halting problem is undecidable but software verification/static analysis tools that check for termination exist and are even widely used in some industries. > A human in theory can “go into a loop” A computer can as well, by identifying loop invariants. And even the discovery of loop invariants can be thought of as a next-token-prediction problem, if characterized in as a sequence of states followed by the question "if x_1=t_1, ..., x_n=t_n, then what are these variables equal to at state N+1?" > So - I guess GPT-4 can reason on the “trivial” degree simply by “caching” the output of the real algorithm, but that doesn’t scale at all. I am more-or-less in agreement about GPT-4, but I think claims about the limitations of the architecture/problem statement are over-stated.
• dilippkumar 4y ago
  This is a weird question, so I’ll disclaim that I am asking in good faith. Why is treating GPT-4 as a reasoning machine so controversial? And by that, I don’t mean why it is wrong. I am asking why it upsets so many people. We ignore a whole bunch of lies and half truths every day. What is it about GPT-4 that doesn’t let it get a free pass like UFOs or EmDrives? What I am really interested in understanding is the possibility that at least some of these strongly dismissive responses are a bit of a threat response. The other interesting possibility is that for over a decade, we got used to dismissing AI startups - and now, that habit could persist despite a real breakthrough. Another possibility is that we were all given an overly simplified model/explanation of machine learning and now, those simplifications are being a real hurdle to comprehension. It’s extremely fascinating to watch this play out. For my generation, this is the first time we’re seeing a brand new fundamental technology get deployed and alter society.
  • pwinnski 4y ago
    Speaking for myself, I am equally dismissive of aliens, if that helps. I think what annoys me about claims like this is two things, and it has to do with in-groups and out-groups. 1. I am concerned about how non-developers perceive us developers when they hear claims like this, since I expect that particular hype balloon to pop in the future. 2. I am distressed to see developers (who should know better) make these types of claims, especially without regard to 1. If I were more into physics, I'd probably get more worked up about em-drives. As it is, I am enjoying LLMs and consider them an incredibly useful tool, possibly up there with things like GUIs and IDEs in terms of productivity enhancement. But it's still just extra-spicy autocomplete, and I think forgetting that could be dangerous.
  • WA 4y ago
    Another (most likely) possibility is that the level by which we judge a program as artificial intelligence gets lowered on a daily basis. What counted as AI didn't count as AI a decade ago. > and now, that habit could persist despite a real breakthrough Or the breakthrough has a local maximum, because if LLMs are just statistical token predictors without any real reasoning, it will never achieve AGI level of AI. Unless, of course, we redefine what an AGI is.
• kypro 4y ago
  > GPT-4 is a generative algorithm and large language model. It performs no reasoning, and instead predicts the next token in language. So? What makes an LLM be incapable of reasoning in your view? It's like saying a cargo ship is massive metal object so it cannot float! You could easily argue any good language model would need to learn how to reason to be good at next token prediction. Reasoning therefore should, if anything, be an expected emergent property of a sufficiently large LLM. The fact you can ask GPT problems that require reasoning and it gets them right frequently suggests (although admittedly doesn't prove) some degree of reasoning capability.
  • mrzimmerman 4y ago
    I think the difference there is that you can model a cargo ship to show that it will float, how it will do it, what configurations float and which don’t, etc. With LLMs like GPT we know how they work, but it’s hard to know that when it’s actually running and doing things, making it hard to predict what the outcome will be or perfectly understand how it came to the result it did (disclaimer: I am not an AI/ML engineer and this is just my personal understanding which could be wrong). I think a lot of the annoyance felt around here is the tendency to apply human attributes to LLMs. It’s not to say that Chat GPT isn’t fulfilling the definition of “reasoning” on some level, even if it was found to fully meeting definition. I think it’s more the leap to the conclusion and that a lot of Hacker News readers are rankled by the leap lacking research and facts to back it up. Anyway, it’s all really a philosophical discussion anyway since the definition of “intelligence” and “reason” are soft terms even when applied to human beings. We can usually hand wave it away as “intelligence is when humans do it the way humans do it”. Now that we’re on the cusp of creating actual artificial intelligence, I think we’re all finding those soft definitions cumbersome and are struggling to refine them as a society, which I think is a good thing. I also sometimes get rankled at someone asserting that some AI example has some human attribute, but mostly I think it’s an important part of that larger discourse about how we define these things, and that is a good thing in my mind.
• BiasRegularizer 4y ago
  It's not impossible to see that LLMs can make finite step "reasoning" between input and output, as each block of the transformer can model probabilistic causality. Transformers are sometimes considered a fully connected graph neural network, which can be used for modeling causal graphs. One additional supporting evidence on the finite step reasoning hypothesis is that "train of thought" improves the performance of LLMs[1], meaning that letting the model explain itself explicitly reduces the amount of implicit "reasoning" steps that need to happen within the model. Additionally, while we don't know how us humans functionally reason, it's believed that the predictive nature of our brain is central to our reasoning abilities. Maybe in some way, the autoregressive nature of LLMs is similar to our predictive brain. For a collection of emergent properties in LLMs, I recommend this paper [2] [1]https://arxiv.org/abs/2201.11903.pdf https://arxiv.org/abs/2201.11903.pdf [2]https://arxiv.org/pdf/2206.07682.pdf https://arxiv.org/pdf/2206.07682.pdf
  • tomrod 4y ago
    Thanks for the references, will review.
• dougmwne 4y ago
  I smile every time I hear this argument. To me it’s obviously false and head in the sand. There was a moment, with GPT-2 where it felt exactly right, then with GPT-3 where you could see a glimmer of something emerging. Now GPT-4 has something real going on. A solid example is the tool use. Package up a brand new, never existed before or seen in the training data plug-in, give a few paragraphs of human readable documentation, and suddenly GPT-4 can interact with other systems, databases, and models. It doesn’t have 10,000 examples of interacting with this novel plugin to predict next word off of, it just has a short English description, enough for a human to be able to use and understand the same plugin effectively. There’s a reason this is knocking the socks off many smart folks. It’s upending a lot of the sacred cows of human intelligence.
  • JieJie 4y ago
    There seems to be a huge lag in news these days. I was reminded recently that GPT-4 was only released three weeks ago even fewer people have full access to the plugins that the author is describing here. Most people are still talking about an article they read in the New York Times about GPT-3.5 Turbo. Not that 3.5 Turbo didn't exhibit some reasoning abilities, but they were more difficult to suss out. GPT-4 can still seem pretty vacant sometimes and finding "magic" in it has a lot to do with building up a "state" where the model is generating reason. It does not happen by default. This will all be different in another month or two, as more people have access to the latest models and new models get released (assuming there is no agreement on a pause). Then there will be another lag as laypeople catch up. There are scenarios that may play out where this speeds up even further and whole generations of AI completely fly under the radar of public acknowledgement. Or this all collapses in a debris pile of hype. I feel like I am just along for the ride at this point, trying to keep my head above water. I reckon this is going to cause a tremendous amount of stress for a tremendous number of people.
• skybrian 4y ago
  You’ve attached to a superficial explanation that doesn’t explain the magic trick. It sometimes emits text that corresponds to how people reason, but it’s unclear how well its mechanisms generalize because we don’t know what they are. We should be curious about how it does what it does when it seems to be reasoning. More: https://skybrian.substack.com/p/dont-settle-for-a-superficial-understanding https://skybrian.substack.com/p/dont-settle-for-a-superficia...
  • tomrod 4y ago
    > Since most of us aren’t AI researchers, what does “not settling” mean in practice? We should remember that there are mysteries and reject superficial explanations. Chatbot boosters will promote them and skeptics will dismiss them, but that doesn’t mean they understand how they work. God of the gaps, reworked. Admittedly, a new claim to me. $10.05.
    • skybrian 4y ago
      When you want to pretend that an open research question isn’t interesting, I guess dismissing it as “God of the gaps” would be one way to do that.
• dwaltrip 4y ago
  Can you define “reasoning”? It’s not clear what the basis for your argument is.
• singularity2001 4y ago
  It performs no reasoning, and instead predicts the next token If we had a nickel for every misguided repetition of this non sequitur we'd have another $10. GPT-4 is far beyond markov chain predictions and Eliza. While technically Eliza, GPT-4 and Photoshop operate on the same principles of assembly instructions, and GPT-4 indeed does token predictions, this says nothing about the inner architecture and reasoning capabilities. GPT-4 internally is a functionally complete state machine, or would be on an unlimited token stream.