8 ms·

Chiang makes some insightful points, e.g. about what we mean by magic. Then I come to > [LLMs] can get better at reproducing patterns found online, but they d

• mewpmewp2 2y ago
  Even most intelligent people can hallucinate, we still haven't fixed this problem. There's a lot of training material and bias which leads many to repeat those things "LLM's are just a stochastic parrot, glorified auto complete/google search, Markov chains, just statistics", etc. The thing is, these sentences sound really good and so it's easy to repeat them when you have made up your mind. It's a shortcut.
  • compiler_queen 2y ago
    > Even most intelligent people can hallucinate, we still haven't fixed this problem. No we have not, neurodiverse people like me need accommodations not fixing.
  • cyrillite 2y ago
    They’re right until they’re wrong. AI is (was?) a stochastic parrot. At some point AI will likely be more than that. The tipping point may not be obvious.
  • dutchbookmaker 2y ago
    I feel like at this point we have to separate LLMs and reasoning models too. I can see the argument against chatGPT4 reasoning. The reasoning models though I think get into some confusing language but I don't know what else you would call it. If you say a car is not "running" the way a human runs, you are not incorrect even though a car can "outrun" any human obviously in terms of moving speed on the ground. To say since a car can't run , it can't move though is obviously completely absurd.
    • bonoboTP 2y ago
      This was precisely what motivated Turing to come up with the test named after him, to avoid such semantic debates. Yet here we are still in the same loop. "The terminator isn't really hunting you down, it's just imitating doing so..."
  • kortilla 2y ago
    LLMs don’t go into a different mode when they are hallucinating. That’s just how they work. Using the word “hallucinate” is extremely misleading because it’s nothing like what people do when they hallucinate (thinking there are sensory inputs when there aren’t). It’s much closer to confabulation, which is extremely rare and is usually a result of brain damage. This is why a big chunk of people (including myself) think the current LLMs are fundamentally flawed. Something with a massive database to statistically confabulate correct stuff 95% of the time and not have a clue when it’s completely made up is not anything like intelligence. Compressing all of the content of the internet into an LLM is useful and impressive. But these things aren’t going to start doing any meaningful science or even engineering on their own.
  • watwut 2y ago
    It is not hallucination. When people do what we call halucination in chatGPT, it is called "bullshiting", "lying" or "being incompetent".
  • acureau 2y ago
    Intelligent people do not "hallucinate" in the same sense that an LLM does. Counterarguments you don't like aren't "shortcuts". There are certainly obnoxious anti-LLM people, but you can't use them to dismiss everyone else. An LLM does nothing more than predict the next token in a sequence. It is functionally auto-complete. It hallucinates because it has no concept of a fact. It has no "concept", period, it cannot reason. It is a statistical model. The "reasoning" you observe in models like o1 is a neat prompting trick that allows it to generate more context for itself. I use LLMs on a daily basis. I use them at work and at home, and I feel that they have greatly enhanced my life. At the end of the day they are just another tool. The term "AI" is entirely marketing preying on those who can't be bothered to learn how the technology works.
• ustad 2y ago
  Has an AI “proven” a theorem?
  • fragmede 2y ago
    Leading mathematician Terence Tao says yes, with a lot of guidance. https://mathstodon.xyz/@tao/113132503432772494 https://mathstodon.xyz/@tao/113132503432772494
    • abecedarius 2y ago
      I have a similar opinion of Claude Sonnet. Superhuman knowledge; ability to apply it to solve new math/coding problems at roughly the level of a motivated high-schooler (but not corresponding exactly in profile to anything human); less ability to stay on track the longer the effort takes. But ChatGPT a couple years ago was at more like grade-school level at problem-solving. What should I call this thing that the best LLMs can do better than the older ones, if it's not actual reasoning? Sparkling syllogistics? Sorry, that's sarcastic, but... it's from a real exasperation at what seems like a rearguard fight against an inconvenient conclusion. I don't like it either! I think the rate of progress at building machines we don't understand is dangerous. (Understanding the training is not understanding the machinery that comes out.) Compare the first previews of Copilot with current frontier "reasoning" models, and ask how this will develop in the next five years. Maybe it'll fizzle. If you're very confident it will: I'd like to be convinced too.
      • fragmede 2y ago
        you said you said it sarcastically but I like "syllogistic" a lot. We need more volcabulary to describe what LLMs do, and if I tell ChatGPT A implies B implies C, and I tell it A is true, and I can describe that as the LLM syllogisting and not use the words "reasoning" or "thinking", that works for me. As far as if it will fizzle, even if it does, what we have currently is already useful. Society will take time to adjust to ChatGPT-4's level of capabilities, nevermind whatever OpenAI et al releases next. It can't yet replace a software engineer, but it makes projects possible they previously weren't attempted because they required too much investment previously. So unless you're financially exposed to AI directly (which you might be, many people are!), the question of if it's going to fizzle is more academic than something that demands a rigorous answer. Proofs of a negative are really hard. Reusable rockets were "proven" to be impossible right up until it was empirically proven possible.
• mofeien 2y ago
  To me this also feels like a statement that would obviously need strong justification. For if animals are capable of reasoning, probably through being trained on many examples of the laws of nature doing their thing, then why couldn't a statistical model be?
  • fmbb 2y ago
    > For if animals are capable of reasoning Are they? Which animals? Some seem smart and maybe do it. Needs strong justification. > probably through being trained on many examples of the laws of nature doing their thing Is that how they can reason? Why do you think so? Sounds like something that needs strong justification. > then why couldn't a statistical model be? Maybe because that is not how anything in the world attained the ability to reason. A lot of animals can see. They did not have to train for this. They are born with eyes and a brain. Humans are born with the ability to recognize pattern in what we see. We can tell objects apart without training.
    • unification_fan 2y ago
      > Needs strong justification. if animals didn't show problem-solving skills, and thus reasoning, complex ones wouldn't exist anymore by now. Planning is a fundamental skill for survival in a resource-constrained environment and that's how intelligence evolved to begin with. Assuming that intelligence and by extension reasoning are discrete steps is so backwards to me. They are quite obviously continuously connected all the way back to the first nervous systems.
    • ElevenLathe 2y ago
      Are human beings not animals? If animals can't reason, then neither can we.
• sleepytree 2y ago
  If I write a book that contains Einstein's theory of relativity by virtue of me copying it, did I create the theory? Did my copying of it indicate anything about my understanding of it? Would you be justified to think the next book I write would have anything of original value? I think what he is trying to say is that LLMs current architecture seems to mainly work by understanding patterns in the existing body of knowledge. In some senses finding patterns could be considered creative and entail reasoning. And that might be the degree to which LLMs could be said to be capable of reasoning or creativity. But it is clear humans are capable of creativity and reasoning that are not reducible to mere pattern matching and this is the sense of reasoning that LLMs are not currently capable of.
  • vouwfietsman 2y ago
    > But it is clear humans are capable of ... Its not though, nobody really knows what most of the words in that sentence mean in the technical or algorithmical sense, and hence you can't really say whether llms do or don't possess these skills.
    • mrcsd 2y ago
      Words are not reducible to technical statements or algorithms. But, even if they were, then by your suggestion there's not much point in talking about anything at all.
      • vouwfietsman 2y ago
        They absolutely are in the context of a technical, scientific or mathematical subject. Like in the subject of LLMs everyone knows what a "token" or "context" means, even if they might mean different things in a different subject. Yet, nobody knows what "consciousness" means in almost any context, so it is impossible to make falsifiable statements about consciousness and LLMs. Making falsifiable statements is the only way to have an argument, otherwise its just feelings and hunches with window dressing.
    • southernplaces7 2y ago
• freejazz 2y ago
  >and wonder how an intelligent person can still think this, can be so absolute about it. I wonder how people write things like this and don't realize they sound as sanctimonious as exactly whatever they are criticizing. Or, if I was to put it in your words: "how could someone intelligent post like this?"
  • abecedarius 2y ago
    You're right, it was kind of rude. Apologies. I really would rather be wrong, for a reason I gave in another comment. The thing is, you can interact with this new kind of actor as much as you need to to judge this -- make up new problems, ask your own questions. "LLMs can't think" has needed ever-escalating standards for "real" thinking over the last few years. Gary Marcus made a real-money bet about this.
    • freejazz 2y ago
      I think a better question is "what is the value of thought?" when it came to conclusions such as yours: "I should be rude to this poster because they disagree with me"
• Der_Einzige 2y ago
  Ted Chiang revealed himself at his NeurIPS "Pluralism and Creativity" workshop to be... a great book author and not much else. His statements during his panels with the other AI researchers proved that he was not up to date on modern AI research. He's overly sentimental, and so are his books. I wish there were other sci-fi authors that the AI community wanted to contact but after "Arrival" I get it since "Arrival" is the literal wet-dream of many NLP/AI researchers.
• jagged-chisel 2y ago
  Sounds to me like a sci-fi author exploring his thoughts. Perhaps a full treatment of the subject wasn’t on his mind that day. I also don’t expect the article’s author to include anything they feel is mundane to themselves. Or to only include what they personally found interesting.
• eterps 2y ago
  > and wonder how an intelligent person can still think this, can be so absolute about it. What is "actual" reasoning here? Large language models excel at processing and generating text, but they fundamentally operate on existing knowledge. Their creativity appears limited to recombining known information in novel ways, rather than generating truly original insights. True reasoning capability would involve the ability to analyze complex situations and generate entirely new solutions, independent of existing patterns or combinations. This kind of deep reasoning ability seems to be beyond the scope of current language models, as it would require a fundamentally different approach—what we might call a reasoning model. Currently, it's unclear to me whether such models exist or if they could be effectively integrated with large language models.
  • FrustratedMonky 2y ago
    "Their creativity appears limited to recombining known information" There are some theories that this is true for humans also. There are no human created images that weren't observed first in nature in some way. For example, Devils/Demons/Angels were described in terms of human body parts, or 'goats' with horns. Once we got microscopes and started drawing insects then art got a lot weirder, but not before images were observed from reality. Then humans could re-combine them.
    • eterps 2y ago
      I understand your point, but it's not comparable: Humans can suddenly "jump" cognitive levels to see higher-order patterns. Gödel seeing that mathematics could describe mathematics itself. This isn't combining existing patterns, but seeing entirely new levels of abstraction. The human brain excels at taking complex systems and creating simpler mental models. Newton seeing planetary motion and falling apples as the same phenomenon. This compression isn't recombination - it's finding the hidden simplicity. Recombination adds elements together. Insight often removes elements to reveal core principles. This requires understanding and reasoning.
      • dead_gunslinger 2y ago
• southernplaces7 2y ago
  I see absolutely zero wrong with that statement. What he said is indeed much more reasoned and intelligent than the average foolish AI hype i've often found here, written by people who try to absurdly redefine the obvious, complex mystery that is consciousness into some reductionist notion of it being anything that presents the appearance of reasoning through technical tricks. Chiang has it exactly right with his doubts, and the notion that pattern recognition is little different from the deeply complex navigation of reality we living things do is the badly misguided notion.
  • munksbeer 2y ago
    > Chiang has it exactly right with his doubts, and the notion that pattern recognition is little different from the deeply complex navigation of reality we living things do is the badly misguided notion. How do you know this?
    • daveguy 2y ago
      The same way that we know interpolation of a linear regression is not the same as the deeply complex navigation of reality we do as living things.
      • digbybk 2y ago
        I notice that often in these debates someone will make the comparison between a low level mechanism driving LLMs, and a high level emergent behavior of the human mind. I don't think it's deliberate - we don't fully understand how the brain works so we only have emergent behaviors - but how can you be so certain that deeply complex navigation of reality can't emerge from interpolation of a linear regression?
        • daveguy 2y ago
          That's a good question. With sufficient dimensionality, interaction terms, and enough linear regressions, I suppose it's possible. But dynamic and reactive coordination of many multiple linear regressions wouldn't be just a linear regression. The output of a linear regression is simplistic just like LLM token prediction is simplistic. Saying something might be a component of eventual intelligence is far from it being intelligence. LLMs are episodic responses to a fixed context by a fixed model that is programmed to predict tokens. Even the CoT models, while more complex, still use a static model with a recursive feed of model outputs back to the model. I think Dr. Chollet does an excellent job of identifying the fundamental difference between a potential AGI and static models in his ARC-AGI papers and presentations.
• jrflowers 2y ago
  This is a good point. If you prick an LLM does it not bleed? If you tickle it does it not laugh? If you poison one does it not die? If you wrong an LLM shall it not revenge?
• weego 2y ago
  Counter point: what is it about scraping the Internet and indexing it cleverly that makes you believe that would lead to the the creation of the ability to reason above it's programming? No one in neuroscience, psychology or any related field can point to reasoning or 'consciousness' or whatever you wish to call it and say it appeared from X. Yet we have this West Coast IT cultish thinking that if we throw money at it we'll just spontaneously get there. The idea that we're even 1% close should be ridiculous to anyone rationally looking at what we're currently doing.
  • rcxdude 2y ago
    It's more that if you actually work with LLMs they will display reasoning. It's not particularly good or deep reasoning (I would generally say they have a superhuman amount of knowledge but are really quite unintelligent), but it is more than simply recall.
    • abecedarius 2y ago
      Yes, my judgement too from messing with Claude and (previously) ChatGPT. 'Ridiculous' and 'cultish' are overton-window enforcement more than they are justified.
    • d4mi3n 2y ago
      Waters are often muddied here by our own psychology. We (as a species) tend to ascribe intelligence to things that can speak. Even more so when someone (or thing in this case) can not just speak, but articulate well. We know these are algorithms, but how many people fall in love or make friends over nothing but a letter or text message? Capabilities for reasoning aside, we should all be very careful of our perceptions of intelligence based solely on a machines or algorithms apparent ability to communicate.
      • kmmlng 2y ago
        This seems like the classic shifting of goalposts to determine when AI has actually become intelligent. Is the ability to communicate not a form of intelligence? We don't have to pretend like these models are super intelligent, but to deny them any intelligence seems too far for me.
• infBIGlilnums 2y ago
  What is intelligent person? You seem to have approached the article with an existing reverence. Rather 1984 to look at the contribution of an academic and an iron welder and see authority in someone who memorized the book, but not how to keep themselves alive. Chiang and the like are nihilists, indifferent if they die cause it all just goes dark to them. Indifferent to the toll they extract from labor to fly their ass around speaking about glyphs in a textbook. Academics detached from the real work people need are just as draining on society and infuriating as a billionaire CEO and tribal shaman. Especially these days when they derive some small normalization from 100s of years of cataloged work and proclaim their bit of syntactic art is all they should need to spend the rest of their life being celebrated like they’re turning 8 all over again. Grigori Perelman is the only intelligent person out there I respect. Copy-paste college grads all over the US recite the textbook and act like it’s a magical incantation that bends the will of others. Cult of social incompetence in the US.
• unification_fan 2y ago
  >and wonder how an intelligent person can still think this, Read up on the ELIZA effect
• suddenexample 2y ago
  I think the "LLM is intelligence" crowd has a very simplistic view of people. If you feel that natural language and the systems responsible it are pretty much the only things that human intelligence produces, then I can see the argument. But I don't believe that. That a machine that can produce convincing human-language chains of thought says nothing about its "intelligence". Back when basic RNNs/LSTMs were at the forefront of ML research, no one had any delusions about this fact. And just because you can train a token prediction model on all of human knowledge (which the internet is not) doesn't mean the model understands anything. It's surprising to me that the people most knowledgeable about the models often appear to be the biggest believers - perhaps they're self-interestedly pumping a valuation or are simply obsessed with the idea of building something straight from the science fiction stories they grew up with. In the end though, the burden of proof is on the believers, not the deniers.
  • vhantz 2y ago
    > It's surprising to me that the people most knowledgeable about the models often appear to be the biggest believers - perhaps they're self-interestedly pumping a valuation or are simply obsessed with the idea of building something straight from the science fiction stories they grew up with. "Believer" really is the most appropriate label here. Altman or Musk lying and pretending they "AGI" right around the corner to pump their stocks is to be expected. The actual knowledgeable making completely irrational claims is simply incomprehensible beyond narcissism and obscurantism. Interestingly, those who argue against the fiction that current models are reasoning, are using reason to make their points. A non-reasoning system generating plausible text is not at all a mystery can be explained, therefore, it's not sufficient for a system to generate plausible text to qualify as reasoning. Those who are hyping the emergence of intelligence out of statistical models of written language on the other hand rely strictly on the basest empiricism, e.g. "I have an interaction with ChatGPT that proves it's intelligent" or "I put your argument into ChatGPT and here's what it said, isn't that interestingly insightful". But I don't see anyone coming out with any reasoning on how ability to reason could emerge out of a system predicting text. There's also a tacit connection made between those language models being large and complex and their supposed intelligence. The human brain is large and complex, and it's the material basis of human intelligence, "therefore expensive large language models with internal behavior completely unexplainable to us, must be intelligent". I don't think it will, but if the release of the deepseek models effectively shifts the main focus towards efficiency as opposed to "throwing more GPUs at it", that will also force the field to produce models with the current behavior using only the bare minimum, both in terms of architecture and resources. That would help against some aspects of the mysticism. The biggest believers are not the best placed to drive the research forward. They are not looking at it critically and trying to understand it. They are using every generated sentence as a confirmation of their preconceptions. If the most knowledgeable are indeed the biggest believers, we are in for a long dark (mystic) AI winter.
• deleted 2y ago
  [deleted]
• pif 2y ago
  Please, wake me up when artificial so-called intelligence will have proved a new theorem.
• SkiFire13 2y ago
  In theory you can prove a theorem just by enumerating all the possible proofs until you find the one for the theorem you want. This is extremely slow, but do you think there's any reasoning in doing this? Of course we don't know whether an LLM is doing something like this or actually reasoning. But this is also the point, we don't know. If you ask a question to a person you can be confident to some degree that they didn't memorize the answer beforehand, so you can evaluate their ability to "reason" and come up with an answer for it. With an LLM however this is increadibly hard to do, because they could have memorized it.
  • bell-cot 2y ago
    > In theory you can prove a theorem just by enumerating all the possible proofs until ... An interesting hypothesis! I'm neither a mathematical logician, nor decently up to date in that field - is the possibility of this, at least in the abstract, currently accepted as fact? (Yes, there's the perhaps-separate issue of only enumerating correct proofs.)
    • SkiFire13 2y ago
      It depends on what theory you're working in (at which point deciding whether to use one theory or another becomes more like a phisolophical question). I'm mostly familiar with type theory, of which there are many variants, but the most common ones all share the most important characteristics. In particular they identify theorems with types, and proofs with terms, where correct proofs are well-typed terms. The nice thing is that terms are recursively enumerable, so you can list all proofs. Moreover most type theories have decidable type checking, so you can automatically check whether a terms if well-typed (and hence the corresponding proof is correct). This is not just theory, there exist already a bunch of tools that are being used in practice for mechanically checking mathematical proofs, like Coq, Lean, Agda and more. When I said "in theory" however it's because in practice enumerating all proof terms will be very very slow and will take forever to reach proofs for theorems that we might find interesting. Since we're in the LLM topic, there are efforts to use LLMs to speed up this search, though this is more similar to using them as search heuristics though. It does help though that you can have automatic feedback thanks to the aforementioned proof checking tools, meaning you don't need costly human supervision to train them. The hope would be getting something like what Stockfish/Alphazero is for chess.
• ip26 2y ago
  Can an LLM discover a new theory of natural law, such as prove or disprove string theory? This is what I ponder. The creation or discovery of something new that it can’t just copy, that would have to be discovered via human thought otherwise. Something provably true. The equivalent of discovering general relativity before humans had.
  • whatever1 2y ago
    I mean the vast vast majority of people cannot prove/disprove theorems either, but we still consider ourselves as intelligent.
    • ip26 2y ago
      I’m not demanding it discover new science before it can be called intelligent. It’s a thought experiment.
  • RationPhantoms 2y ago
    I feel like there is likely far more useful answers in overlooked science then there is in "new" science.
• slibhb 2y ago
  I agree with Chiang. Reminds me of Searle and The Chinese Room (I agree with Searle too). I do think that at some point everyone is just arguing semantics. Chiang is arguing that "actual reasoning" is, by definition, not something that an LLM can do. And I do think he's right. But the real story is not "LLMs can't do X special thing that only biological life can do," the real story is "X special thing that only biological life can do isn't necessary to build incredibe AI that in many ways surpasses biological life".
• notjoemama 2y ago
  > how an intelligent person can still think this Cognitive neuroscience “qualia” Ray Kurzweil I’ll take “things OP doesn’t know about that an intelligent person does” for 800 Alex. If you’re enamored with LLMs and can’t see the inherent problems, you don’t actually know about AI and machine learning.
• mapt 2y ago
  Chiang is a silly blob of meat, of course he's not capable of actual reasoning, much less "intelligence". We grant him personhood, but personhood, like the LA Review of Books, is just a social construct.