12 ms·
Yann LeCun, Pioneer of AI, Thinks Today's LLM's Are Nearly Obsolete
- deleted 1y ago[deleted]
- ejang0 1y ago"[Yann LeCun] believes [current] LLMs will be largely obsolete within five years."
- onlyrealcuzzo 1y agoObsolete by? This seems like a broken clock having a good chance of being right. There's so much progress, it wouldn't be that surprising if something quite different completely overtakes the current trend within 5 years.
- mdp2021 1y ago> Obsolete by By NN models overcoming the pivot over representing language - according to LeCun in the article. It could be the Joint Embedding Predictive Architecture - we will see. > There's so much progress, it wouldn't be that surprising LeCun's point looks like a denunciation over an excessive focus over the LLM idea ("it works, so let's expand that" vs "it probably will not achieve the level of a satisfactory general model, so let us directly try to go beyond it").
- timewizard 1y agoObsolete by price. This technology only scales linearly. All the investment in it had a different growth expectation. I suspect this level of investment will eventually collapse.
- re-thc 1y ago> believes [current] LLMs will be largely obsolete within five years Well yes in that ChatGPT 4 (current) will be replaced by ChatGPT 5 (future) etc...
- gsf_emergency_2 1y agoRecent talk: https://www.youtube.com/watch?v=ETZfkkv6V7Y https://www.youtube.com/watch?v=ETZfkkv6V7Y LeCun, "Mathematical Obstacles on the Way to Human-Level AI" Slide (Why autoregressive models suck) https://xcancel.com/ravi_mohan/status/1906612309880930641 https://xcancel.com/ravi_mohan/status/1906612309880930641
- gibsonf1 1y agoThe error with that is that human reasoning is not mathematical. Math is just one of the many tools of reason.
- sho_hn 1y agoDid you read the slide? It doesn't make the argument you are responding to, you just seem to have been prompted by "Math".
- csdvrx 1y agoA more generous take on the previous post is that the dominant paradigm of Math (consistent logic, which depends on many things like transitive preference) is wrong, and that another type of Math could work. If you look at the slide, the subtree of correct answers exists, what's missing is just a way to make them more prevalent instead of less. Personally, I think LeCun is just leaping to the wrong conclusion because he's sticking to the wrong tools for the job.
- moralestapia 1y ago[flagged]
- csdvrx 1y ago> Returning to the topic of the limitations of LLMs, LeCun explains, "An LLM produces one token after another. It goes through a fixed amount of computation to produce a token, and that's clearly System 1—it's reactive, right? There's no reasoning," a reference to Daniel Kahneman's influential framework that distinguishes between the human brain's fast, intuitive method of thinking (System 1) and the method of slower, more deliberative reasoning (System 2). Many people believe that "wants" come first, and are then followed by rationalizations. It's also a theory that's supported by medical imaging. Maybe the LLM are a good emulation of system-2 (their perfomance sugggest it is), and what's missing is system-1, the "reptilian" brain, based on emotions like love, fear, aggression, (etc.). For all we know, the system-1 could use the same embeddings, and just work in parallel and produce tokens that are used to guide the system-2. Personally, I trust my "emotions" and "gut feelings": I believe they are things "not yet rationalized" by my system-2, coming straight from my system-1. I know it's very unpopular among nerds, but it has worked well enough for me!
- sho_hn 1y agoRe the "medical imaging" reference, many of those are built on top of one famous study recording movement before conscious realization that isn't as clear-cut as it entered popular knowledge as: https://www.theatlantic.com/health/archive/2019/09/free-will-bereitschaftspotential/597736/ https://www.theatlantic.com/health/archive/2019/09/free-will... I know there are other examples, and I'm not attacking your post; mainly it's a great opportunity to link this IMHO interesting article that interacts with many debates on HN.
- csdvrx 1y ago> IMHO interesting article that interacts with many debates on HN. It's paywalled
- ilaksh 1y agoI think what that shows is that in order for the fast reactions to be useful, they really have to incorporate holistic information effectively. That doesn't mean that slower conscious rational work can't lead to more precision, but does suggest that immediate reactions shouldn't necessarily be ignored. There is an analogy between that and reasoning versus non-reasoning with LLMs.
- GMoromisato 1y agoI remember reading Douglas Hofstadter's Fluid Concepts and Creative Analogies [https://en.wikipedia.org/wiki/Fluid_Concepts_and_Creative_Analogies https://en.wikipedia.org/wiki/Fluid_Concepts_and_Creative_An...] He wrote about Copycat, a program for understanding analogies ("abc is to 123 as cba is to ???"). The program worked at the symbolic level, in the sense that it hard-coded a network of relationships between words and characters. I wonder how close he was to "inventing" an LLM? The insight he needed was that instead of hard-coding patterns, he should have just trained on a vast set of patterns. Hofstadter focused on Copycat because he saw pattern-matching as the core ability of intelligence. Unlocking that, in his view, would unlock AI. And, of course, pattern-matching is exactly what LLMs are good for. I think he's right. Intelligence isn't about logic. In the early days of AI, people thought that a chess-playing computer would necessarily be intelligent, but that was clearly a dead-end. Logic is not the hard part. The hard part is pattern-matching. In fact, pattern-matching is all there is: That's a bear, run away; I'm in a restaurant, I need to order; this is like a binary tree, I can solve it recursively. I honestly can't come up with a situation that calls for intelligence that can't be solved by pattern-matching. In my opinion, LeCun is moving the goal-posts. He's saying LLMs make mistakes and therefore they aren't intelligent and aren't useful. Obviously that's wrong: humans make mistakes and are usually considered both intelligent and useful. I wonder if there is a necessary relationship between intelligence and mistakes. If you can solve a problem algorithmically (e.g., long-division) then there won't be mistakes, but you don't need intelligence (you just follow the algorithm). But if you need intelligence (because no algorithm exists) then there will always be mistakes.
- GeorgeTirebiter 1y agoWhat is Dark Matter? How to eradicate cancer? How to have world peace? I don't quite see how pattern-matching, alone, can solve questions like these.
- SpicyLemonZest 1y agoCancer eradication seems like a clear example of where highly effective pattern matching could be a game changer. That's where cancer research starts: pattern matching to sift through the incredibly large space of potential drugs and find the ones worth starting clinical trials for. If you could get an LLM to pattern-match whether a new compound is likely to work as a BTK inhibitor (https://en.wikipedia.org/wiki/Bruton%27s_tyrosine_kinase https://en.wikipedia.org/wiki/Bruton%27s_tyrosine_kinase), or screen them for likely side effects before even starting synthesis, that would be a big deal.
- asdev 1y agooutside of text generation and search, LLMs have not delivered any significant value
- baumy 1y agoText generation and search are the drivers for some trillions of dollars worth of economic activity around the world.
- mdp2021 1y ago> trillions of dollars That is monetary value. The poster may have meant "delivery" value - which has been limited (and tainted with hype). > Text generation Which «text generation», apart from code generation (quite successful in some models), would amount to «trillions of dollars worth of economic activity» at the current stage? I cannot see it at the moment.
- an_guy 1y agoI cannot see any major use case apart from code generation and autocompletion. Maybe summarizing text and learning new things but that can be achieved by any search engine. A good search engine could probably take over llm.
- ObnoxiousProxy 1y agothis statement is just patently wrong, and even those are still significant value. LLMs have been significantly impacting software engineering and software prototyping.
- SirensOfTitan 1y agoDo you have any evidence to suggest that LLMs have increased productivity in software? And if so what is the effect size? I don’t really see any increase in the quality, velocity, creativity of software I’m either using or working on, but I have seen a ton of candidates with real experience flunk out on interviews because they forget the basics and blame it on LLMs. I’ve honestly never seen candidates struggle with syntax the way I’m seeing them do so the last couple years. I find LLMs very useful for general overviews of domains but I’ve not really seen any use that is a clear unambiguous boost to productivity. You still have to read the code and understand it which takes time, but “vibe coding” prevents you from building cognitive load until that point, making code review costly. I feel like a lot of the hype in this space so far is aspirational, pushed by VCs, or a bunch of engineers who aren’t A/B testing: like for every hour you spend rubber ducking with an LLM, how much could you have gotten thought out with a pen and paper? In fact, usually I can go and enjoy my life more without an LLM: I write down the problem I’m considering and then go on a walk, and come back and have a mind full of new ideas.
- antirez 1y agoAs LLMs do things thought to be impossible before, LeCun adjusts his statements about LLMs, but at the same time his credibility goes lower and lower. He started saying that LLMs were just predicting words using a probabilistic model, like a better Markov Chain, basically. It was already pretty clear that this was not the case as even GPT3 could do summarization well enough, and there is no probabilistic link between the words of a text and the gist of the content, still he was saying that at the time of GPT3.5 I believe. Then he adjusted this vision when talking with Hinton publicly, saying "I don't deny there is more than just probabilistic thing...". He started saying: not longer just simply probabilistic but they can only regurgitate things they saw in the training set, often explicitly telling people that novel questions could NEVER solved by LLMs, with examples of prompts failing at the time he was saying that and so forth. Now reasoning models can solve problems they never saw, and o3 did huge progresses on ARC, so he adjusted again: for AGI we will need more. And so forth. So at this point it does not matter what you believe about LLMs: in general, to trust LeCun words is not a good idea. Add to this that LeCun is directing an AI lab that as the same point has the following huge issues: 1. Weakest ever LLM among the big labs with similar resources (and smaller resources: DeepSeek). 2. They say they are focusing on open source models, but the license is among the less open than the available open weight models. 3. LLMs and in general all the new AI wave puts CNNs, a field where LeCun worked (but that didn't started himself) a lot more in perspective, and now it's just a chapter in a book that is composed mostly of other techniques. Btw, other researchers that were in the LeCun side, changed side recently, saying that now "is different" because of CoT, that is the symbolic reasoning they were blabling before. But CoT is stil regressive next token without any architectural change, so, no, they were wrong, too.
- gcr 1y agoWhy is changing one’s mind when confronted with new evidence a negative signifier of reputation for you?
- antirez 1y agoBecause there were plenty of evidences that the statements were either not correct or not based on enough information, at the time they were made. And to be wrong because of personal biases, and then don't clearly state you were wrong when new evidenced appeared, is not a trait of a good scientist. For instance: the strong summarization abilities where already something that, alone, without any further information, were enough to seriously doubt about the stochastic parrot mental model.
- grandempire 1y agoIs this the guy who tweets all day and gets in online fights?
- dyarosla 1y agoNo he obviously quit twitter /s
- djoldman 1y agoThe idolatry and drama surrounding LeCun, Hinton, Schmidhuber, etc. is likely a distraction. This includes their various predictions. More interesting is their research work. JEPA is what LeCun is betting on: https://ai.meta.com/blog/v-jepa-yann-lecun-ai-model-video-joint-embedding-predictive-architecture/ https://ai.meta.com/blog/v-jepa-yann-lecun-ai-model-video-jo...
- varelse 1y ago[dead]
- redox99 1y agoLeCun has been very salty of LLMs ever since ChatGPT came out.
- deleted 1y ago[deleted]
- bitethecutebait 1y agothere's a bunch of stuff imperative to his thriving that has become obsolete to others 15 years ago ... maybe it's time for a few 'sabbatical' years ...