10 ms·
Understanding Reasoning LLMs
- behnamoh 2y agodoesn't it seem like these models are getting to the point where even conceiving their training and development is less and less possible for the general public? I mean, we already knew only a handful of companies with capital could train them, but at least the principles, algorithms, etc. were accessible to individuals who wanted to create their own - much simpler - models. it seems that era is quickly ending, and we are entering the era of truly "magic" AI models that no one knows how they work because companies keep their secret sauces...
- HarHarVeryFunny 2y agoI don't think it's realistic to expect to have access to the same training data as the big labs that are paying people to generate it for them, but hopefully there will be open source ones that are still decent. At the end of the day current O1-like reasoning models are still just fine-tuned LLMs, and don't even need RL if you have access to (or can generate) a suitable training set. The DeepSeek R1 paper outlined their bootstrapping process, and HuggingFace (and no doubt others) are trying to duplicate it.
- antirez 2y agoIn recent weeks what's happening is exactly the contrary.
- fspeech 2y agoRecent developments like V3, R1 and S1 are actually clarifying and pointing towards more understandable, efficient and therefore more accessible models.
- tmnvdb 2y agoWe have been in the 'magic scaling' era for a while now. While the basic architecture of language models is reasonably simple and well understood, the emergent effects of making models bigger are largely magic even to the researchers, only to be studied emperically after the fact.
- dr_dshiv 2y agoHow important is it that the reasoning takes place in another thread versus just chain-of-thought in the same thread? I feel like it makes a difference, but I have no evidence.
- vector_spaces 2y agoIs there any work being done in training LLMs on more restricted formal languages? Something like a constraint solver or automated theorem prover, but much lower level. Specifically something that isn't natural language. That's the only path I could see towards reasoning models being truly effective I know there is work being done with e.g. Lean integration with ChatGPT, but that's not what I mean exactly -- there's still this shakey natural-language-trained-LLM glue in the driver's seat Like I'm envisioning something that has the creativity to try different things, but then JIT compile their chain of thought, and avoid bad paths
- mindwok 2y agoHow would that be different from something like ChatGPT executing Lean? That's exactly what humans do, we have messy reasoning that we then write down in formal logic and compile to see if it holds.
- gsam 2y agoIn my mind, the pure reinforcement learning approach of DeepSeek is the most practical way to do this. Essentially it needs to continually refine and find more sound(?) subspaces of the latent (embedding) space. Now this could be the subspace which is just Python code (or some other human-invented subspace), but I don't think that would be optimal for the overall architecture. The reason why it seems the most reasonable path is because when you create restrictions like this you hamper search viability (and in a high multi-dimensional subspace, that's a massive loss because you can arrive at a result from many directions). It's like regular genetic programming vs typed-genetic programming. When you discard all your useful results, you can't go anywhere near as fast. There will be a threshold where constructivist, generative schemes (e.g. reasoning with automata and all kinds of fun we've neglected) will be the way forward, but I don't think we've hit that point yet. It seems to me that such a point does exist because if you have fast heuristics on when types unify, you no longer hamper the search speed but gain many benefits in soundness. One of the greatest human achievements of all time is probably this latent embedding space -- one that we can actually interface with. It's a new lingua franca. These are just my cloudy current thoughts.
- 2y ago
- prideout 2y agoThis article has a superb diagram of the DeepSeek training pipeline.
- aithrowawaycomm 2y agoI like Raschka's writing, even if he is considerably more optimistic about this tech than I am. But I think it's inappropriate to claim that models like R1 are "good at deductive or inductive reasoning" when that is demonstrably not true, they are incapable of even the simplest "out-of-distribution" deductive reasoning: https://xcancel.com/JJitsev/status/1883158738661691878 https://xcancel.com/JJitsev/status/1883158738661691878 They are certainly capable of doing is a wide variety of computations that simulate reasoning, and maybe that's good enough for your use case. But it is unpredictably brittle unless you spend a lot on o1-pro (and even then...). Raschka has a line about "whether and how an LLM actually 'thinks' is a separate discussion" but this isn't about semantics. R1 clearly sucks at deductive reasoning and you will not understand "reasoning" LLMs if you take DeepSeek's claims at face value. It seems especially incurious for him to copy-paste the "a-ha moment" from Deepseek's technical report without critically investigating it. DeepSeek's claims are unscientific, without real evidence, and seem focused on hype and investment: This moment is not only an "aha moment" for the model but also for the researchers observing its behavior. It underscores the power and beauty of reinforcement learning: rather than explicitly teaching the model on how to solve a problem, we simply provide it with the right incentives, and it autonomously develops advanced problem-solving strategies. The "aha moment" serves as a powerful reminder of the potential of RL to unlock new levels of intelligence in artificial systems, paving the way for more autonomous and adaptive models in the future. Perhaps it was able to solve that tricky Olympiad problem, but there are an infinite variety of 1st grade math problems it is not able to solve. I doubt it's even reliably able to solve simple variations of that root problem. Maybe it is! But it's frustrating how little skepticism there is about CoT, reasoning traces, etc.
- scarmig 2y ago> But I think it's inappropriate to claim that models like R1 are "good at deductive or inductive reasoning" when that is demonstrably not true, they are incapable of even the simplest "out-of-distribution" deductive reasoning: https://xcancel.com/JJitsev/status/1883158738661691878 https://xcancel.com/JJitsev/status/1883158738661691878 Your link says that R1, not all models like R1, fails at generalization. Of particular note: > We expose DeepSeek R1 to the variations of AIW Friends problem and compare model behavior to o1-preview, o1-mini and Claude 3.5 Sonnet. o1-preview handles the problem robustly, DeepSeek R1 shows strong fluctuations across variations with distribution very similar to o1-mini.
- oxqbldpxo 2y agoAmazing accomplishments by brightest minds only to be used to write history by the stupidest people.
- gibsonf1 2y agoThere are no LLMs that reason, its an entirely different statistical process as compared to human reasoning.
- tmnvdb 2y ago"There are no LLMS that reason" is a claim about language, namely that the word 'reason' can only ever be applied to humans.
- gibsonf1 2y agoNot at all, we are building conceptual reasoning machines, but it is an entirely different technology than GPT/LLM dl/ml etc. [1] [1] https://graphmetrix.com/trinpod-server https://graphmetrix.com/trinpod-server
- freilanzer 2y agoIf LLMs can't reason, then this cannot either - whatever this is supposed to be. Not a good argument. Also, since you're apparently working on that product: 'It is difficult to get a man to understand something when his salary depends on his not understanding it.'
- tmnvdb 2y agoConceptual reasoning machines rely on concrete, explicit and intelligble concepts and rules. People like this because it 'looks' like reasoning on the inside. However, our brains, like language models, rely on implicit, distributed representations of concepts and rules. So the intelligble representations of conceptual reasoning machines are maybe too strong a requirement for 'reasoning' unless you want to exclude humans too.
- gibsonf1 2y agoIt’s also possible that you do not have information on our technology which models conceptual awareness of matter and change through space-time which is different than any previous attempts?
- dhfbshfbu4u3 2y agoGreat post, but every time I read something like this I feel like I am living in a prequel to the Culture.
- BarryMilo 2y agoIs that bad? The Culture is pretty cool I think. I doubt the real thing would be so similar to us but who knows.
- dhfbshfbu4u3 2y agoOh no, I’d live on an Orbital in a heartbeat. No, it’s just that all of these kinds of posts make me feel like we’re about to live through “The Bad Old Days”.
- robertlagrant 2y agoIt's cool to read about, but there's a reason most of the stories are not about living as a person in the Culture. It sounds extremely dull.
- mrob 2y agoIt doesn't sound dull to me. The stories are about the periphery of the Culture because that gets the most storytelling value out of the effort that went into worldbuilding, not because it would be impossible to write interesting stories about ordinary Culture members. I don't think you need external threats to give life meaning. Look at the popularity of sports in real life. The challenge there is self-imposed, but people still care greatly about who wins.
- robertlagrant 2y ago> I don't think you need external threats to give life meaning. I didn't say people did. But overcoming real challenges seems to be a big part of feeling alive, and I wonder if we really all would settle back into going for walks all day or whatever we could do that entertain us without needing others to work to provide the entertainment. Perhaps the WALL-E future, where we sit in chairs? But with AI-generated content?
- ngneer 2y agoNice article. >Whether and how an LLM actually "thinks" is a separate discussion. The "whether" is hardly a discussion at all. Or, at least one that was settled long ago. "The question of whether a computer can think is no more interesting than the question of whether a submarine can swim." --Edsger Dijkstra
- cwillu 2y agoThe document that quote comes from is hardly a definitive discussion of the topic. “[…] it tends to divert the research effort into directions in which science can not—and hence should not try to—contribute.” is a pretty myopic take. --http://www.cs.utexas.edu/users/EWD/ewd08xx/EWD898.PDF http://www.cs.utexas.edu/users/EWD/ewd08xx/EWD898.PDF
- ngneer 2y agoDijkstra myopic. Got it.
- alonsonic 2y agoDijkstra is clearly approaching the subject from an engineer/scientist more practical pov. His focus is on the application of the technology to solve problems, from that pov whether AI fits the definition of "human thinking" is indeed uninteresting.
- deleted 2y ago[deleted]
- 2y ago
- janalsncm 2y agoNice explainer. The R1 paper is a relatively easy read. Very approachable, almost conversational. I say this because I am constantly annoyed by poor, opaque writing in other instances. In this case, DS doesn’t need to try to sound smart. The results speak for themselves. I recommend anyone who is interested in the topic to read the R1 paper, their V3 paper, and DeepSeekMath paper. They’re all worth it.
- yosito 2y agoAre there any websites that show the results of popular models on different benchmarks, which are explained in plain language? As an end user, I'd love a quick way to compare different models suitability for different tasks.
- champdebloom 2y agoHere’s a site with graphs you can use to visually compare model benchmarks: https://artificialanalysis.ai https://artificialanalysis.ai
- sigbottle 2y agoOne thing I don't like about the trend in reasoning LLMs is the over-optimization to coding problems / math problems in particular. A lot of things that aren't well-defined require reasoning, and not just in a "SWE is ambiguous" kind of way - for example, thinking about how to present/teach something in a good way, iterating with the learner, thinking about what context they could be missing, etc. I find that all of these reasoning models really will overfit and overthink if you attach some level of math problem to it but it will barely think for anything else. I had friends suggest to me (I can't tell if in jest or seriously) that other fields don't require thinking, but I dunno, a lot of these "soft things" I think about really hard and don't have great solutions to. I've always been a fan of self-learning, for example - wouldn't it be great to have a conversation partner who can both infer and understand your misconceptions about complex topics when trying to learn, just from a few sentences, and then guide you for that? It's not like it's fundamentally impossible. These LLMs definitely can solve harder coding problems when you make them think. It's just that, I'm pretty sure (and it's really noticable with deepseek) that they're overfit towards coding/math puzzles in particular. It's really noticable with deepseek when you ask its reasoning model to just write some boilerplate code... you can tell it's completely overfit because it will just overthink and overthink and overthink. But it doesn't do that for example, with "soft" questions. In my opinion, this points to the idea that it's not really deciding for itself "how much thinking is enough thinking" and that it's just really overfit. Which I think can be solved, again, but I think it's more of a training decision issue.
- bloomingkales 2y agoIt's a human bias that also exists outside of this current problem space. Take programmers for example, there is a strong bias that is pushed about how mathematically oriented minds are better at programming. This bias has shown up in the training phase of AI, as we believe programming patterns lead to better reasoning (train them on code examples, and then distill the model down, as it now has the magical prowess of a mathematically oriented mind, so they say). When it comes to AI ethics, this is an ethical problem for those that don't think about this stuff. We're seeding these models with our own agenda. These concepts will be shattered in the long run hopefully, because they are so small.
- 2y ago
- bloomingkales 2y agoAbout three months ago, I kinda casually suggested to HN that I was using a form of refining to improve my LLMs, which is now being described as "reasoning" in this article and other places. My response a few months ago (Scroll down to my username and read that discussion): https://news.ycombinator.com/item?id=41997727 https://news.ycombinator.com/item?id=41997727 If only I knew DeepSeek was going to tank the market with something as simple as that lol. Note to self, take your intuition seriously.
- aqueueaqueue 2y agohttps://news.ycombinator.com/item?id=42001061 https://news.ycombinator.com/item?id=42001061 is the link I think....
- daxfohl 2y agoI wonder what it would look like in multi modal, if the reasoning part was an image or video or 3D scene instead of text.
- ttul 2y agoOr just embeddings that only make sense to the model. It’s really arbitrary, after all.
- daxfohl 2y agoThat's what I was thinking too, though with an image you could do a convolution layer and, idk, maybe that makes it imagine visually. Or actually, the reasoning is backwards: the convolution layer is what (potentially) makes that part behave like an image. It's all just raw numbers at the IO layers. But the convolution could keep it from overfitting. And if you also want to give it a little binary array as a scratch pad that just goes straight to the RELUs, why not? Seems more like human reasoning. A little language, a little visual, a little binary / unknown.
- daxfohl 2y agoBut how on earth do you train it? With regular LLMs, you get feedback on each word / token you generate, as you can match against training text. With these, you've got to generate hundreds of tokens in the thinking block fiest, and even after that, there's no "matching" next word, only a full solution. And it's either right or wrong, no probabilities to do a gradient on.
- NitpickLawyer 2y ago> only a full solution. And it's either right or wrong, no probabilities to do a gradient on. You could use reward functions that do a lot more complicated stuff than "ground_truth == boxed_answer". You could, for example split the "CoT" in paragraphs, and count how many paragraphs match whatever you consider a "good answer" in whatever topic you're trying to improve. You can use embeddings, or fuzzy string matches, or even other LLMs / reward models. I think math and coding were explored first because they're easier to "score", but you could attempt it with other things as well.
- daxfohl 2y agoBut it has to emit hundreds of tokens per test. Does that mean it takes hundreds of times longer to train? Or longer because I imagine the feedback loop can cause huge instabilities in gradients. Or are all GPTs trained on longer formats now; i.e. is "next word prediction" just a basic thing from the beginning of the transformers era?
- Davidzheng 2y agotakes a long time yes, but not longer than pretraining. sparse rewards are a common issue in RL and addressed by many techniques (I'm not expert so I can't say more). Model only does next word prediction and generates a number of trajectories, the correct ones get rewarded (those predictions in the correct trajectory have their gradients propagated back and reinforced).
- daxfohl 2y ago
- mohsen1 2y agoThe guys at Unsloth did a great job making this workflow accessible: https://news.ycombinator.com/item?id=42969736 https://news.ycombinator.com/item?id=42969736
- colordrops 2y agoThe article talks about how you should choose the right tool for the job, meaning that reasoning and non reasoning models have tradeoffs, and lists a table of criteria for selecting between model classes. Why couldn't a single model choose to reason or not itself? Or is this what "mixture of experts" is?
- goingcrazythro 2y agoI was having a look at the DeepSeek-R1 technical report and found the "aha moment" claims quite smelly, given that they do not disclose if the base model contains any chain of thought or reasoning data. However, we know the base model is DeepSeek V3. From the DeepSeek V3 technical report, paragraph in 5.1. Supervised Fine-Tuning: > Reasoning Data. For reasoning-related datasets, including those focused on mathematics, code competition problems, and logic puzzles, we generate the data by leveraging an internal DeepSeek-R1 model. Specifically, while the R1-generated data demonstrates strong accuracy, it suffers from issues such as overthinking, poor formatting, and excessive length. Our objective is to balance the high accuracy of R1-generated reasoning data and the clarity and conciseness of regularly formatted reasoning data. In 5.4.1 they also talk about some ablation experiment by not using the "internal DeepSeek-R1" generated data. While the "internal DeepSeek-R1" model is not explained, I would assume this is a DeepSeek V2 or V2.5 tuned for chain of thought. Therefore, it seems to me the "aha moment" is just promoting the behaviour that was already present in V3. In the "Self-evolution Process of DeepSeek-R1-Zero"/ Figure 3 they claim reinforcement learning also leads to the model generating longer CoT sequences, but again, this comes from V3, they even mention the fine tuning with "internal R1" led to "excessive length". None of the blogpost, news, articles I have read explaining or commenting on DeepSeek R1 takes this into account. The community is scrambling to re-implement the pipeline (see open-r1). At this point, I feel like I took a crazy pill. Am I interpreting this completely wrong? Can someone shed some light on this?
- nvtop 2y agoI'm also very skeptical of the significance of this "aha moment". Even if they didn't include chain-of-thoughts to the base model's training data (unlikely), there are still plenty of it on the modern Internet. OpenAI released 800k of reasoning steps which are publicly available, github repositories, examples in CoT papers... It's definitely not a novel concept for a model, that it somehow discovered by its own.
- tmnvdb 2y agohttps://oatllm.notion.site/oat-zero https://oatllm.notion.site/oat-zero
- mike_hearn 2y agoIt's curious that the models switch between languages and that has to be trained out of them. I guess the ablations have been done already, but it makes me wonder if they do this because it's somehow easier to do some parts of the reasoning on languages other than English and maybe they should be just allowed to get on with it?
- Dansvidania 2y agoAre reasoning models -basically- generating their own context? as in, if a user were to feed prompt + those reasoning tokens as a prompt to a non-reasoning model, would the effect be functionally similar? I am sure this is improperly worded, I apologise.
- aldanor 2y agoYes, more or less. Just like any LLM "generates its own context", during inference it doesn't care where the previous tokens came from. Inference doesn't have to change much, it's the training process that's different.
- Dansvidania 2y agothank you, that makes sense. Now it's time to really read the article to understand if the difference is the training data or the network topology to be different (although I lean towards the latter).
- lysecret 2y agoI think the next big problem we will run into with these line of reasoning models is "over-thinking" you can already start to see it. Thinking harder is not the universal pareto improvement everyone seems to think it is. (I understand the irony of using think 4 times here haha)
- seydor 2y agoReasoning is about serially applying a set of premises over and over to come to conclusions. But some of our biggest problems require thinking outside the box, sometimes way outside it, and a few times ingeniously making up a whole new set of premises, seemingly ex nihilo (or via divine inspiration). We are still in very early stages of making thinking machines.
- tpswa 2y agoThis is a natural next area of research. Nailing "adaptive compute" implies figuring out which problems to use more compute on, but I imagine this will get better as the RL does.
- resource_waste 2y ago100% I do philosophy and it will take an exaggeration I give it, and call it fact. The non reasoning models will call me out. lol
- EncomLab 2y agoHaven't we seen real life examples of this occurring in AI for medical imaging? Models trained on images of tumors state that tumors circled in purple ink or images of tumors that also include a visual scale are over identified as cancerous because they reason that both of those items indicate cancer due to the training data leading them that way?
- efitz 2y agoWhat everyone needs to understand about [reasoning] LLMs is that LLMs can’t reason. https://arxiv.org/pdf/2410.05229 https://arxiv.org/pdf/2410.05229
- MoneCollinsgdjd 2y ago[dead]
- MikeBenemorshdj 2y ago[dead]
- charleneusanhsk 2y ago[dead]
- michealforbes 2y ago[dead]
- jamielwenger 2y ago[dead]
- shellasteve 2y ago[flagged]
- MikeBenemorhbc 2y ago[dead]