10 ms·
Explainer: What's r1 and everything else?
- comrade1234 2y agoThe benchmarks for the different models focus on math and coding accuracy. I have a use-case for a model where those two functions are completely irrelevant and I’m only interested in writing (chat, stories, etc). I guess you can’t really benchmark ‘concepts’ as easily as logic. With distillation, can a model be made that strips out most of the math and coding stuff?
- ben30 2y agoYes, you can create a writing-focused model through distillation, but it's tricky. *Complete removal* of math/coding abilities is challenging because language models' knowledge is interconnected - the logical thinking that helps solve equations also helps structure coherent stories.
- mercer 2y agoI understood that at least some of these big models (llama?) is basically bootstrapped with code. is there truth to that?
- ben30 2y agoYes, code is a key training component. Open-Llama explicitly used programming data as one of seven training components. However, newer models like Llama 3.1 405B have shifted to using synthetic data instead. Code helps develop structured reasoning patterns but isn't the sole foundation - models combine it with general web text, books, etc.
- NitpickLawyer 2y ago> completely irrelevant and I’m only interested in writing (chat, stories, etc) There's a person keeping track of a few writing prompts and the evolution of the quality of text with each new shiny model. They shared this link somewhere, can't find the source but I had it bookmarked for further reading. Have a look at it and see if it's something you'd like. https://eqbench.com/results/creative-writing-v2/deepseek-ai__DeepSeek-R1.txt https://eqbench.com/results/creative-writing-v2/deepseek-ai_...
- comrade1234 2y agoThose outputs are really good and come from deepseek-R1 (I assume the full version, not a distilled version). R1 is quite large (685B params). I’m wondering if you can make a distilled R1 without the coding and math content. 7B works well for me locally. When I go up to 32B I seem to get worse results - I assume it’s just timing out in its think mode… I haven’t had time to really investigate though.
- afro88 2y agoHere's a better link: https://eqbench.com/creative_writing.html https://eqbench.com/creative_writing.html The R1 sample reads way better than anything else on the leaderboard to me. Quite a jump.
- exikyut 2y agoCompletely agree. The only measurable flaw I could find was the errant use of an opening quote (‘) in > He huffed a laugh. "Lucky you." His gaze drifted to the stained-glass window, where rain blurred the world into watercolors. "I bombed my first audition. Hamlet, uni production. Forgot ‘to be or not to be,' panicked, and quoted Toy Story." It's pretty amazing I can find no fault with the actual text. No grammar errors, I like the writing, it competes with the quality and engagingness of a large swath of written fiction (yikes), I wanna read the next chapter.
- NitpickLawyer 2y ago> It's pretty amazing I can find no fault with the actual text. No grammar errors, I like the writing, it competes with the quality and engagingness of a large swath of written fiction (yikes), I wanna read the next chapter. The lack of "gpt-isms" is really impressive IMO.
- polynomial 2y agoWhy is the main character named Rhys in most (?) of them? Llama[1], Claude[3], Mistral[4] & DeepSeek-r1[5] samples all named the main character Rhys, even though that was no where specified in the prompt? GPT-4o gives the character a different name[6]. Gemini[2] names the bookshop person Rhys instead! Am I just missing something really obvious? I feel like I'm missing something big that's right in front of me [1] https://eqbench.com/results/creative-writing-v2/meta-llama__Llama-3-70b-chat-hf.txt https://eqbench.com/results/creative-writing-v2/meta-llama__... [2] https://eqbench.com/results/creative-writing-v2/gemini-1.5-flash-001.txt https://eqbench.com/results/creative-writing-v2/gemini-1.5-f... [3] https://eqbench.com/results/creative-writing-v2/claude-3-opus-20240229.txt https://eqbench.com/results/creative-writing-v2/claude-3-opu... [4] https://eqbench.com/results/creative-writing-v2/mistralai__Mistral-Large-Instruct-2407.txt https://eqbench.com/results/creative-writing-v2/mistralai__M... [5] https://eqbench.com/results/creative-writing-v2/deepseek-ai__DeepSeek-R1.txt https://eqbench.com/results/creative-writing-v2/deepseek-ai_... [6] https://eqbench.com/results/creative-writing-v2/gpt-4o-2024-11-20.txt https://eqbench.com/results/creative-writing-v2/gpt-4o-2024-...
- polotics 2y agoInteresting article, but the flourish ending """AI will soon (if not already) increase in abilities at an exponential rate.""" is not at all substantiated. Would be nice to know how the author gets to that conclusion.
- ioulaum 2y agoThe exponential part may be iffy, but it is self improving. And this same RL is also creating improvements in small model performance. So, more LLMs are about to rise in quality.
- immibis 2y agoIt's self-improving? So, we can ask AI how to improve AI, and the suggestions actually work?
- sesm 2y agoIt's more like Intel in early days using their CPUs to compute layout for bigger CPUs.
- ethbr1 2y agoEffectively: is the limiting factor to improvement addressable by the thing being improved? If yes, then you get exponential increases very trivially. If no, then something external continues to bottleneck progress.
- FridgeSeal 2y agoIt’s basically the line for all the AI-hype people: “all the problems are going away!”, “soon it’ll all magically make things exponentially good-er-er!”
- brookst 2y agoAlternatively, it’s a restatement of the obvious empirical truth that technology tends in improve on an exponential and not linear curve. Seems like a simpler explanation that doesn’t even require insulting people.
- geuis 2y ago[flagged]
- JimmyBuckets 2y agoWhy are you so angry? I thought it was a wonderful overview. An even if not, insults are hardly necessary.
- deleted 2y ago[deleted]
- petemir 2y agoPerhaps you are not the intended audience for this article.
- gregoriol 2y ago[flagged]
- 0xEF 2y agoWhat a useless comment. Point out your qualms with it, explain how you would have done it better or clarify inaccuracies you find. It helps promote discussion and opens the door for more collective information to be considered.
- fragmede 2y ago> Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- justlikereddit 2y agoShort version: It's Hype. Long version: It's marketing efforts stirring up hype around incremental software updates. If this was software being patched in 2005 we'd call it "ChatGPT V1.115" >Patch notes: >Added bells. >Added whistles.
- Havoc 2y ago>people re-creating R1 (some claim for $30) R1 or the R1 finetunes? Not the same thing... HF is busy recreating R1 itself but that seems to be a pretty big endevour not a $30 thing
- littlestymaar 2y agoThis is indeed a massive exaggeration, I'm pretty sure the $30 experiment is this one: https://threadreaderapp.com/thread/1882839370505621655.html https://threadreaderapp.com/thread/1882839370505621655.html (github: https://github.com/Jiayi-Pan/TinyZero https://github.com/Jiayi-Pan/TinyZero). And while this is true that this experiment shows that you can reproduce the concept of direct reinforcement learning of an existing LLM, in a way that makes it develop reasoning in the same fashion Deepseek-R1 did, this is very far from a re-creation of R1!
- m3kw9 2y agoMaybe they mistake recreation for the cp command
- 1123581321 2y agoNice explainer. R1 hit sensational mainstream news which has resulted in some confusion and alarm among family and friends. It’s hard to succinctly explain this doesn’t mean China is destroying us, that Americans immediately started working with the breakthrough, cost optimization is inevitable in computing, etc.
- huqedato 2y agoI know that I'll get a lot of hate and downvotes for this comment. I want to say that I have all the respect and admiration for these Chinese people, their ingenuity and their way of doing innovation even if they achieve this through technological theft and circumventing embargoes imposed by US (we all know how GPUs find their way into their hands). We are living a time with a multi-faceted war between the US, China, EU, Russia and others. One of the battlegrounds is AI supremacy. This war (as any war) isn’t about ethics; it’s about survival, and anything goes. Finally, as someone from Europe, I confess that here is well known that the "US innovates while EU regulates" and that's a shame IMO. I have the impression that EU is doing everything possible to keep us, European citizens, behind, just mere spectators in this tech war. We are already irrelevant, niche players.
- travisporter 2y agoBut Anthropic is a French company isn’t that in the EU
- throw310822 2y agoNo it's American, you probably confuse them with Mistral AI.
- travisporter 2y agoOh wow thank you yes I did
- huqedato 2y agoAnthropic is not French. You probably meant Mistral, that is a French company, but a niche player in this AI game.
- hackandthink 2y ago"We are living a time with a multi-faceted war between the US, China, EU, Russia and others." The only way to win this war is to deescalate. Everybody wins. And AI competition is a good thing for Europe especially when it lags behind technologically.
- simonw 2y agoFrom that article: > ARC-AGI is a benchmark that’s designed to be simple for humans but excruciatingly difficult for AI. In other words, when AI crushes this benchmark, it’s able to do what humans do. That's a misunderstanding of what ARC-AGI means. Here's what ARC-AGI creator François Chollet has to say: https://bsky.app/profile/fchollet.bsky.social/post/3les3izgdj22j https://bsky.app/profile/fchollet.bsky.social/post/3les3izgd... > I don't think people really appreciate how simple ARC-AGI-1 was, and what solving it really means. > It was designed as the simplest, most basic assessment of fluid intelligence possible. Failure to pass signifies a near-total inability to adapt or problem-solve in unfamiliar situations. > Passing it means your system exhibits non-zero fluid intelligence -- you're finally looking at something that isn't pure memorized skill. But it says rather little about how intelligent your system is, or how close to human intelligence it is.
- dartos 2y ago> That's a misunderstanding of what ARC-AGI means Misunderstanding benchmarks seems to be the first step to claiming human level intelligence. Additionally: > > ARC-AGI is a benchmark that’s designed to be simple for humans but excruciatingly difficult for AI. In other words, when AI crushes this benchmark, it’s able to do what humans do. Doesn’t even make logical sense.
- ethbr1 2y agoThis feels like a generalized extension of the classic mis-reasoned response to 'A computer can now play chess.' Common non-technical chain of thought after learning this: 'Previously, only humans could play chess. Now, computers can play chess. Therefore, computers can now do other things that previously only humans could do.' The error is assuming that problems can only be solved via levels of human-style general intelligence. Obviously, this is false from the way that computers calculate arithmetic, optimize via gradient descent, and innumerable other examples, but it does seem to be a common lay misunderstanding. Probably why IBM abused it with their Watson marketing. In reality, for reliable capabilities reasoning, the how matters very much.
- antonvs 2y ago
- bikamonki 2y agoSo, is AI already reasoning or not?
- ozten 2y agoYes. ARC AGI benchmark was supposed to last years and is already saturated. The authors are currently creating the second version.
- brookst 2y agoDepends on your definition of reasoning. Creating valid chains of thought? Yes. Sentient? No.
- jug 2y agoYes, that's what OpenAI o1 does, and DeepSeek R1. Also Google Gemini 2.0 Thinking models. It's a way to significantly improve benchmark scores, especially in math. It's funny to watch too. I played with Gemini 2.0 on Google AI Studio and asked it to "come up with your favorite song as you take a long walk to really think this through". The reasoning can then be shown, and it talked to itself, saying things like "since I'm an AI, I can't take walks, but with a request like this, the user seems to imply that I should choose something that's introspective and meaningful", and went on with how it picked candidates.
- erlendstromsvik 2y agoI just tried that prompt with gemini-2.0-flash-thinking-exp-01-21 In the reasoning process it concludes on: From the brainstormed genres/artists, select a specific song. It's better to be concrete than vague. For this request, "Nuvole Bianche" by Ludovico Einaudi emerges as a strong candidate. Craft the Explanation and Scenario: Now, build the response around "Nuvole Bianche." Then in the actual answer it proposes: "Holocene" by Bon Iver. =)
- didibus 2y agoNo. AI learns to predict reasons, and doing so as it predicts the answer improves its accuracy at predicting the answer. In summary, even though they are called "reasoning" models, they are still based on prediction and pattern matching, not true logical reasoning. The improvement in accuracy is likely due to better leveraging of the model's statistical knowledge, rather than any deeper understanding of the problem's logic. And the reasons you see it output have nothing to do with the actual reasons it used to determine the answer. In fact, R1.Zero hints that, it might be even better to let the AI follow a chain of thought that doesn't actually make logical sense or is understandable, and that doing so could even further improve its ability to accurately predict solutions to code, math and logic problems.
- whimsicalism 2y agothis is a pretty hype-laden/twitter-laden article, i would not trust it to explain things to you
- thih9 2y agoSure. But perhaps some hype is justified? Here's what a senior research scientist from nvidia says: > We are living in a timeline where a non-US company is keeping the original mission of OpenAI alive - truly open, frontier research that empowers all. It makes no sense. The most entertaining outcome is the most likely. > DeepSeek-R1 not only open-sources a barrage of models but also spills all the training secrets. They are perhaps the first OSS project that shows major, sustained growth of an RL flywheel. (…) https://x.com/DrJimFan/status/1881353126210687089 https://x.com/DrJimFan/status/1881353126210687089
- rahimnathwani 2y agoMost important, R1 shut down some very complex ideas (like DPO & MCTS) and showed that the path forward is simple, basic RL. This isn't quite true. R1 used a mix of RL and supervised fine-tuning. The data used for supervised fine-tuning may have been model-generated, but the paper implies it was human-curated: they kept only the 'correct' answers.
- Philpax 2y agoI think what you're saying is consistent with the quote: human curation of SFT data is indeed not complex. There might be extra work on top of that RL, but it's the same work that's been done throughout LLM development.
- tkellogg 2y agoAdditionally, in the following days, I've seen evidence suggesting that the SFT part might not even be necessary. I'd argue that work wouldn't have happened if R1 wasn't released in the open.
- fullstackchris 2y agoSo the conclusion is AI is about to "increase in abilities at an exponential rate", with the only data point being that R1 was sucessfully able to acheive o1 levels as an open source model? In other words, two extremely unrelated themes? Does this guy know people were writing verbatim the same thing in like... 2021? Still always incredible to me the same repeated hype over and over rise to the surface. Oh well... old man gonna old man
- raincole 2y ago> Does this guy know people were writing verbatim the same thing in like... 2021? Given how far gen AIs have improved since 2021, these people were quite spot on.
- WesleyJohnson 2y agoSomeone should author an ELI5 (or slightly older) guide to how LLMs, RL, Agents, CoT, etc, all work and what all these acronyms mean. And then, add to it, daily or weekly, as new development arise. I don't want to keep reading dozens of articles, white papers, tweets, etc, as new developments happen. I want to go back to the same knowledge base, that's authored by the same person (or people), that maintains a consistent reading and comprehension level, and builds on prior points. It seems like the AI space is moving impossibly fast, and its just ridiculously hard to keep up unless 1) you work in this space, 2) are very comfortable with the technology behind it, so you can jump in at any point and understand it.
- raincole 2y agoPeople keep saying that DeepSeek R1's training cost is just $5.6M. Where is the source? I'm not asking for the proof. Just the source, even a self-claimed statement. I've read the R1's paper and it doesn't say the number of $5.6M. Is it somewhere in DeepSeek's press release?
- richardatlarge 2y agoT or F? Nobody really saw the LLM leap coming Nobody really saw R1 coming We don’t know what’s coming