8 ms·
The Illustrated DeepSeek-R1
- caithrin 2y agoThis is fantastic work, thank you!
- jasonjmcghee 2y agoFor the uninitiated, this is the same author as the many other "The Illustrated..." blog posts. A particularly popular one: https://jalammar.github.io/illustrated-transformer/ https://jalammar.github.io/illustrated-transformer/ Always very high quality.
- punkspider 2y agoThanks so much for mentioning this. His name carries a lot of weight for me as well.
- jamestimmins 2y agoHave you read his book Hands-On Large Language Models? Looks interesting, but I'm skeptical that a book can feasibly stay up to date with the speed of development.
- jasonjmcghee 2y agoI have not, but Jay has created a ton of value and knowledge for free and don't fault him for throwing an ad for his book / trying to benefit a bit financially.
- jamestimmins 2y agoYeah no shade for someone selling their knowledge; I'm just trying to suss out how useful the book is for learning foundations.
- whoisburbansky 2y agoFoundations don't change much with "the speed of development"
- jamestimmins 2y agoThat's a good point
- jampekka 2y ago> Looks interesting, but I'm skeptical that a book can feasibly stay up to date with the speed of development. The basic structure of the base models has not really changed since the first GPT launched in 2018. You still have to understand gradient descent, tokenization, embeddings, self-attention, MLPs, supervised fine tuning, RLHF etc for the foreseeable future. Adding RL based CoT training would be a relatively straightforward addendum to a new edition, and it's an application of long established methods like PPO. All "generations" of models are presented as revolutionary -- and results-wise they maybe are -- but technically they are usually quite incremental "tweaks" to the previous architecture. Even more "radical" departures like state space models are closely related to same basic techniques and architectures.
- mistrial9 2y ago> gradient descent funny mentioning the math but not the Transformer encoders..
- jampekka 2y agoTransformer encoders are not really popular anymore, and all the top LLMs are decoder-only architectures. But encoder models like BERT are used for some tasks. In any case, self-attention and MLP is the crux of Transformer blocks, be they in the decoder or the encoder.
- mistrial9 2y ago> Transformer encoders are not really popular anymore references, please
- deleted 2y ago[deleted]
- blackeyeblitzar 2y agoThe thing I still don’t understand is how DeepSeek built the base model cheaply, and why their models seem to think they are GPT4 when asked. This article says the base model is from their previous paper, but that paper also doesn’t make clear what they trained on. The earlier paper is mostly a description of optimization techniques they applied. It does mention pretraining on 14.8T tokens with 2.7M H800 GPU hours to produce the base DeepSeek-V3. But what were those tokens? The paper describes the corpus only in vague ways.
- moralestapia 2y agoA friend just sent me a screenshot where he asks DeepSeek if it has an app for Mac and it replies that they have a ChatGPT app from OpenAI, lol. I 100% believe they distilled GPT-4, hence the low "training" cost.
- Philpax 2y agoEr, how would that reduce the cost? You still need to train the model, which is the expensive bit. Also, the base model for V3 and the only-RL-tuned R1-Zero are available, and they behave like base models, which seems unlikely if they used data from OpenAI as their primary data source. It's much more likely that they've consumed the background radiation of the web, where OpenAI contamination is dominant.
- Salgat 2y agoHypothetical question: is the chinese government capable of exploiting chatgpt to get around the query limit? For example, making queries through compromised devices or even snooping local traffic on devices? Let's face it, these models are closely alligned with China's national security so it's not a farfetched question to ask.
- kenjackson 2y agoThey fixed that. Now it replies: "Hi! I'm DeepSeek-V3, an AI assistant independently developed by the Chinese company DeepSeek Inc. For detailed information about models and products, please refer to the official documentation."
- whoistraitor 2y agoIt’s remarkable we’ve hit a threshold where so much can be done with synthetic data. The reasoning race seems an utterly solvable problem now (thanks mostly to the verifiability of results). I guess the challenge then becomes non-reasoning domains, where qualitative and truly creative results are desired.
- kenjackson 2y agoIt seems like we need an evaluation model for creativity. I'm curious, is there research on this -- for example, can one score a random painting and output how creative/good a given population is likely to find it?
- virgildotcodes 2y agoHow do you account for the impact of culture/lived experience of the specific population viewing the painting? Intuitively it seems like that would be the biggest factor, rather than the objective attributes of the painting, no?
- bentograd 2y agoAll art is subjective. Any attempt to "verify" a piece of art would be entirely dependent on cultural and personal sensitivities. Art isn't a math problem with a solution.
- baq 2y agoBut you can dissect it into concepts and see if it is something truly new to the model - if the output contains things which aren’t there in the weights, you have a nice specimen to study and, crucially, a recipe to get a bunch of matrices to output untrained things.
- cubefox 2y agoThis is like saying: All cooks are equally good, even the most disgusting slop (e.g. water/flour soup) isn't any better than a dish from a cook with several Michelin stars. Of course the latter is better. And if it is better, it is objectively better. Even if 0.001% of people prefer flour soup.
- raphaelj 2y agoDo we know which changes made DeepSeek V3 so much faster and better at training than other models? DeepSeek R1's performances seem to be highly related to V3 being a very good model to start with. I went through the paper and I understood they made these improvements compared to "regular" MoE models: 1. Latent Multi-head Attention. If I understand correctly, they were able to do some caching on the attention computation. This one is still a little bit confusing to me; 2. New MoE architecture with one shared expert and a large number of small routed experts (256 total, but 8 in use in the same token inference). This was already used in DeepSeek v2; 3. Better load balancing of the training of experts. During training, they add bias or "bonus" value to experts that are less used, to make them more likely to be selected in the future training steps; 4. They added a few smaller transformer layers to predict not only the first next token, but a few additional tokens. Their training error/loss function then uses all these predicted tokens as input, not only the first one. This is supposed to improve the transformer capabilities in predicting sequences of tokens; 5. They are using FP8 instead of FP16 when it does not impact accuracy. It's not clear to me which changes are the most important, but my guess would be that 4) is a critical improvement. 1), 2), 3) and 5) could explain why their model trains faster by some small factor (+/- 2x), but neither the 10x advertised boost nor how is performs greatly better than models with way more activated parameters (e.g. llama 3).
- alecco 2y agoThey also did bandwidth scaling to handle work around the nerfed H800 interconnects. > efficient cross-node all-to-all communication kernels to fully utilize IB and NVLink bandwidths > The key idea of DualPipe is to overlap the computation and communication within a pair of individual forward and backward chunks. To be specific, we divide each chunk into four components: attention, all-to-all dispatch, MLP, and all-to-all combine. Specially, for a backward chunk, both attention and MLP are further split into two parts, backward for input and backward for weights, like in ZeroBubble (Qi et al., 2023b). In addition, we have a PP communication component. (I know some of those words) https://arxiv.org/html/2412.19437v1 https://arxiv.org/html/2412.19437v1
- whoisburbansky 2y agoThe key idea of Latent MHA is that "regular" multi-headed attention needs you to keep a bunch of giant key-value (KV) matrices around in memory to do inference. The "Latent" part just means that DeepSeek takes the `n` KV matrices in a given n-headed attention block and replaces them with a lower-rank approximation (think of this as compressing the matrices), so that they take up less VRAM in a GPU at the cost of a little extra compute and a little lost accuracy. So not caching, strictly speaking, but weight compression to trade compute off for better memory usage, which is good because the KV matrices are one of the more expensive part of this transformer architecture. MoE addresses the other expensive part (the fully-connected layers) by making it so only a subset of the fully-connected layers are active at any given forward pass.
- deleted 2y ago[deleted]
- alecco 2y agoHow is this very high signal vs noise post out of the front page in 2hs? Are people so upset with the stock market crash that they are flagging it?
- jasonjmcghee 2y agoMaybe too much of the same topic? "How R1 was trained" also seemed to quickly fall off. But the big arxiv paper with 1000+ upvotes stuck around a while.
- htk 2y agoSpot on. I've read the very accessible paper and it's better than any of the how-to's written elsewhere. Nothing against good content being written, but the source material is already pretty good.
- khazhoux 2y agodang has provided an answer for how the algorithm works whenever I've asked similar question. But I still don't get it. 6 hours + 170 points and it's on third page. Meanwhile second page has "Null Byte on Steroids" at 12 hours + 20 points. ??
- youssefabdelm 2y agoThe "illustrated"... He needs to read up on Tufte or Bret Victor or something, these are just diagrams with text inside of boxes.
- 8n4vidtmkvmk 2y ago> This is a large number of long chain-of-thought reasoning examples (600,000 of them). These are very hard to come by and very expensive to label with humans at this scale. Which is why the process to create them is the second special thing to highlight I didn't know the reasonings were part of the training data. I thought we basically just told the LLM to "explain its thinking" or something as an intermediate step, but the fact that the 'thinking' is part of the training step makes more sense and I can see how this improves things in a non-trivial way. Still not sure if using word tokens as the intermediate "thinking" is the correct or optimal way of doing things, but I don't know. Maybe after everything is compressed into latent space it's essentially the same stuff.
- QuadrupleA 2y agoAm I the only one not that impressed with Deepseek R1? Its "thinking" seems full of the usual LLM blindsides, and ultimately generating more of it then summarizing doesn't seem to overcome any real limits. It's like making mortgage backed securities out of bad mortgages, you never really overcome the badness of the underlying loans, no matter how many layers you pile on top I haven't used or studied DeepSeek R1 (or o1) in exhaustive depth, but I guess I'm just not understanding the level of breathless hype right now.
- Giorgi 2y ago[flagged]
- itissid 2y agoIt is leaps and bounds better than LLMs. For one you are doing RL which is classic AI like tuning that optimizes a reward function with nice qualities- it's the same stuff used to train chess games and Go by showing them the actual moves. LLMs pre o1 and deepseek R1 were RHLF tuned which is like if you trained a LM how to play chess by showing people two boards and doing a vibe check on which "looks" better. Think of it this way say you were dropped in a maze that you had to solve but you could do only one of two things: 1. Look at two random moves from your start position and selected which one looked better to get out. 2. Made a series of moves and then backtracked, then use a quantitative function to exploit the best path. The latter is what R1 does and it chooses optimal and more certain path to success. Apply this to math and coding tokens and you have a competitive LLM.
- itissid 2y agoI am using the 32Gb distilled model on my local 3090 with Continue in VSCode. It beats everything out of the water.
- dontwearitout 2y agoHow many tokens/s do you get on a 3090? With the extra tokens for the internal monologue, is it still performant enough for smooth VSCode integration?
- terryjiao 2y ago[dead]
- distantsounds 2y agoWe all knew The Chinese government was going to censor it. The censoring happening in ChatGPT is arguably more interesting since they are not beholden to the US government. I'm more interested in that report.
- ForOldHack 2y ago"DeepSeek-R1 is the latest resounding beat in the steady drumroll of AI progress. " IBM's Intellect, from 1983 cost $47,000 dollars a month. Let me know when DeepSleep-Rx exceeds Windows (tm) version numbers or makes a jump like AutoCADs version numbers.