6 ms·

Am I the only one not that impressed with Deepseek R1? Its "thinking" seems full of the usual LLM blindsides, and ultimately generating more of it then summariz

Am I the only one not that impressed with Deepseek R1? Its "thinking" seems full of the usual LLM blindsides, and ultimately generating more of it then summarizing doesn't seem to overcome any real limits. It's like making mortgage backed securities out of bad mortgages, you never really overcome the badness of the underlying loans, no matter how many layers you pile on top I haven't used or studied DeepSeek R1 (or o1) in exhaustive depth, but I guess I'm just not understanding the level of breathless hype right now.

Giorgi 2y ago
[flagged]
itissid 2y ago
It is leaps and bounds better than LLMs. For one you are doing RL which is classic AI like tuning that optimizes a reward function with nice qualities- it's the same stuff used to train chess games and Go by showing them the actual moves. LLMs pre o1 and deepseek R1 were RHLF tuned which is like if you trained a LM how to play chess by showing people two boards and doing a vibe check on which "looks" better. Think of it this way say you were dropped in a maze that you had to solve but you could do only one of two things: 1. Look at two random moves from your start position and selected which one looked better to get out. 2. Made a series of moves and then backtracked, then use a quantitative function to exploit the best path. The latter is what R1 does and it chooses optimal and more certain path to success. Apply this to math and coding tokens and you have a competitive LLM.
- itissid 2y ago
  I am using the 32Gb distilled model on my local 3090 with Continue in VSCode. It beats everything out of the water.
  - dontwearitout 2y ago
    How many tokens/s do you get on a 3090? With the extra tokens for the internal monologue, is it still performant enough for smooth VSCode integration?
  - MarcelOlsz 2y ago
    Any idea how to use a cloud hosted version with cursor?
throwup238 2y ago
You’re not the only one. It’s not as impressive at coding compared to O1 as people make it out to be and it’s explicitly spelled out in DeepSeek’s R1 paper that they had trouble with improving over DeepSeek-V3: > Software Engineering Tasks: Due to the long evaluation times, which impact the efficiency of the RL process, large-scale RL has not been applied extensively in software engineering tasks. As a result, DeepSeek-R1 has not demonstrated a huge improvement over DeepSeek-V3 on software engineering benchmarks. Future versions will address this by implementing rejection sampling on software engineering data or incorporating asynchronous evaluations during the RL process to improve efficiency. (last bullet point of page 16, which is the last page of the paper before the bibliography - hmm…) [1] It does even worse on my real world coding problems than the benchmarks would suggest. Some of my tests: write a Qt QAbstractListModel in C++ that parses markdown into block using a C/C++ MD parsing library, write Rust cxx-qt bindings for QTextDocument (all docs included in context), write a window switching script for Wayland with an alternative to wmctrl, etc. I also asked it some geochemistry questions that I had trialed O1 with previously, and the reasoning had a lot hallucinations. The answers were suboptimal to say the least. Having access to the thought process in <think></think> tags is cool but it just reveals how underdeveloped the actual reasoning is compared to whatever o1 is doing. I’ve had it get stuck on silly things like whether a C++ library for markdown exists because I underspecified that I’m okay with header only C libs. O1 has faired much better on my qualitative test suite - even more so when using it via API with a “high” reasoning_effort. With all of the hype over the last few days, it feels like I’m taking crazy pills (loving all the action though). [1] https://arxiv.org/abs/2501.12948 https://arxiv.org/abs/2501.12948
lm28469 2y ago
I've been using deepseek for a while, I never paid for chat gpt or any other services. The fact that r1 is now free and unlimited VS chat gpt 200$ a month subscription is impressive enough for me. If the development cost is anywhere close to what they advertise publicly it's even more impressive It's as good or better than chat gpt free, gemini free, &c. and that's all I care about
- redcobra762 2y ago
  Why is cost your only concern? You don’t care at all what data the model was trained on? The motivations of the people who trained it? I mean maybe the importance of those things isn’t super high to you, but “don’t care”? How odd. Most Westerners I know would care…
  - lm28469 2y ago
    Ah yes, because chat gpt is ethical AI lmao I'm European, seeing the clowns in charge of US tech companies I'm equally happy using Chinese tools, especially if they're open source. Stop twisting the debate as if it was "good US" vs "evil China", I have no horses in this race
    - redcobra762 2y ago
      That’s fine, it’s just not what Westerners generally say, in my experience. Not that distaste for Chinese authoritarianism is inherently pro Western, just that Westerners I know tend to have an ideological issue with the substantially higher levels of censorship and oppression that takes place in China. It’s surprising to find people who genuinely don’t care about any of that, is all.
      - lm28469 2y ago
        "westerners" as if it was a monolithic block. I envy the simplicity of seeing life in such terms, China bad, US good, it must be very relaxing You don't have any concerns with the US government, the nepotism, the corruption, the conflict of interests, the insider trading, the massive concentration of wealth and power in tech, the insane lobbyism, PRISM, &c. It's surprising to find people who can so clearly see how China is bad but are completely oblivious to their own problems. It's not a football game, you don't have to chose a side and be a boot licker for eternity
assimpleaspossi 2y ago
fwiw, National Public Radio (NPR) news in the USA said that AI experts stated it was "almost as good" as other current offerings like chatGPT and Gemini. That its real advantage is the low cost of providing the information. However, this is only a claim made by the company without any proof.
TrackerFF 2y ago
What's there not to understand? If it matches the latest GPT O-N model in performance - or is just close, even, at a fraction of the compute (50x less?) and it is free, then that's huge news. They just upended the current LLM/AI/ML dominance, or at least the perceived dominance. Billions and billions have been pumped into the race, where investors are betting on the winner - and here comes a Chinese hedge fund side-project on shoestring budget, matching those billion dollar behemoths. And they'll continue to release their work. They just made the OpenAI et. al. secret sauce a lot less valuable.
- infecto 2y ago
  In my tests it does not come close to O1-Pro. Still huge news but it does not quite make it.
  - rallyforthesun 2y ago
    The results i did get from deepseek-r1 on their webpage did not match the results i did get from o1-pro. I did ask it go to a github repo, find the part where the logic of the “export” button is and explain why it doesn’t work (the whole logic is actually missing, won’t work at all). O1 pro did get it right in the first try while deepseek r1 was heavily hallucinating. Maybe i am using the wrong model?
    - throwup238 2y ago
      No, you’re not. They explicitly mention in the R1 paper (in the last paragraph before the bibliography) that R1 isn’t a “huge” improvement over DeepSeek-V3 in coding - where “huge” is an academic weasel word. It’s just a lot of hype. In my coding tests it significantly underperforms o1 (haven’t tried o1-pro), often getting stuck in a reasoning loop because I underspecified something (that I don’t have to with o1).
      - infecto 2y ago
        Same anecdotal experience. Its definitely an improvement and they have made operational improvements at runtime but I am still concerned they are have over fit for the tests.
creer 2y ago
The brute force approach is the expensive one ("probably, so far anyway" add these 4 words everywhere) - and impossible to make "always correct" aka "usual blindsides". They seem to be trying a bunch of specialized training ideas here and there in the system - just like a Mixture of Experts does, in different places than was obvious so far, and with an eye toward reasonning. In particular trying to build a reasonning-oriented training base from minimal seed. It's still not going to give an "always correct" result but we are nowhere near the point where that's needed. We are only at the point where a new idea can get you a percentage point further in benchmarks. Some fundamental limits were baked into the previous assumptions - easy to get past by leaving these assumptions behind.
dkjaudyeqooe 2y ago
As with most early tech of a particular category, it's not the current capabilities that are the point but the direction of travel. DeepSeek has upended the conventional wisdom about model performance with respect to training, and it's a shock to the system. It demonstrates something that has become obvious: you don't need massive scale or funding to innovate and have impact, you just need good ideas.
- danielscrubs 2y ago
  I guess this is why Googles CEO said that they have no LLM moat, as cleverness is not a moat.
HarHarVeryFunny 2y ago
It's the cost comparison with O1, both to train and run (per their pricing), that is causing most of the shock, perhaps as well as the fact that it's a GPU poor Chinese company that has caught up with O1, not a US one (Anthropic, Google, Meta, X.ai, Microsoft). The fact that it's open weights and training is fairly detailed in the paper they released is also significant. The best comparison for R1 is O1, but given different training data, hard to compare outside of benchmarks. At the moment these "reasoning models" are not necessarily the best thing to use for non-reasoning tasks, but Anthropic have recently indicated that they expect to release models that are more universal.
vineyardmike 2y ago
I get the hype, even if I don’t necessarily agree with it. TLDR: it’s technically impressive in training infrastructure and a geopolitical surprise. As others commenters said, it compares favorably against American Flagship models in benchmarks. This is geopolitically interesting since the model is Chinese, and subject to trade restrictions and America thinks of itself as the world’s best model builders. What makes it interesting technically, is the trade restrictions required them to focus on efficiency and reusing old hardware. This made it wildly cheaper to run the training. The They make a ton of very low-level optimizations to make training efficient. This is impressive engineering, and shows how a small amount of effort can free up a nvidia-lock-in. They had to totally bypass a lot of CUDA libraries and drop down even lower to control the GPUs. Nvidia has been capturing a huge fraction of industry wide AI spend, and surely no one wants that. This is, IMO, the actual part to watch. I suspect it’ll drive a new wave of efficiency-focused LLM tools which will also unlock competitors GPUs. They also had some novel-for-LLMS training techniques but it’s suspected that the big AI companies elsewhere are doing it now too, but not disclosed. (Mostly reinforcement learning). What I think is hype, meanwhile, is the actual benchmarks. Most models are trained on their competitors output. This is just a reality of the industry. Especially non-flagships being trained against flagships data. DeepSeek was almost certainly trained against OpenAI models, so it makes sense that it would approach the output quality. That’s very different from being capable of outperforming or “taking the lead” or whatever. We’ll need to wait longer and see how the future goes to make that determination. “China” has long had a great history of machine learning tech, so there is no reason to think that it’s structurally impossible for a Chinese organization to be on the leading edge, but it has to happen before we can say it happened. What is also hype is calling this a “side project” of a financial firm. The firm spun it out as a dedicated company. China cracked down on hedge funds, so the company looked for ways to repurpose the talent and compute infrastructure. This isn’t some side project during lunch breaks for a few bored quants. PS, thinking models are very different in use-case than normal models. It’s great for tasks with a “right answer” like math problems. Consider a simple example, in calculus, your teacher made you “show your work”, and doing so certainly reduced the likelihood of errors. That’s the purpose of a thinking model, and it excels at similar tasks.

Giorgi 2y ago

[flagged]

itissid 2y ago

It is leaps and bounds better than LLMs. For one you are doing RL which is classic AI like tuning that optimizes a reward function with nice qualities- it's the same stuff used to train chess games and Go by showing them the actual moves. LLMs pre o1 and deepseek R1 were RHLF tuned which is like if you trained a LM how to play chess by showing people two boards and doing a vibe check on which "looks" better. Think of it this way say you were dropped in a maze that you had to solve but you could do only one of two things: 1. Look at two random moves from your start position and selected which one looked better to get out. 2. Made a series of moves and then backtracked, then use a quantitative function to exploit the best path. The latter is what R1 does and it chooses optimal and more certain path to success. Apply this to math and coding tokens and you have a competitive LLM.

itissid 2y ago
I am using the 32Gb distilled model on my local 3090 with Continue in VSCode. It beats everything out of the water.
- dontwearitout 2y ago
  How many tokens/s do you get on a 3090? With the extra tokens for the internal monologue, is it still performant enough for smooth VSCode integration?
- MarcelOlsz 2y ago
  Any idea how to use a cloud hosted version with cursor?

throwup238 2y ago

You’re not the only one. It’s not as impressive at coding compared to O1 as people make it out to be and it’s explicitly spelled out in DeepSeek’s R1 paper that they had trouble with improving over DeepSeek-V3: > Software Engineering Tasks: Due to the long evaluation times, which impact the efficiency of the RL process, large-scale RL has not been applied extensively in software engineering tasks. As a result, DeepSeek-R1 has not demonstrated a huge improvement over DeepSeek-V3 on software engineering benchmarks. Future versions will address this by implementing rejection sampling on software engineering data or incorporating asynchronous evaluations during the RL process to improve efficiency. (last bullet point of page 16, which is the last page of the paper before the bibliography - hmm…) [1] It does even worse on my real world coding problems than the benchmarks would suggest. Some of my tests: write a Qt QAbstractListModel in C++ that parses markdown into block using a C/C++ MD parsing library, write Rust cxx-qt bindings for QTextDocument (all docs included in context), write a window switching script for Wayland with an alternative to wmctrl, etc. I also asked it some geochemistry questions that I had trialed O1 with previously, and the reasoning had a lot hallucinations. The answers were suboptimal to say the least. Having access to the thought process in <think></think> tags is cool but it just reveals how underdeveloped the actual reasoning is compared to whatever o1 is doing. I’ve had it get stuck on silly things like whether a C++ library for markdown exists because I underspecified that I’m okay with header only C libs. O1 has faired much better on my qualitative test suite - even more so when using it via API with a “high” reasoning_effort. With all of the hype over the last few days, it feels like I’m taking crazy pills (loving all the action though). [1] https://arxiv.org/abs/2501.12948 https://arxiv.org/abs/2501.12948

lm28469 2y ago

I've been using deepseek for a while, I never paid for chat gpt or any other services. The fact that r1 is now free and unlimited VS chat gpt 200$ a month subscription is impressive enough for me. If the development cost is anywhere close to what they advertise publicly it's even more impressive It's as good or better than chat gpt free, gemini free, &c. and that's all I care about

redcobra762 2y ago
Why is cost your only concern? You don’t care at all what data the model was trained on? The motivations of the people who trained it? I mean maybe the importance of those things isn’t super high to you, but “don’t care”? How odd. Most Westerners I know would care…
- lm28469 2y ago
  Ah yes, because chat gpt is ethical AI lmao I'm European, seeing the clowns in charge of US tech companies I'm equally happy using Chinese tools, especially if they're open source. Stop twisting the debate as if it was "good US" vs "evil China", I have no horses in this race
  - redcobra762 2y ago
    That’s fine, it’s just not what Westerners generally say, in my experience. Not that distaste for Chinese authoritarianism is inherently pro Western, just that Westerners I know tend to have an ideological issue with the substantially higher levels of censorship and oppression that takes place in China. It’s surprising to find people who genuinely don’t care about any of that, is all.
    - lm28469 2y ago
      "westerners" as if it was a monolithic block. I envy the simplicity of seeing life in such terms, China bad, US good, it must be very relaxing You don't have any concerns with the US government, the nepotism, the corruption, the conflict of interests, the insider trading, the massive concentration of wealth and power in tech, the insane lobbyism, PRISM, &c. It's surprising to find people who can so clearly see how China is bad but are completely oblivious to their own problems. It's not a football game, you don't have to chose a side and be a boot licker for eternity

assimpleaspossi 2y ago

fwiw, National Public Radio (NPR) news in the USA said that AI experts stated it was "almost as good" as other current offerings like chatGPT and Gemini. That its real advantage is the low cost of providing the information. However, this is only a claim made by the company without any proof.

TrackerFF 2y ago

What's there not to understand? If it matches the latest GPT O-N model in performance - or is just close, even, at a fraction of the compute (50x less?) and it is free, then that's huge news. They just upended the current LLM/AI/ML dominance, or at least the perceived dominance. Billions and billions have been pumped into the race, where investors are betting on the winner - and here comes a Chinese hedge fund side-project on shoestring budget, matching those billion dollar behemoths. And they'll continue to release their work. They just made the OpenAI et. al. secret sauce a lot less valuable.

infecto 2y ago
In my tests it does not come close to O1-Pro. Still huge news but it does not quite make it.
- rallyforthesun 2y ago
  The results i did get from deepseek-r1 on their webpage did not match the results i did get from o1-pro. I did ask it go to a github repo, find the part where the logic of the “export” button is and explain why it doesn’t work (the whole logic is actually missing, won’t work at all). O1 pro did get it right in the first try while deepseek r1 was heavily hallucinating. Maybe i am using the wrong model?
  - throwup238 2y ago
    No, you’re not. They explicitly mention in the R1 paper (in the last paragraph before the bibliography) that R1 isn’t a “huge” improvement over DeepSeek-V3 in coding - where “huge” is an academic weasel word. It’s just a lot of hype. In my coding tests it significantly underperforms o1 (haven’t tried o1-pro), often getting stuck in a reasoning loop because I underspecified something (that I don’t have to with o1).
    - infecto 2y ago
      Same anecdotal experience. Its definitely an improvement and they have made operational improvements at runtime but I am still concerned they are have over fit for the tests.

creer 2y ago

The brute force approach is the expensive one ("probably, so far anyway" add these 4 words everywhere) - and impossible to make "always correct" aka "usual blindsides". They seem to be trying a bunch of specialized training ideas here and there in the system - just like a Mixture of Experts does, in different places than was obvious so far, and with an eye toward reasonning. In particular trying to build a reasonning-oriented training base from minimal seed. It's still not going to give an "always correct" result but we are nowhere near the point where that's needed. We are only at the point where a new idea can get you a percentage point further in benchmarks. Some fundamental limits were baked into the previous assumptions - easy to get past by leaving these assumptions behind.

dkjaudyeqooe 2y ago

As with most early tech of a particular category, it's not the current capabilities that are the point but the direction of travel. DeepSeek has upended the conventional wisdom about model performance with respect to training, and it's a shock to the system. It demonstrates something that has become obvious: you don't need massive scale or funding to innovate and have impact, you just need good ideas.

danielscrubs 2y ago
I guess this is why Googles CEO said that they have no LLM moat, as cleverness is not a moat.

HarHarVeryFunny 2y ago

It's the cost comparison with O1, both to train and run (per their pricing), that is causing most of the shock, perhaps as well as the fact that it's a GPU poor Chinese company that has caught up with O1, not a US one (Anthropic, Google, Meta, X.ai, Microsoft). The fact that it's open weights and training is fairly detailed in the paper they released is also significant. The best comparison for R1 is O1, but given different training data, hard to compare outside of benchmarks. At the moment these "reasoning models" are not necessarily the best thing to use for non-reasoning tasks, but Anthropic have recently indicated that they expect to release models that are more universal.

vineyardmike 2y ago

I get the hype, even if I don’t necessarily agree with it. TLDR: it’s technically impressive in training infrastructure and a geopolitical surprise. As others commenters said, it compares favorably against American Flagship models in benchmarks. This is geopolitically interesting since the model is Chinese, and subject to trade restrictions and America thinks of itself as the world’s best model builders. What makes it interesting technically, is the trade restrictions required them to focus on efficiency and reusing old hardware. This made it wildly cheaper to run the training. The They make a ton of very low-level optimizations to make training efficient. This is impressive engineering, and shows how a small amount of effort can free up a nvidia-lock-in. They had to totally bypass a lot of CUDA libraries and drop down even lower to control the GPUs. Nvidia has been capturing a huge fraction of industry wide AI spend, and surely no one wants that. This is, IMO, the actual part to watch. I suspect it’ll drive a new wave of efficiency-focused LLM tools which will also unlock competitors GPUs. They also had some novel-for-LLMS training techniques but it’s suspected that the big AI companies elsewhere are doing it now too, but not disclosed. (Mostly reinforcement learning). What I think is hype, meanwhile, is the actual benchmarks. Most models are trained on their competitors output. This is just a reality of the industry. Especially non-flagships being trained against flagships data. DeepSeek was almost certainly trained against OpenAI models, so it makes sense that it would approach the output quality. That’s very different from being capable of outperforming or “taking the lead” or whatever. We’ll need to wait longer and see how the future goes to make that determination. “China” has long had a great history of machine learning tech, so there is no reason to think that it’s structurally impossible for a Chinese organization to be on the leading edge, but it has to happen before we can say it happened. What is also hype is calling this a “side project” of a financial firm. The firm spun it out as a dedicated company. China cracked down on hedge funds, so the company looked for ways to repurpose the talent and compute infrastructure. This isn’t some side project during lunch breaks for a few bored quants. PS, thinking models are very different in use-case than normal models. It’s great for tasks with a “right answer” like math problems. Consider a simple example, in calculus, your teacher made you “show your work”, and doing so certainly reduced the likelihood of errors. That’s the purpose of a thinking model, and it excels at similar tasks.