7 ms·

Here’s a take I haven’t seen yet: If training and inference just got 40x more efficient, but OpenAI and co. still have the same compute resources, once they’ve

Here’s a take I haven’t seen yet: If training and inference just got 40x more efficient, but OpenAI and co. still have the same compute resources, once they’ve baked in all the DeepSeek improvements, we’re about to find out very quickly whether 40x the compute delivers 40x the performance / output quality, or if output quality has ceased to be compute-bound.

BenFranklin100 2y ago
That seems to be the key question.
deleted 2y ago
[deleted]
xbmcuser 2y ago
Yeah this was my first thought as well. If it got so efficient how good all the models will be 2-3 months from now
zeven7 2y ago
I've missed the stories on this until now. Is it known (and is there an ELI5) how they were able to do it so much more efficiently?
- riffraff 2y ago
  IIUC they released a paper, it's partially algorithmic improvements partially good old low level optimization.
  - CursedSilicon 2y ago
    There's something to be said about the idea that instead of just dumping oceans of money into buying Nvidia cards they just...optimized what they had I'd say the wider industry could learn a thing or two, but as other commentors have joked. The line must go up
    - Neonlicht 2y ago
      Ah but why care about efficiency when you have basically unlimited investor money? Reminds me of Japanese cars and the OPEC boycott in the 1970s...
    - plussed_reader 2y ago
      The double edged sword of export controls.
      - mnky9800n 2y ago
        Creativity always delivers when minds are constrained.
        plussed_reader 2y ago
        Is that what we can call Project2025? Creativity from constrained minds?
        mnky9800n 2y ago
        surely more relevant comments could exist in the aether of the internet.
        plussed_reader 2y ago
        Your reply doesn't read like a refute to me, mnky9800n.
- rx_tx 2y ago
  This article has good background, context, and explanations [1] They skipped CUDA and instead used PTX which is a lower level instruction set where they were able to implement more performant cross-chip comms to make up for the less-performant H800 chips. [1]: https://stratechery.com/2025/deepseek-faq/ https://stratechery.com/2025/deepseek-faq/
  - yread 2y ago
    So can people do the same in SPIR for OpenCL or amdgcn? https://en.wikipedia.org/wiki/Standard_Portable_Intermediate_Representation https://en.wikipedia.org/wiki/Standard_Portable_Intermediate... https://www.khronos.org/spir/ https://www.khronos.org/spir/ Or even better in the unified language like SYCL? https://cdrdv2-public.intel.com/786536/Heidelberg_IWOCL__SYCLCon_2023_paper_2566-1.pdf https://cdrdv2-public.intel.com/786536/Heidelberg_IWOCL__SYC...
  - saagarjha 2y ago
    > Moreover, if you actually did the math on the previous question, you would realize that DeepSeek actually had an excess of computing; that’s because DeepSeek actually programmed 20 of the 132 processing units on each H800 specifically to manage cross-chip communications. This is actually impossible to do in CUDA. You can do this just fine in CUDA, no PTX required. Of course all the major shops are using inline PTX at the very least to access the Tensor cores effectively.
frontalier 2y ago
there's this and that little desktop computer they announced earlier this month - digits they claim it's able to run models with 200B parameters on a single node and 400B when paired with another node
gloflo 2y ago
Does line go up forever?
- gdiamos 2y ago
  It goes up at least until LLMs match humans - ie until an LLM can write Windows
  - winwang 2y ago
    Slightly tangential, but I want an LLM which can debug windows.
    - cyanydeez 2y ago
      Debug And locally fix security holes.
      - Muromec 2y ago
        Or sell the found exploit to the friendly nation state actor to pay for its compute behind your back
  - z3phyr 2y ago
    I want the LLM to decide not to do anything, or write a new OS. Whenever I prompt: "Do not do anything" It always does <something>.
    - busyant 2y ago
      > Whenever I prompt: "Do not do anything" It always does <something>. Yep. A lot of times, the responses I get remind me of Simone in Ferris Bueller's Day Off: https://www.youtube.com/watch?v=swBtLPWeKbU https://www.youtube.com/watch?v=swBtLPWeKbU If you end up making a new model, please teach it that less is more and call it "LAIconic".
    - segmondy 2y ago
      Do not think of a pink elephant. Were you able to do so?
AnthonyMouse 2y ago
> If training and inference just got 40x more efficient Did training and inference just get 40x more efficient, or just training? They trained a model with impressive outputs on a limited number of GPUs, but DeepSeek is still a big model that requires a lot of resources to run. Moreover, which costs more, training a model once or using it for inference across a hundred million people multiple times a day for a year? It was always the second one, and doing the training cheaper makes it even more so. But this implies that we could use those same resources to train even bigger models, right? Except that you then have the same problem. You have a bigger model, maybe it's better, but if you've made inference cost linearly more because of the size and the size is now 40x bigger, you now need that much more compute for inference.
- cyberax 2y ago
  > DeepSeek is still a big model that requires a lot of resources to run I can run the largest model at 4 tokens per second on a 64GB card. Smaller models are _faster_ than Phi-4. I've just switched to it for my local inference.
  - KeplerBoy 2y ago
    Isn't the largest model still like 130GB after heavy quantization[1] and 4 tok/s borderline unusable for interactive sessions with those long outputs? [1] https://unsloth.ai/blog/deepseekr1-dynamic https://unsloth.ai/blog/deepseekr1-dynamic
    - thot_experiment 2y ago
      OP probably means "the largest distilled model"
      - menaerus 2y ago
        So not an actual DeepSeek-R1 model but a distilled Qwen or Llama model. From DeepSeek-R1 paper: > As shown in Table 5, simply distilling DeepSeek-R1’s outputs enables the efficient DeepSeekR1-7B (i.e., DeepSeek-R1-Distill-Qwen-7B, abbreviated similarly below) to outperform nonreasoning models like GPT-4o-0513 across the board. and > DeepSeek-R1-14B surpasses QwQ-32BPreview on all evaluation metrics, while DeepSeek-R1-32B and DeepSeek-R1-70B significantly exceed o1-mini on most benchmarks. and > These [Distilled Model Evaluation] results demonstrate the strong potential of distillation. Additionally, we found that applying RL to these distilled models yields significant further gains. We believe this warrants further exploration and therefore present only the results of the simple SFT-distilled models here.
nejsjsjsbsb 2y ago
That's a take I've seen in many HN comments
DebtDeflation 2y ago
In the long run (which in the AI world is probably ~1 year) this is very good for Nvidia, very good for the hyperscalers, and very good for anyone building AI applications. The only thing it's not good for is the idea that OpenAI and/or Anthropic will eventually become profitable companies with market caps that exceed Apple's by orders of magnitude. Oh no, anyway.
- infecto 2y ago
  Yes! I have had the exact same mental model. The biggest losers in this news are the groups building frontier models. They are the ones with huge valuations but if the optimizations becomes even close to true, its a massive threat to their business model. My feet are on the ground but I do still believe that the world does not comprehend how much compute it can use...as compute gets cheaper we will use more of it. Ignoring equity pricing, this benefits all other parties.
  - sigmoid10 2y ago
    My big current conspiracy theory is that this negative sentiment toward Nvidia from Deepseek's release is spread by people who actually want to buy more stock at a cheaper price. Like, if you know anything about the topic, it's wild to assume that this will drive demand for GPUs anywhere but up. If Nvidia came out with a Jetson like product that can run the full 670B R1, they could make infinite money. And in the datacenter section, companies will stumble over each other to get the necessary hardware (which corresponds to a dozen H100s or so right now). Especially once HF comes out with their uncensored reproduction. There's so much opportunity to turn more compute into more money because of this, almost every company could theoretically benefit.
- matco11 2y ago
  Can you guys explain what this would be bad for the OpenAI and Anthropic of the world? Wasn't the story always outlined to be we build better and better models, then we eventually get to AGI, AGI works on building better and better models even faster, and we eventually get to super AGI, which can work on building better and better models even faster... Isn't "super-optimization"(in the widest sense) what we expect to happen in the long run?
paul_e_warner 2y ago
Yes, but I think most of the rout is caused by the fact that there really isn't anything protecting AI from being disrupted by a new player - They're fairly simple technology compared to some of the other things tech companies build. That means openai really doesn't have much ability to protect it's market leader status. I don't really understand why the stock market has decided this affects nvidia's stock price though.
qwe----3 2y ago
Like everything, I expect improvement to be logarithmic
teleforce 2y ago
>If training and inference just got 40x more efficient The jury is still out on how much improvement DeepSeek made in terms of training and inference compute efficiency, but personally I think 10x is probably the actual improvement that's being made But in business/engineering/manufacturing/etc if you have 10x more efficiency, you're basically going to obliterate the competitions. >output quality has ceased to be compute-bound You raised an interesting conjecture and it seems that it's very likely the case. I know that it's not even a full two years that ChatGPT-4 has been released but it seems that it take OpenAI a very long time to release ChatGPT-5. Is it because they're taking their own sweet time to release the software not unlike GIMP, or they genuinely cannot justify the improvement to jump from 4 to 5? This stagnation however, has allowed others to catch up. Now based on DeekSeek claims, anyone can has their own ChatGPT-4 under their desk with Nvidia project Digits mini PCs [1]. For running DeepSeek, 4 units mini PCs will be more than enough of 4 PFLOPS and cost only USD12K. Let's say on average one subscriber user pays OpenAI monthly payment of USD$10, for 1000 persons organization it will be USD$10K, and the investment will pays for itself within a month, and no data ever leave the organization since it's a private cloud! For training similar system to ChatGPT-4 based on DeepSeeks claims, a few millions USD$ is more than enough. Apparently, OpenAI, Softbank and Oracle just announced USD$500 Billions joint ventures to bring the AI forward with the new announced Stargate AI project but that's 10,000x money [2],[3]. But the elephant in the room question is that, can they even get 10x quality improvement of the existing ChatGPT-4? I really seriously doubt it. [1] NVIDIA Puts Grace Blackwell on Every Desk and at Every AI Developer’s Fingertips: https://nvidianews.nvidia.com/news/nvidia-puts-grace-blackwell-on-every-desk-and-at-every-ai-developers-fingertips https://nvidianews.nvidia.com/news/nvidia-puts-grace-blackwe... [2] Trump unveils $500bn Stargate AI project between OpenAI, Oracle and SoftBank: https://www.theguardian.com/us-news/2025/jan/21/trump-ai-joint-venture-openai-oracle-softbank https://www.theguardian.com/us-news/2025/jan/21/trump-ai-joi... [3] Announcing The Stargate Project: https://openai.com/index/announcing-the-stargate-project/ https://openai.com/index/announcing-the-stargate-project/

BenFranklin100 2y ago

That seems to be the key question.

deleted 2y ago

[deleted]

xbmcuser 2y ago

Yeah this was my first thought as well. If it got so efficient how good all the models will be 2-3 months from now

zeven7 2y ago

I've missed the stories on this until now. Is it known (and is there an ELI5) how they were able to do it so much more efficiently?

riffraff 2y ago
IIUC they released a paper, it's partially algorithmic improvements partially good old low level optimization.
- CursedSilicon 2y ago
  There's something to be said about the idea that instead of just dumping oceans of money into buying Nvidia cards they just...optimized what they had I'd say the wider industry could learn a thing or two, but as other commentors have joked. The line must go up
  - Neonlicht 2y ago
    Ah but why care about efficiency when you have basically unlimited investor money? Reminds me of Japanese cars and the OPEC boycott in the 1970s...
  - plussed_reader 2y ago
    The double edged sword of export controls.
    - mnky9800n 2y ago
      Creativity always delivers when minds are constrained.
      - plussed_reader 2y ago
        Is that what we can call Project2025? Creativity from constrained minds?
        mnky9800n 2y ago
        surely more relevant comments could exist in the aether of the internet.
        plussed_reader 2y ago
        Your reply doesn't read like a refute to me, mnky9800n.
rx_tx 2y ago
This article has good background, context, and explanations [1] They skipped CUDA and instead used PTX which is a lower level instruction set where they were able to implement more performant cross-chip comms to make up for the less-performant H800 chips. [1]: https://stratechery.com/2025/deepseek-faq/ https://stratechery.com/2025/deepseek-faq/
- yread 2y ago
  So can people do the same in SPIR for OpenCL or amdgcn? https://en.wikipedia.org/wiki/Standard_Portable_Intermediate_Representation https://en.wikipedia.org/wiki/Standard_Portable_Intermediate... https://www.khronos.org/spir/ https://www.khronos.org/spir/ Or even better in the unified language like SYCL? https://cdrdv2-public.intel.com/786536/Heidelberg_IWOCL__SYCLCon_2023_paper_2566-1.pdf https://cdrdv2-public.intel.com/786536/Heidelberg_IWOCL__SYC...
- saagarjha 2y ago
  > Moreover, if you actually did the math on the previous question, you would realize that DeepSeek actually had an excess of computing; that’s because DeepSeek actually programmed 20 of the 132 processing units on each H800 specifically to manage cross-chip communications. This is actually impossible to do in CUDA. You can do this just fine in CUDA, no PTX required. Of course all the major shops are using inline PTX at the very least to access the Tensor cores effectively.

frontalier 2y ago

there's this and that little desktop computer they announced earlier this month - digits they claim it's able to run models with 200B parameters on a single node and 400B when paired with another node

gloflo 2y ago

Does line go up forever?

gdiamos 2y ago
It goes up at least until LLMs match humans - ie until an LLM can write Windows
- winwang 2y ago
  Slightly tangential, but I want an LLM which can debug windows.
  - cyanydeez 2y ago
    Debug And locally fix security holes.
    - Muromec 2y ago
      Or sell the found exploit to the friendly nation state actor to pay for its compute behind your back
- z3phyr 2y ago
  I want the LLM to decide not to do anything, or write a new OS. Whenever I prompt: "Do not do anything" It always does <something>.
  - busyant 2y ago
    > Whenever I prompt: "Do not do anything" It always does <something>. Yep. A lot of times, the responses I get remind me of Simone in Ferris Bueller's Day Off: https://www.youtube.com/watch?v=swBtLPWeKbU https://www.youtube.com/watch?v=swBtLPWeKbU If you end up making a new model, please teach it that less is more and call it "LAIconic".
  - segmondy 2y ago
    Do not think of a pink elephant. Were you able to do so?

AnthonyMouse 2y ago

> If training and inference just got 40x more efficient Did training and inference just get 40x more efficient, or just training? They trained a model with impressive outputs on a limited number of GPUs, but DeepSeek is still a big model that requires a lot of resources to run. Moreover, which costs more, training a model once or using it for inference across a hundred million people multiple times a day for a year? It was always the second one, and doing the training cheaper makes it even more so. But this implies that we could use those same resources to train even bigger models, right? Except that you then have the same problem. You have a bigger model, maybe it's better, but if you've made inference cost linearly more because of the size and the size is now 40x bigger, you now need that much more compute for inference.

cyberax 2y ago
> DeepSeek is still a big model that requires a lot of resources to run I can run the largest model at 4 tokens per second on a 64GB card. Smaller models are _faster_ than Phi-4. I've just switched to it for my local inference.
- KeplerBoy 2y ago
  Isn't the largest model still like 130GB after heavy quantization[1] and 4 tok/s borderline unusable for interactive sessions with those long outputs? [1] https://unsloth.ai/blog/deepseekr1-dynamic https://unsloth.ai/blog/deepseekr1-dynamic
  - thot_experiment 2y ago
    OP probably means "the largest distilled model"
    - menaerus 2y ago
      So not an actual DeepSeek-R1 model but a distilled Qwen or Llama model. From DeepSeek-R1 paper: > As shown in Table 5, simply distilling DeepSeek-R1’s outputs enables the efficient DeepSeekR1-7B (i.e., DeepSeek-R1-Distill-Qwen-7B, abbreviated similarly below) to outperform nonreasoning models like GPT-4o-0513 across the board. and > DeepSeek-R1-14B surpasses QwQ-32BPreview on all evaluation metrics, while DeepSeek-R1-32B and DeepSeek-R1-70B significantly exceed o1-mini on most benchmarks. and > These [Distilled Model Evaluation] results demonstrate the strong potential of distillation. Additionally, we found that applying RL to these distilled models yields significant further gains. We believe this warrants further exploration and therefore present only the results of the simple SFT-distilled models here.

nejsjsjsbsb 2y ago

That's a take I've seen in many HN comments

DebtDeflation 2y ago

In the long run (which in the AI world is probably ~1 year) this is very good for Nvidia, very good for the hyperscalers, and very good for anyone building AI applications. The only thing it's not good for is the idea that OpenAI and/or Anthropic will eventually become profitable companies with market caps that exceed Apple's by orders of magnitude. Oh no, anyway.

infecto 2y ago
Yes! I have had the exact same mental model. The biggest losers in this news are the groups building frontier models. They are the ones with huge valuations but if the optimizations becomes even close to true, its a massive threat to their business model. My feet are on the ground but I do still believe that the world does not comprehend how much compute it can use...as compute gets cheaper we will use more of it. Ignoring equity pricing, this benefits all other parties.
- sigmoid10 2y ago
  My big current conspiracy theory is that this negative sentiment toward Nvidia from Deepseek's release is spread by people who actually want to buy more stock at a cheaper price. Like, if you know anything about the topic, it's wild to assume that this will drive demand for GPUs anywhere but up. If Nvidia came out with a Jetson like product that can run the full 670B R1, they could make infinite money. And in the datacenter section, companies will stumble over each other to get the necessary hardware (which corresponds to a dozen H100s or so right now). Especially once HF comes out with their uncensored reproduction. There's so much opportunity to turn more compute into more money because of this, almost every company could theoretically benefit.
matco11 2y ago
Can you guys explain what this would be bad for the OpenAI and Anthropic of the world? Wasn't the story always outlined to be we build better and better models, then we eventually get to AGI, AGI works on building better and better models even faster, and we eventually get to super AGI, which can work on building better and better models even faster... Isn't "super-optimization"(in the widest sense) what we expect to happen in the long run?

paul_e_warner 2y ago

Yes, but I think most of the rout is caused by the fact that there really isn't anything protecting AI from being disrupted by a new player - They're fairly simple technology compared to some of the other things tech companies build. That means openai really doesn't have much ability to protect it's market leader status. I don't really understand why the stock market has decided this affects nvidia's stock price though.

qwe----3 2y ago

Like everything, I expect improvement to be logarithmic

teleforce 2y ago

>If training and inference just got 40x more efficient The jury is still out on how much improvement DeepSeek made in terms of training and inference compute efficiency, but personally I think 10x is probably the actual improvement that's being made But in business/engineering/manufacturing/etc if you have 10x more efficiency, you're basically going to obliterate the competitions. >output quality has ceased to be compute-bound You raised an interesting conjecture and it seems that it's very likely the case. I know that it's not even a full two years that ChatGPT-4 has been released but it seems that it take OpenAI a very long time to release ChatGPT-5. Is it because they're taking their own sweet time to release the software not unlike GIMP, or they genuinely cannot justify the improvement to jump from 4 to 5? This stagnation however, has allowed others to catch up. Now based on DeekSeek claims, anyone can has their own ChatGPT-4 under their desk with Nvidia project Digits mini PCs [1]. For running DeepSeek, 4 units mini PCs will be more than enough of 4 PFLOPS and cost only USD12K. Let's say on average one subscriber user pays OpenAI monthly payment of USD$10, for 1000 persons organization it will be USD$10K, and the investment will pays for itself within a month, and no data ever leave the organization since it's a private cloud! For training similar system to ChatGPT-4 based on DeepSeeks claims, a few millions USD$ is more than enough. Apparently, OpenAI, Softbank and Oracle just announced USD$500 Billions joint ventures to bring the AI forward with the new announced Stargate AI project but that's 10,000x money [2],[3]. But the elephant in the room question is that, can they even get 10x quality improvement of the existing ChatGPT-4? I really seriously doubt it. [1] NVIDIA Puts Grace Blackwell on Every Desk and at Every AI Developer’s Fingertips: https://nvidianews.nvidia.com/news/nvidia-puts-grace-blackwell-on-every-desk-and-at-every-ai-developers-fingertips https://nvidianews.nvidia.com/news/nvidia-puts-grace-blackwe... [2] Trump unveils $500bn Stargate AI project between OpenAI, Oracle and SoftBank: https://www.theguardian.com/us-news/2025/jan/21/trump-ai-joint-venture-openai-oracle-softbank https://www.theguardian.com/us-news/2025/jan/21/trump-ai-joi... [3] Announcing The Stargate Project: https://openai.com/index/announcing-the-stargate-project/ https://openai.com/index/announcing-the-stargate-project/