5 ms·

I believe you that it had to do with the selloff, but I believe that efficiency improvements are good news for NVIDIA: each card just got 20x more useful

amazingamazing 2y ago
each card is not 20x more useful lol. there's no evidence yet that the deepseek architecture would even yield a substantially (20x) more performant model with more compute. if there's evidence to the contrary I'd love to see. in any case I don't think a h800 is even 20x better than a h100 anyway, so the 20x increase has to be wrong.
- jdietrich 2y ago
  We need GPUs for inference, not just training. The Jevons Paradox suggests that reducing the cost per token will increase the overall demand for inference. Also, everything we know about LLMs points to an entirely predictable correlation between training compute and performance.
  - amazingamazing 2y ago
    the jevons paradox isn't about any particular product or company's product, so is irrelevant here. the relevant resource here is compute, which is already a commodity. secondly, even if it were about GPUs in particular, there's no evidence that nvidia would be able to sustain such high margins if fewer were necessary for equivalent performance. things are currently supply constrained, which gives nvidia price optionality.
    - Scoundreller 2y ago
      Uhhh, isn’t it about coal?
  - tshaddox 2y ago
    Jevons paradox doesn't really suggest anything by itself. Jevons paradox is something that occurs in some instances of increased efficiency, but not all. I suppose the important question here is "What is the price elasticity of demand of inference?"
    - mrbungie 2y ago
      People act like Jevons Paradox is an universal law thanks to Satya's tweet.
    - ckw 2y ago
      Personally, in the six months prior to the release of the deepseekv3 api, I'd made probably 100-200 api calls per month to llm services. In the past week I made 2.8 million api calls to dsv3.
      - shawabawa3 2y ago
        can i ask what kind of api calls you're making to dsv3? Crunching through huge amounts of unstructured data or something?
        ckw 2y ago
        Processing each english (word, part-of-speech, sense) triple in various ways. Generating (very silly) example sentences for each triple in various styles. Generating 'difficulty' ratings for each triple. Two examples: High difficulty: id = 37810 word = dendroid pos = noun sense = (mathematics) A connected continuum that is arcwise connected and hereditarily unicoherent. elo = 2408.61936886416 sentence2 = The dendroid, that arboreal structure of the Real, emerges not as a mere geometric curiosity but as the very topology of desire, its branches both infinite and indivisible, a map of the unconscious where every detour is already inscribed in the unicoherence of the subject's jouissance. Low difficulty: id = 11910 word = bed pos = noun sense = A flat, soft piece of furniture designed for resting or sleeping. elo = 447.32459484266 sentence2 = The city outside my window never closed its eyes, but I did, sinking into the cold embrace of a bed that smelled faintly of whiskey and regret.
- numba888 2y ago
  > there's no evidence yet that the deepseek architecture would even yield a substantially more performant model with more compute. It's supposed to. There was an info that the longer length of 'thinking' makes o3 model better than o1. I.e. at least at inference compute power still matters.
  - amazingamazing 2y ago
    > It's supposed to. There was an info that the longer length of 'thinking' makes o3 model better than o1. I.e. at least at inference compute power still matters. compute matters, but performance doesn't scale with compute from what I've heard about o3 vs o1. you shouldn't take my word for it - go on the leaderboards and look at the top models from now, and then the top models from 2023 and look at the compute involved for both. there's obviously a huge increase, but it isn't proportional
segasaturn 2y ago
That still means that that AI firms don't have to buy as many of Nvidia's chips, which is the whole thing that Nvidia's price was predicated on. FB, Google and Microsoft just had their their billions of dollars in Nvidia GPU capex blown out by $5M side-project. Tech firms are probably not going to be as generous shelling out whatever overinflated price Nvidia was asking for as they were a week ago.
- chgs 2y ago
  Imagine what you can do with all that Nvidia hardware using the deep mind techniques.
- flowerlad 2y ago
  > That still means that that AI firms don't have to buy as many of Nvidia's chips Couldn’t you say that about Blackwell as well? Blackwell is 25x more energy-efficient for generative AI tasks and offer up to 2.5x faster AI training performance overall.
  - Jlagreen 2y ago
    And yet, Blackwell is sold out. What does that tell us? The industry is compute starved and that makes totally sense. The tranformer model on which current LLMs are based on are 8 years old. But why took it so much time to get to the LLMs only 2 years ago? Simple, Nvidia first had to push the compute at scale strongly. Try training GPT4 on Voltas from 2017. Good luck with that! Current LLMs are possible thanks to the compute Nvidia has provided in the past decade. You could technically use 20 year old CPUs for LLMs but you might need to connect a billion of them.
- alfalfasprout 2y ago
  Wrong conclusion, IMO. This makes inference more cost effective which means self-hosting suddenly becomes more attractive to a wider share of the market. GPUs will continue to be bought up as fast as fabs can spit them out.
  - 2y ago
jcgrillo 2y ago
They only got more useful if the AI goldrush participants actually strike, well, gold. Otherwise it's not useful at all. Afaict it remains to be seen whether any of this AI stuff has actual commercial value. It's all just speculation predicated on thoughts and prayers.
sumeno 2y ago
When your business is selling a large number of cards to giant companies you don't want them to be 20x more useful because then people will buy fewer of them to do the same amount of work
- HDThoreaun 2y ago
  or people do 30x more work and buy 50% more cards

amazingamazing 2y ago

each card is not 20x more useful lol. there's no evidence yet that the deepseek architecture would even yield a substantially (20x) more performant model with more compute. if there's evidence to the contrary I'd love to see. in any case I don't think a h800 is even 20x better than a h100 anyway, so the 20x increase has to be wrong.

jdietrich 2y ago
We need GPUs for inference, not just training. The Jevons Paradox suggests that reducing the cost per token will increase the overall demand for inference. Also, everything we know about LLMs points to an entirely predictable correlation between training compute and performance.
- amazingamazing 2y ago
  the jevons paradox isn't about any particular product or company's product, so is irrelevant here. the relevant resource here is compute, which is already a commodity. secondly, even if it were about GPUs in particular, there's no evidence that nvidia would be able to sustain such high margins if fewer were necessary for equivalent performance. things are currently supply constrained, which gives nvidia price optionality.
  - Scoundreller 2y ago
    Uhhh, isn’t it about coal?
- tshaddox 2y ago
  Jevons paradox doesn't really suggest anything by itself. Jevons paradox is something that occurs in some instances of increased efficiency, but not all. I suppose the important question here is "What is the price elasticity of demand of inference?"
  - mrbungie 2y ago
    People act like Jevons Paradox is an universal law thanks to Satya's tweet.
  - ckw 2y ago
    Personally, in the six months prior to the release of the deepseekv3 api, I'd made probably 100-200 api calls per month to llm services. In the past week I made 2.8 million api calls to dsv3.
    - shawabawa3 2y ago
      can i ask what kind of api calls you're making to dsv3? Crunching through huge amounts of unstructured data or something?
      - ckw 2y ago
        Processing each english (word, part-of-speech, sense) triple in various ways. Generating (very silly) example sentences for each triple in various styles. Generating 'difficulty' ratings for each triple. Two examples: High difficulty: id = 37810 word = dendroid pos = noun sense = (mathematics) A connected continuum that is arcwise connected and hereditarily unicoherent. elo = 2408.61936886416 sentence2 = The dendroid, that arboreal structure of the Real, emerges not as a mere geometric curiosity but as the very topology of desire, its branches both infinite and indivisible, a map of the unconscious where every detour is already inscribed in the unicoherence of the subject's jouissance. Low difficulty: id = 11910 word = bed pos = noun sense = A flat, soft piece of furniture designed for resting or sleeping. elo = 447.32459484266 sentence2 = The city outside my window never closed its eyes, but I did, sinking into the cold embrace of a bed that smelled faintly of whiskey and regret.
numba888 2y ago
> there's no evidence yet that the deepseek architecture would even yield a substantially more performant model with more compute. It's supposed to. There was an info that the longer length of 'thinking' makes o3 model better than o1. I.e. at least at inference compute power still matters.
- amazingamazing 2y ago
  > It's supposed to. There was an info that the longer length of 'thinking' makes o3 model better than o1. I.e. at least at inference compute power still matters. compute matters, but performance doesn't scale with compute from what I've heard about o3 vs o1. you shouldn't take my word for it - go on the leaderboards and look at the top models from now, and then the top models from 2023 and look at the compute involved for both. there's obviously a huge increase, but it isn't proportional

segasaturn 2y ago

That still means that that AI firms don't have to buy as many of Nvidia's chips, which is the whole thing that Nvidia's price was predicated on. FB, Google and Microsoft just had their their billions of dollars in Nvidia GPU capex blown out by $5M side-project. Tech firms are probably not going to be as generous shelling out whatever overinflated price Nvidia was asking for as they were a week ago.

chgs 2y ago
Imagine what you can do with all that Nvidia hardware using the deep mind techniques.
flowerlad 2y ago
> That still means that that AI firms don't have to buy as many of Nvidia's chips Couldn’t you say that about Blackwell as well? Blackwell is 25x more energy-efficient for generative AI tasks and offer up to 2.5x faster AI training performance overall.
- Jlagreen 2y ago
  And yet, Blackwell is sold out. What does that tell us? The industry is compute starved and that makes totally sense. The tranformer model on which current LLMs are based on are 8 years old. But why took it so much time to get to the LLMs only 2 years ago? Simple, Nvidia first had to push the compute at scale strongly. Try training GPT4 on Voltas from 2017. Good luck with that! Current LLMs are possible thanks to the compute Nvidia has provided in the past decade. You could technically use 20 year old CPUs for LLMs but you might need to connect a billion of them.
alfalfasprout 2y ago
Wrong conclusion, IMO. This makes inference more cost effective which means self-hosting suddenly becomes more attractive to a wider share of the market. GPUs will continue to be bought up as fast as fabs can spit them out.
- 2y ago

jcgrillo 2y ago

They only got more useful if the AI goldrush participants actually strike, well, gold. Otherwise it's not useful at all. Afaict it remains to be seen whether any of this AI stuff has actual commercial value. It's all just speculation predicated on thoughts and prayers.

sumeno 2y ago

When your business is selling a large number of cards to giant companies you don't want them to be 20x more useful because then people will buy fewer of them to do the same amount of work

HDThoreaun 2y ago
or people do 30x more work and buy 50% more cards