7 ms·
Not quite, I believe this sell off was caused by DeepSeek showing with their new model that the hardware demands of AI are not necessarily as high as everyone h
by KiwiJohnno 2y ago
Not quite, I believe this sell off was caused by DeepSeek showing with their new model that the hardware demands of AI are not necessarily as high as everyone has assumed (as required by competing models).
I've tried their 7b model, running locally on a 6gb laptop GPU. Its not fast, but the results I've had have rivaled GPT4. Its impressive.
- donsupreme 2y ago> I've tried their 7b model Anything other than their 671b model are just distilled models on top of Qwen and Llama using their 671b reasoning data output, right?
- KiwiJohnno 2y agoCorrect. Its the best model I've been able to run locally, by a long shot
- xiphias2 2y agoI believe you that it had to do with the selloff, but I believe that efficiency improvements are good news for NVIDIA: each card just got 20x more useful
- amazingamazing 2y agoeach card is not 20x more useful lol. there's no evidence yet that the deepseek architecture would even yield a substantially (20x) more performant model with more compute. if there's evidence to the contrary I'd love to see. in any case I don't think a h800 is even 20x better than a h100 anyway, so the 20x increase has to be wrong.
- jdietrich 2y agoWe need GPUs for inference, not just training. The Jevons Paradox suggests that reducing the cost per token will increase the overall demand for inference. Also, everything we know about LLMs points to an entirely predictable correlation between training compute and performance.
- amazingamazing 2y agothe jevons paradox isn't about any particular product or company's product, so is irrelevant here. the relevant resource here is compute, which is already a commodity. secondly, even if it were about GPUs in particular, there's no evidence that nvidia would be able to sustain such high margins if fewer were necessary for equivalent performance. things are currently supply constrained, which gives nvidia price optionality.
- Scoundreller 2y agoUhhh, isn’t it about coal?
- tshaddox 2y agoJevons paradox doesn't really suggest anything by itself. Jevons paradox is something that occurs in some instances of increased efficiency, but not all. I suppose the important question here is "What is the price elasticity of demand of inference?"
- mrbungie 2y agoPeople act like Jevons Paradox is an universal law thanks to Satya's tweet.
- ckw 2y agoPersonally, in the six months prior to the release of the deepseekv3 api, I'd made probably 100-200 api calls per month to llm services. In the past week I made 2.8 million api calls to dsv3.
- shawabawa3 2y agocan i ask what kind of api calls you're making to dsv3? Crunching through huge amounts of unstructured data or something?
- ckw 2y ago
- numba888 2y ago> there's no evidence yet that the deepseek architecture would even yield a substantially more performant model with more compute. It's supposed to. There was an info that the longer length of 'thinking' makes o3 model better than o1. I.e. at least at inference compute power still matters.
- amazingamazing 2y ago> It's supposed to. There was an info that the longer length of 'thinking' makes o3 model better than o1. I.e. at least at inference compute power still matters. compute matters, but performance doesn't scale with compute from what I've heard about o3 vs o1. you shouldn't take my word for it - go on the leaderboards and look at the top models from now, and then the top models from 2023 and look at the compute involved for both. there's obviously a huge increase, but it isn't proportional
- segasaturn 2y agoThat still means that that AI firms don't have to buy as many of Nvidia's chips, which is the whole thing that Nvidia's price was predicated on. FB, Google and Microsoft just had their their billions of dollars in Nvidia GPU capex blown out by $5M side-project. Tech firms are probably not going to be as generous shelling out whatever overinflated price Nvidia was asking for as they were a week ago.
- chgs 2y agoImagine what you can do with all that Nvidia hardware using the deep mind techniques.
- flowerlad 2y ago> That still means that that AI firms don't have to buy as many of Nvidia's chips Couldn’t you say that about Blackwell as well? Blackwell is 25x more energy-efficient for generative AI tasks and offer up to 2.5x faster AI training performance overall.
- Jlagreen 2y agoAnd yet, Blackwell is sold out. What does that tell us? The industry is compute starved and that makes totally sense. The tranformer model on which current LLMs are based on are 8 years old. But why took it so much time to get to the LLMs only 2 years ago? Simple, Nvidia first had to push the compute at scale strongly. Try training GPT4 on Voltas from 2017. Good luck with that! Current LLMs are possible thanks to the compute Nvidia has provided in the past decade. You could technically use 20 year old CPUs for LLMs but you might need to connect a billion of them.
- alfalfasprout 2y agoWrong conclusion, IMO. This makes inference more cost effective which means self-hosting suddenly becomes more attractive to a wider share of the market. GPUs will continue to be bought up as fast as fabs can spit them out.
- 2y ago
- jcgrillo 2y agoThey only got more useful if the AI goldrush participants actually strike, well, gold. Otherwise it's not useful at all. Afaict it remains to be seen whether any of this AI stuff has actual commercial value. It's all just speculation predicated on thoughts and prayers.
- sumeno 2y agoWhen your business is selling a large number of cards to giant companies you don't want them to be 20x more useful because then people will buy fewer of them to do the same amount of work
- HDThoreaun 2y agoor people do 30x more work and buy 50% more cards
- siwakotisaurav 2y agoNone of the models other than the 600b one are R1. They’re just prev gen models like llama or qwen trained on r1 output making them slightly better
- doctorpangloss 2y agoYeah but the second comment you see believes they are, and belief is truth when it comes to stock market gambling.
- int_19h 2y ago"Slightly" is an understatement, though. Distillations of R1 are significantly better than the underlying models.
- 2-3-7-43-1807 2y agothat's a Milchmädchenrechnung. if it turns out that you can achieve status quo with 1% of the expected effort then that just mean you can achieve approximately 10 times the status quo (assuming O(exp)) with the established budget! and this race is a race to the sky (as opposed to the bottom) ... he who reaches AGI first takes the cake, buddy.
- subarctic 2y agoIf that's the case, then I have high hopes that the increase in efficiency will result in more demand, not less. If only I could figure out how to buy NV stock quickly before it rebounds
- khazhoux 2y agoNot quite, I believe this sell off was caused by Shockley showing with their "transistor" that the electricity demands of computers are not necessarily as high as everyone has assumed (as required by vacuum tubes). Electricity demands will plummet when transistors take the place of vacuum tubes.
- deleted 2y ago[deleted]
- ipv6ipv4 2y agoTo me this rings a lot like “640KB ought to be enough for anybody” Similarly, as fast as processors have gotten, people still complain their applications are slow. Because they do so much more. Generally applicable ML is still in its infancy, and usage is exploding. All those newfound spare cycles will get soaked up fairly quickly.
- nickthegreek 2y agoIt’s made a NVIDIA Digits even more attractive to me now.
- Jlagreen 2y agoThe good thing is: Blackwell DC is $40k per piece and Digits is $3k per piece. So if 13x Digits are sold then it's the same turnover as a DC GPU for Nvidia. Yes, maybe lower margin but Nvidia can easily scale digits into masses compareds to Blackwell DC GPUs. In the end, the winner is Nvidia because Nvidia doesn't care if DC GPU, Gaming GPU, Digits GPU, Jetson GPU is used for AI as long as Nvidia is used 98% of time for AI workloads. That is the world domination goal, simple as that. And that's what Wallstreet doesn't get. Digits is 50% more turnover than the largest RTX GPU. On average gaming GPU turnover is probably around $500 per GPU. Nvidi probably sells 5 million gaming GPUs per quarter. Imagine they could reach such amounts of Digits. That would be $15b revenue and almost half of current DC revenue with Digits only.
- m00x 2y agoThat's a pretty terrible take. People who can use the 585B model will use the best model they can have. What DeepSeek really did was start an AI "space race" to AGI with China, and this race is running on Nvidia GPUs. Some hobbyists will run the smaller model, but if you could, why not use the bigger & better one? Model distillation has been a thing for over a decade, and LLM distillation has been widespread since 2023 [1]. There is nothing new in being able to leverage a bigger model to enrich smaller models. This is what people that don't understand the AI space got out of it, but it's clearly wrong. OpenAI has smaller models too with o1 mini and o4 mini, and phi-1 has shown that distillation could make a model 10x smaller perform as well as a much bigger model. The issue with these models is that they can't generalize as well. Bigger models will always win at first, then you can specialize them. Deepseek also showed that Nvidia GPUs could be more memory-efficient, which catapults Nvidia even further ahead of upcoming processors like Groq or AMD. [1] https://arxiv.org/abs/2305.02301 https://arxiv.org/abs/2305.02301
- datavirtue 2y agoExactly, and firing up reactors to train models just lost all its luster. Those standing before the Stargate will be bored with the whole thing by then end of the week.
- rsanek 2y agoI've run their distilled 70B model and didn't come away too impressed -- feels similar to the existing base model it was trained on, which also rivaled GPT4