4 ms·
It does, but proving that it can be done with cheaper (and more importantly for NVidia), lower margin chips breaks the spell that NVidia will just be eating eve
by yifanl 2y ago
It does, but proving that it can be done with cheaper (and more importantly for NVidia), lower margin chips breaks the spell that NVidia will just be eating everybody's lunch until the end of time.
- aurareturn 2y agoIf demand for AI chips will increase due to Jevon’s paradox, why would Nvidia’s chips become cheaper? In the long run, yes, they will be cheaper due to more competition and better tech. But next month? It will be more expensive.
- yifanl 2y agoThe usage of existing but cheaper nvidia chips to make models of similar quality is the main takeaway. It'll be much harder to convince people to buy the latest and greatest with this out there.
- UncleOxidant 2y agoThe sweet spot for running local LLMs (from what I'm seeing on forums like r/localLlama) is 2 to 4 3090s each with 24GB of VRAM. NVidia (or AMD or Intel) would clean up if they offered a card with 3090 level performance but with 64GB of VRAM. Doesn't have to be the leading edge GPU, just a decent GPU with lots of VRAM. This is kind of what Digits will be (though the memory bandwidth is going to be slower with because it'll be DDR5) and kind of what AMD's Strix Halo is aiming for - unified memory systems where the CPU & GPU have access to the same large pool of memory.
- redlock 2y agoThe issue here is that, even with a lot of VRAM, you may be able to run the model, but with a large context, it will still be too slow. (For example, running LLaMA 70B with a 30k+ context prompt takes minutes to process.)
- aurareturn 2y agoThe usage of existing but cheaper nvidia chips to make models of similar quality is the main takeaway. So why not buy a more expensive Nvidia chip to run a better model?
- yifanl 2y agoIs there still evidence that more compute = better model?
- aurareturn 2y agoYes. Plenty of evidence. The DeepSeek R1 model people are freaking out about, runs better with more compute because it's a chain of thoughts model.
- Vegenoid 2y agoBecause if you don't have infinite money, considering whether to buy a thing is about the ratio of price to performance, not just performance. If you can get enough performance for your needs out of a cheaper chip, you buy the cheaper chip.
- aurareturn 2y agoThe AI industry isn't pausing because DeepSeek is good enough. The industry is in an arms race to AGI. Having a more efficient method to train and use LLMs only accelerates progress, leading to more chip demand.
- ozgrakkurt 2y agoThere is no indication that adding more compute will give AGI