5 ms·
Assuming you want to maintain full bandwidth. Which I don't care too much about. However, even 16->24GB is a big step, since a lot of the model are developed
by frognumber 3y ago
Assuming you want to maintain full bandwidth.
Which I don't care too much about.
However, even 16->24GB is a big step, since a lot of the model are developed for 3090/4090-class hardware. 36GB would place it lose to the class of the fancy 40GB data center cards.
If Intel decided to push VRAM, it will definitely have a market. Critically, a lot of folks will also be incentivized to make software compatible, since it will be the cheapest way to run models.
- 0cf8612b2e1e 3y agoAt this point, I cannot run an entire class of models without OOM. I will take a performance hit if it lets me run it at all. I want a consumer card that can do some number of tokens per second. I do not need a monster that can serve as the basis for a startup.
- hnfong 3y agoA maxed out Mac Studio probably fits your requirements as stated.
- 0cf8612b2e1e 3y agoIf I were willing to drop $4k on that setup, I might as well get the real NVidia offering. The hobbyist market needs something priced well under $1k to make it accessible.
- rnewme 3y agoHow comes you don't care about full bandwidth?
- Dalewyn 3y agoThe thing about RAM speed (aka bandwidth) is that it becomes irrelevant if you run out and have to page out to slower tiers of storage.
- frognumber 3y agoMostly because I use this for development. If a model takes twice as long to run.... I'll live. Worst-case, it will be mildly annoying. If I can't run a model, that's a critical failure. There's a huge step up CPU->GPU which I need, but 3060 versus 4090 isn't a big deal at all. Indeed, the 24GB versus 16GB is a bigger difference than the number of CUDA cores.