4 ms·
Cheap is relative I suppose. I’m running Vicuna 13b 16f locally and it needs 26GB of VRAM, which won’t even fit on a single RTX 4090. The next gen RTX Titan mi
by valine 4y ago
Cheap is relative I suppose.
I’m running Vicuna 13b 16f locally and it needs 26GB of VRAM, which won’t even fit on a single RTX 4090. The next gen RTX Titan might have enough vram but that won’t come cheap. I’m expecting a price point above $2500.
- 22c 4y agoI'm not sure if it's the point GP is trying to make, but I would like to see GPUs with extra VRAM that don't have the extra compute. eg. similar performance of a 4070Ti but with 24GB or 32GB of VRAM. I don't see a really good reason why OEMs couldn't do that now, in the past there have been OEM cards that have more VRAM than the reference design. I'm sure there's an appetite for cards like that for people who don't want to refinance their home loan to get 2 x RTX 4090 cards.
- Tepix 4y agoI'm fairly sure that NVIDIA is making sure that consumer cards are no good alternative to their $10000 80GB VRAM A100 cards.
- MacsHeadroom 4y ago13B in GPTQ 4bit has practically no quality loss and runs in 8GB. I get 8 tokens/second on my laptop CPU and 4 tokens/second on my phone CPU. Even 33B only needs 20GB of VRAM in GPTQ 4bit. 8bit has zero perplexity loss, so there's really no reason to run in 16bit. Even a $200 P40 24GB is enough to run 33B at extremely high speeds in GPTQ 4bit.