5 ms·
Over the last few days people have asked me if they think NVIDIA is fkd.. It still takes two H100s to run inference on the DS v3 671b @ <200 tokens per second.
by TechDebtDevin 2y ago
Over the last few days people have asked me if they think NVIDIA is fkd.. It still takes two H100s to run inference on the DS v3 671b @ <200 tokens per second.
- htrp 2y agoonly 2 ? what kind of h100s do you have?
- TechDebtDevin 2y agoI meant quantized versions but yea, I get your point.
- dathinab 2y agoThere are different versions of the model as well as using it with different levels of quantization. Some variants of DeepSeek-R1 can be run on 2x H100 GPUs, and some people managed to get still quite decent results with a even stronger distilled mode running it on consumer hardware. For DeepSeek-V3 even with 4bit quantization you need more like 16x H100.