4 ms·
only 2 ? what kind of h100s do you have?
by htrp 2y ago
only 2 ? what kind of h100s do you have?
- TechDebtDevin 2y agoI meant quantized versions but yea, I get your point.
- dathinab 2y agoThere are different versions of the model as well as using it with different levels of quantization. Some variants of DeepSeek-R1 can be run on 2x H100 GPUs, and some people managed to get still quite decent results with a even stronger distilled mode running it on consumer hardware. For DeepSeek-V3 even with 4bit quantization you need more like 16x H100.