2 ms·
If previous quantization results hold up, fp8 will have nearly identical performance while using 405GiB for weights, but the KV cache size will still be signifi
by AaronFriel 2y ago
If previous quantization results hold up, fp8 will have nearly identical performance while using 405GiB for weights, but the KV cache size will still be significant.
Too bad, too, I don't think my PC will fit 20 4090s (480GiB).
- knicholes 2y agoI've got a motherboard that will support 8!
- Zambyte 2y ago40,320 4090s?? What witchcraft is this?! :D
- sebastiennight 2y agoAll the more impressive when you realize that Groq's infrastructure (based on LPUs) was built using only 6!