3 ms·
> or even a single socket Epyc system with >1TB of RAM How many tokens/second would this likely achieve?
by gpm 1y ago
> or even a single socket Epyc system with >1TB of RAM
How many tokens/second would this likely achieve?
- deleted 1y ago[deleted]
- neuroelectron 1y ago1
- kachapopopow 1y agoaround 1 by the time you try to do anything useful with it (>10000 tokens)
- chithanh 1y agoKTransformers now supports Kimi K2 for MoE offloading They claim 14 tps for the 4-bit quant on a single socket system with 600 GB RAM and 14 GB GPU memory.