3 ms·
What I'd like to know is how well those dual-Epyc machines run the 1.58 bit dynamic quant model. It really does seem to be almost as good as the full Q8.
by CamperBob2 2y ago
What I'd like to know is how well those dual-Epyc machines run the 1.58 bit dynamic quant model. It really does seem to be almost as good as the full Q8.
- DrNosferatu 2y agoI tried that that: ~1.5 to 3 tokens/sec.
- CamperBob2 2y agoOuch, thanks. About what I get now on a single-CPU box with 128 GB+a 4090. Was hoping for a major speedup.
- DrNosferatu 2y agoPeak performance is achieved at ~21 cores. Bottleneck - without any special configs - is RAM to CPU bandwidth. Let me know if you find some config that really leverages more cores!