3 ms·
I tried that that: ~1.5 to 3 tokens/sec.
by DrNosferatu 2y ago
I tried that that: ~1.5 to 3 tokens/sec.
- CamperBob2 2y agoOuch, thanks. About what I get now on a single-CPU box with 128 GB+a 4090. Was hoping for a major speedup.
- DrNosferatu 2y agoPeak performance is achieved at ~21 cores. Bottleneck - without any special configs - is RAM to CPU bandwidth. Let me know if you find some config that really leverages more cores!