4 ms·
how's the generation speed on CPU?
by FrozenSynapse 3y ago
how's the generation speed on CPU?
- selfhoster11 3y agoOn Ryzen 5600X, 7B and 13B run quite fast. Off the top of my head, pure CPU performance is about 25% slower than with an NVIDIA GPU of some kind. I don't remember the numbers off the top of my head, but the generation speed only starts to get annoying for 33B+ models.