3 ms·
On a CPU I'd estimate it would get a maximum of around 5 tokens per second (a token being a sub-word token, so generally a couple of letters). I suspect it'd be
by nl 4y ago
On a CPU I'd estimate it would get a maximum of around 5 tokens per second (a token being a sub-word token, so generally a couple of letters). I suspect it'd be more like 1 token per second on the large model without additional optimisation.
Yes models can be split up. See eg Hugging Face Accelerate.
- davrosthedalek 4y agoThat's actually a lot better than I would have thought. Almost usable, and a good exercise in patience.
- nl 4y agoI'd expect significant performance improvements over the next few months are more people work on this in the same way the stable diffusion is now fairly usable on a CPU. It's always going to be slow on a CPU, but the smaller models might be usable for experimentation at some point.