0.05 to 0.1 per sec could still be quite useful if it was the speed for inferring a whole batch of tokens concurrently. Of course this actually requires fairly

0.05 to 0.1 per sec could still be quite useful if it was the speed for inferring a whole batch of tokens concurrently. Of course this actually requires fairly good SSD read performance (since you need to read a sizeable fraction of the complete model at every token batch in order to get good reuse) and is ultimately limited by CPU/GPU thermals which are a tight constraint on typical inference platforms. It's also only really feasible with tiny KV caches, which requires either a very small context or sticking to KV-cache efficient models such as the DeepSeek V4 series. Still, this might be one way of making use of existing lower-end hardware for practical inference of non-tiny models.