DeepSeek's KV cache is tiny compared to other open weight models. This actually makes very large inference batches viable even on consumer hardware, even when

DeepSeek's KV cache is tiny compared to other open weight models. This actually makes very large inference batches viable even on consumer hardware, even when resorting to SSD offload for weights. Once support is added to the main inference frameworks, it should be an absolute game changer for SOTA local inference.