4 ms·
Memory is expensive? If reads are as rare as they claim you can just stash the KV-cache on spinning disk.
by zozbot234 7mo ago
Memory is expensive? If reads are as rare as they claim you can just stash the KV-cache on spinning disk.
- stingraycharles 7mo agoAren’t those latency sensitive though?