4 ms·

It's not about storing data on disk, it's about keeping data resident in memory.

by simonw 1y ago

criemen 1y ago
Fascinating, so I have to think more "pay for RAM/redis" than "pay for SSD"?
- nthypes 1y ago
  "pay for data on VRAM" RAM of GPU
  - criemen 1y ago
    But that doesn't make sense? Why would they keep the cache persistent in the VRAM of the GPU nodes, which are needed for model weights? Shouldn't they be able to swap in/out the kvcache of your prompt when you actually use it?
    - minimaxir 1y ago
      That is slow.
    - dotancohen 1y ago
      They are not caching to save network bandwidth. They are caching to increase interference speed and reduce (their own) costs.
    - tazjin 1y ago
      Your intuition is correct and the sibling comments are wrong. Modern LLM inference servers support hierarchical caches (where data moves to slower storage tiers), often with pluggable backends. A popular open-source backend for the "slow" tier is Mooncake: https://github.com/kvcache-ai/Mooncake https://github.com/kvcache-ai/Mooncake
      - simonw 1y ago
        OK that's pretty fascinating, turns out Mooncake includes a trick that can populate GPU VRAM directly from NVMe SSD without it having to go through the host's regular CPU and RAM first! https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/transfer-engine.md#2-nvmeof-segment https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/tran... > Transfer Engine also leverages the NVMeof protocol to support direct data transfer from files on NVMe to DRAM/VRAM via PCIe, without going through the CPU and achieving zero-copy.
jbellis 1y ago
Deepseek pioneered automatic prefix caching and caches on SSD. SSD reads are so fast compared to LLM inference that I can't think of a reason to waste ram on it.
- jychang 1y ago
  It’s not instantly fast though. Context is probably ~20gb of VRAM at max context size. That’s gonna take some time to get from SSD no matter what. TtFT will get slower if you export kv cache to SSD.

It's not about storing data on disk, it's about keeping data resident in memory.

by simonw 1y ago

It's not about storing data on disk, it's about keeping data resident in memory.

criemen 1y ago

Fascinating, so I have to think more "pay for RAM/redis" than "pay for SSD"?

nthypes 1y ago
"pay for data on VRAM" RAM of GPU
- criemen 1y ago
  But that doesn't make sense? Why would they keep the cache persistent in the VRAM of the GPU nodes, which are needed for model weights? Shouldn't they be able to swap in/out the kvcache of your prompt when you actually use it?
  - minimaxir 1y ago
    That is slow.
  - dotancohen 1y ago
    They are not caching to save network bandwidth. They are caching to increase interference speed and reduce (their own) costs.
  - tazjin 1y ago
    Your intuition is correct and the sibling comments are wrong. Modern LLM inference servers support hierarchical caches (where data moves to slower storage tiers), often with pluggable backends. A popular open-source backend for the "slow" tier is Mooncake: https://github.com/kvcache-ai/Mooncake https://github.com/kvcache-ai/Mooncake
    - simonw 1y ago
      OK that's pretty fascinating, turns out Mooncake includes a trick that can populate GPU VRAM directly from NVMe SSD without it having to go through the host's regular CPU and RAM first! https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/transfer-engine.md#2-nvmeof-segment https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/tran... > Transfer Engine also leverages the NVMeof protocol to support direct data transfer from files on NVMe to DRAM/VRAM via PCIe, without going through the CPU and achieving zero-copy.

jbellis 1y ago

Deepseek pioneered automatic prefix caching and caches on SSD. SSD reads are so fast compared to LLM inference that I can't think of a reason to waste ram on it.

jychang 1y ago
It’s not instantly fast though. Context is probably ~20gb of VRAM at max context size. That’s gonna take some time to get from SSD no matter what. TtFT will get slower if you export kv cache to SSD.