3 ms·
Your intuition is correct and the sibling comments are wrong. Modern LLM inference servers support hierarchical caches (where data moves to slower storage tiers
by tazjin 1y ago
Your intuition is correct and the sibling comments are wrong. Modern LLM inference servers support hierarchical caches (where data moves to slower storage tiers), often with pluggable backends. A popular open-source backend for the "slow" tier is Mooncake: https://github.com/kvcache-ai/Mooncake https://github.com/kvcache-ai/Mooncake
- simonw 1y agoOK that's pretty fascinating, turns out Mooncake includes a trick that can populate GPU VRAM directly from NVMe SSD without it having to go through the host's regular CPU and RAM first! https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/transfer-engine.md#2-nvmeof-segment https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/tran... > Transfer Engine also leverages the NVMeof protocol to support direct data transfer from files on NVMe to DRAM/VRAM via PCIe, without going through the CPU and achieving zero-copy.