3 ms·
Someone really needed a few hundred TB to waste on inference and went looking under the rugs…
by sroussey 8d ago
Someone really needed a few hundred TB to waste on inference and went looking under the rugs…
- why_only_15 8d agoCPU DRAM can't really be used for inference efficiently -- inference mostly wants memory bandwidth, not memory capacity, and GPU DRAM has >10x more bandwidth. The fabs can switch between them but you can't switch after the fact.
- sroussey 8d agoThose machines with GPUs still need RAM of their own, and they generally want large caches to avoid SSD penalties. You even see this spill out in the form of costs for KV cache in <1min, 5m, 1hr rates etc.
- why_only_15 5d agoyeah there's some KV cache offload
- halJordan 8d agoThe majority of inference actually does happen in cpu.
- why_only_15 5d agoThis is just wrong unless you're using some confusing definition. Notice that companies trying to do lots of inference aren't looking for CPUs, they're looking for GPUs/TPUs.