3 ms·
In general, are the per-layer KV caches all independent of each other? If we split a model layer-wise for inference across X GPUs, does the cache split as well
by algo_trader 2y ago
In general, are the per-layer KV caches all independent of each other?
If we split a model layer-wise for inference across X GPUs, does the cache split as well?