4 ms·
only 44GB * 3 of VRAM per rack :O I guess you'd need a DOZEN(s) of these to host a large model with long context KV caches?
by charlielidbury 2mo ago
only 44GB * 3 of VRAM per rack :O
I guess you'd need a DOZEN(s) of these to host a large model with long context KV caches?