3 ms·
> The factor of 72 is accidentially also the number of GPUs in a full GB200 DGX rack Sure, but in the article it is already mentioned as 1000 TPS/User for an 8
by supermatt 1y ago
> The factor of 72 is accidentially also the number of GPUs in a full GB200 DGX rack
Sure, but in the article it is already mentioned as 1000 TPS/User for an 8 GPU node, and the rack contains 9 nodes - i.e. 9x more GPUs, not 72x - so the 72000 TPS/Server simply being a multiple of 72 seems like a red herring.
But yeah, I agree that 72x seems high - although only 9x seems low given vLLM showing over 20x speedups with continuous batching. I guess there are a lot of variables.