4 ms·
If 8 x RTX 6000 is getting you 20s before initial token, how are cloud vendors doing this?
by alfiedotwtf 6mo ago
If 8 x RTX 6000 is getting you 20s before initial token, how are cloud vendors doing this?
- CamperBob2 6mo agoRTX6000s are great but they are several times slower than a real datacenter-grade GPU. They still use DDR memory rather than HBM, for example.