3 ms·
Anyone here has tips for the code and hardware setup to get best per-GPU throughput on H200 or B200 hardware for large reasoning traces and inputs of around 10k
by pama 1y ago
Anyone here has tips for the code and hardware setup to get best per-GPU throughput on H200 or B200 hardware for large reasoning traces and inputs of around 10k–40k tokens? Is there an equivalent effort to sglang’s optimization of the V3/R1 throughput for this class of models?