2 ms·
Single request depth on vllm with dspark, I'm getting ~200 tps, I'd say it's pretty snappy.
by ericd 2mo ago
Single request depth on vllm with dspark, I'm getting ~200 tps, I'd say it's pretty snappy.
- JacobAsmuth 2mo agoWell sure but you're running on tens of thousands of dollars of hardware.
- ericd 2mo agoIt's much faster than other models on that same hardware in the same size class. I've tested a few, it's by far the fastest I've tested. And it wasn't tens* until recently. Didn't expect this to be one of my best performing assets this year.
- k__ 2mo agoI get like 80.
- ericd 2mo agoWhat's your setup? Happy to try to point you in the direction that worked for me.