3 ms·
yeah i'm curious how this would stack up to vllm in a batch setting long-term, bodes well as both methodologies should be combineable, just curious for current
by lmeyerov 3y ago
yeah i'm curious how this would stack up to vllm in a batch setting
long-term, bodes well as both methodologies should be combineable, just curious for current point-in-time wrt being relevant for production use
- chillee 3y agoI wouldn't recommend using it for a batch serving setting today. One crucial optimization for batched serving (which you need if you have a large number of requests) is continual batching, which this implementation doesn't have.