3 ms·
Isn't it only the case when inference isn't batched?
by littlestymaar 2y ago
Isn't it only the case when inference isn't batched?
- Tostino 2y agoEven in a local setting, batched inference is useful to be able to run more "complex" workflows (with multiple, parallel LLM calls for a single interaction). There is very little reason to optimize for just single stream inference at the expense of your batch inference performance.