4 ms·
the useful split for me is interactive vs batch. keep a small model warm for private jobs and measure queue time, not just tokens/sec.
by po_westnet26 28d ago
the useful split for me is interactive vs batch. keep a small model warm for private jobs and measure queue time, not just tokens/sec.
- Ganeshan007 27d agoagreed queue time should be measured and added to the total wait time for the end user