3 ms·
the useful split for me is interactive vs batch. keep a small model warm for private jobs and measure queue time, not just tokens/sec.
by po_westnet26 2mo ago
the useful split for me is interactive vs batch. keep a small model warm for private jobs and measure queue time, not just tokens/sec.
- Ganeshan007 2mo agoagreed queue time should be measured and added to the total wait time for the end user