4 ms·
Cool! Is there tok/s too? TTFT and tok/s also decreases given a larger context window, maybe a metric of tok/s given # tok in context would also be nice.
by chaidhat 10mo ago
Cool! Is there tok/s too? TTFT and tok/s also decreases given a larger context window, maybe a metric of tok/s given # tok in context would also be nice.