4 ms·
So throughput was already good but TTFT was the metric that needed more improvement?
by danudey 8mo ago
So throughput was already good but TTFT was the metric that needed more improvement?
- freehorse 8mo agoLet’s say TTFT needed the most improvement. At some point, loading the model with enough context size may take tens of seconds in some macs.
- convenwis 8mo agoGood is relative but first token was clearly the biggest limitation.
- zamadatix 8mo agoTo add to the sibling "good is relative" it also depends what you're running, not just your relative tolerances of what good is. E.g. in a MoE the decode speedup means the speed of prompt processing delay is more noticeable for the same size model in RAM.
- brookst 8mo agoYeah TTFT was terrible. I don’t think it’s unreasonable to benchmark the most-improved metric.