4 ms·
In previous generations, throughout was excellent for an integrated GPU, but the time to first token was lacking.
by GeekyBear 8mo ago
In previous generations, throughout was excellent for an integrated GPU, but the time to first token was lacking.
- danudey 8mo agoSo throughput was already good but TTFT was the metric that needed more improvement?
- freehorse 8mo agoLet’s say TTFT needed the most improvement. At some point, loading the model with enough context size may take tens of seconds in some macs.
- convenwis 8mo agoGood is relative but first token was clearly the biggest limitation.
- zamadatix 8mo agoTo add to the sibling "good is relative" it also depends what you're running, not just your relative tolerances of what good is. E.g. in a MoE the decode speedup means the speed of prompt processing delay is more noticeable for the same size model in RAM.
- brookst 8mo agoYeah TTFT was terrible. I don’t think it’s unreasonable to benchmark the most-improved metric.