8 ms·
NPUs are more useful for prefill than decode anyway. Memory bandwidth is not the bottleneck for prefill.
by zozbot234 8mo ago
NPUs are more useful for prefill than decode anyway. Memory bandwidth is not the bottleneck for prefill.
loading thread…