3 ms·
Wrt. language models/transformers, the neural engine/NPU is still potentially useful for the pre-processing step, which is generally compute-limited. For token
by zozbot234 1y ago
Wrt. language models/transformers, the neural engine/NPU is still potentially useful for the pre-processing step, which is generally compute-limited. For token generation you need memory bandwidth so GPU compute with neural/tensor accelerators is preferable.
- fooblaster 1y agoI think I'd still rather have the hardware area put into tensor cores for the GPU instead of this unit that's only programmable with onnx.