It’s even worse in battery constrained devices - they tend to also be memory constrained and run with batch size 1 during extend. IOW the entire model (or parts

It’s even worse in battery constrained devices - they tend to also be memory constrained and run with batch size 1 during extend. IOW the entire model (or parts thereof, if the model is MoE), gets read for every generated token. Utilization of compute is truly abysmal in that case and almost all energy is spent pushing bytes through the memory bus, which on battery powered devices doesn’t have high throughput