2 ms·
This could still be useful for battery constrained devices, right?
by entropicdrifter 2y ago
This could still be useful for battery constrained devices, right?
- ein0p 2y agoIt’s even worse in battery constrained devices - they tend to also be memory constrained and run with batch size 1 during extend. IOW the entire model (or parts thereof, if the model is MoE), gets read for every generated token. Utilization of compute is truly abysmal in that case and almost all energy is spent pushing bytes through the memory bus, which on battery powered devices doesn’t have high throughput