4 ms·
Even with smaller models & more optimized hardware, I think edge compute is going to be power-limited first. Batteries today just won’t support constantly runni
by cgearhart 4y ago
Even with smaller models & more optimized hardware, I think edge compute is going to be power-limited first. Batteries today just won’t support constantly running LLMs. But I joked recently that as long as they prove useful then consumers would be willing to swap their iPhone for the old car battery with a phone handle attached.
- BulgarianIdiot 4y agoThe first step is for it to be viable for smaller models on desktops. The rest will follow, as hardware catches up. Hybrid analog-hybrid NN hardware is on the horizon, maybe in 3-4 years. This would allow GPT-4 level performance on an iPhone with plausible battery life. The current hardware of course can't pull anything like this yet. But iPhone supports on-device facial recognition, object recognition, dictation and translation, so small steps...
- boppo1 4y agoWho is making said hardware?
- BulgarianIdiot 4y agoThere are few efforts under way. https://research.ibm.com/blog/why-we-need-analog-AI-hardware https://research.ibm.com/blog/why-we-need-analog-AI-hardware https://news.mit.edu/2022/analog-deep-learning-ai-computing-0728 https://news.mit.edu/2022/analog-deep-learning-ai-computing-...