3 ms·

There are billions of computers, but only a few computers can run very large local LLMs. So instead, imagine billions of laptops (or phones) available in store

by rvz 10mo ago

There are billions of computers, but only a few computers can run very large local LLMs. So instead, imagine billions of laptops (or phones) available in store that have local LLMs installed which allows you to have intelligence-like chatbot features available with the equivalent performance of Claude Opus 4.5 or GPT-5 if not better and they are far smaller. Thank you for telling us that efficient + performant multi-modal local AI models is the endgame; a bonus if they are smaller to fit in phones.

zamadatix 10mo ago
Centralized vs distributed feels like a largely orthogonal matter. E.g. phones are great at being clients, even for things which they could technically be the server for today, but they make pretty bad servers in terms of resource tradeoffs. Because of this, I think the endgame cares little about where the LLM runs as much as how much it costs for those devices to use one (which can be more than just money too - e.g. on phones it's battery, bandwidth, latency, privacy, or whatever else could be valued/scarce). That it's specifically on device is just one method of how to meet some of these attributes, not the exclusive method to use it. Even as a techie tinkerer my dream is not that every device I have run its own LLM - it's that I can point them to where to find the LLM (and swap that at will). The dumber I can make my edges the more they can end up doing overall.
fenwick67 10mo ago
I'm skeptical of this. Users are not going to want to sacrifice on the quality or performance of the model for a warm fuzzy feeling about it running locally on their phone. Outside of hobbyists I can't imagine this really taking off for the same reason secure scuttlebutt never had a chance of beating Facebook. The only exception I can imagine is very sensitive topics, AI girlfriends or maybe image processing

by rvz 10mo ago

zamadatix 10mo ago

Centralized vs distributed feels like a largely orthogonal matter. E.g. phones are great at being clients, even for things which they could technically be the server for today, but they make pretty bad servers in terms of resource tradeoffs. Because of this, I think the endgame cares little about where the LLM runs as much as how much it costs for those devices to use one (which can be more than just money too - e.g. on phones it's battery, bandwidth, latency, privacy, or whatever else could be valued/scarce). That it's specifically on device is just one method of how to meet some of these attributes, not the exclusive method to use it. Even as a techie tinkerer my dream is not that every device I have run its own LLM - it's that I can point them to where to find the LLM (and swap that at will). The dumber I can make my edges the more they can end up doing overall.

fenwick67 10mo ago

I'm skeptical of this. Users are not going to want to sacrifice on the quality or performance of the model for a warm fuzzy feeling about it running locally on their phone. Outside of hobbyists I can't imagine this really taking off for the same reason secure scuttlebutt never had a chance of beating Facebook. The only exception I can imagine is very sensitive topics, AI girlfriends or maybe image processing