3 ms·
It’s pretty great that despite having large data centers capable of doing this kind of computation, Apple continues to make things work locally. I think there i
by imoverclocked 1y ago
It’s pretty great that despite having large data centers capable of doing this kind of computation, Apple continues to make things work locally. I think there is a lot of value in being able to hold the entirety of a product in hand.
- xnx 1y agoGoogle has a family of local models too! https://ai.google.dev/gemma/docs https://ai.google.dev/gemma/docs
- ivape 1y agoGemma and Llama can’t be bundled commercially, which sucks because they make two of the leading small llms. Qwen3 might be the last one with an Apache license.
- nolist_policy 1y agoYou can bundle and use Gemma commercially[1]. [1] https://ai.google.dev/gemma/terms https://ai.google.dev/gemma/terms
- ivape 1y agoI’ll have to read that, thanks.
- coliveira 1y agoIt's very convenient for Apple to do this: less expenses on costly AI chips, and more excuses to ask customers to buy their latest hardware.
- nine_k 1y agoUsers have to pay for the compute somehow. Maybe by paying for models run in datacenters. Maybe paying for hardware that's capable enough to run models locally.
- Bootvis 1y agoI can upgrade to a bigger LLM I use through an API with one click. If it runs on my device device I need to buy a new phone.
- nine_k 1y agoI* can run the model on my device, no matter if I have an internet connection, nor if I have a permission from whoever controls the datacenter. I can run the model against highly private data while being certain that the private data never leaves my device. It's a different set of trade-offs. * Theoretically; I don't own an iPhone.
- eru 1y agoWell, unless it's open source, you can't be so certain. But more certain than when processing in the cloud, that's true.
- lostlogin 1y agoBut also: if Apple's way works, it’s incredibly wasteful. Server side means shared resources, shared upgrades and shared costs. The privacy aspect matters, but at what cost?
- shakna 1y agoServer side means an excuse to not improve model handling everywhere you can, and increasing global power usage by noticable percentage point, at a time when we're approaching "point of no return" with burning out the only planet we can live on. The cost, so far, is greater.
- hu3 1y ago> Server side means an excuse to not improve model handling everywhere you can... How so if efficiency is key for datacenters to be competitive? If anything it's the other way around.
- v5v3 1y agoWith no company having a clear lead in everyday ai for the non technical mainstream user, there is only going to be a race to the bottom for subscription and API pricing. Local doesn't cost the company anything, and increases the minimum hardware customers need to buy.
- eru 1y ago> Local doesn't cost the company anything, [...] Not completely true: those models are harder to develop. The logistics are a hassle.
- ivape 1y agoIt takes about a $400 dollar graphics card to comfortably run something like a 3b-8b model. Comfortable as in fast inference, good sized context. 3b-5b models are what devices can somewhat fit. That means for us to get good running local models, we’d have to shrink one of those $400 dollar graphics cards down to a phone. I don’t see this happening in the next 5 years. The Mac mini being shrunk down to phone size is probably the better bet. We’d have to bring down the power consumption requirements too by a lot. Edge hardware is a ways off.
- nolist_policy 1y agoGemma 3n E4B runs at 35tk/s prompt processing and 7-8 tk/s decode on my last last last gen flagship Android.
- ivape 1y agoI doubt this. What kind of t/s are you getting once your context window is reasonably saturated? Probably slows down to a crawl making it not good enough yet (the hardware that is).