5 ms·
On-device models are the future. Users prefer them. No privacy issues. No dealing with connectivity, tokens, or changes to vendors implementations. I have an ap
by abu_ameena 6mo ago
On-device models are the future. Users prefer them. No privacy issues. No dealing with connectivity, tokens, or changes to vendors implementations. I have an app using Foundation Model, and it works great. I only wish I could backport it to pre macOS 26 versions.
- whazor 6mo agoObviously hardware wise the real blocker is memory cost. But there is no reason why future devices couldn't bundle 256GB of mem by default.
- michaelmior 6mo ago> no reason why future devices couldn't bundle 256GB of mem by default Cost is a pretty big reason.
- raw_anon_1111 6mo agoUsers don’t care about “privacy”. If they did, Meta and Alphabet wouldn’t be worth $1T+. Users really don’t matter at all. The revenue for AI companies will be B2B where the user is not the customer - including coding agents. Most people don’t even use computers as their primary “computing device” and most people are buying crappy low end Android phones - no I’m not saying all Android phones are crappy. But that’s what most people are buying with the average selling price of an Android phone being $300.
- barelysapient 6mo agoDifferent users. Many people care about privacy and aren’t using Meta products. And many businesses care about it too and have information policies to protect their IP.
- raw_anon_1111 6mo ago70% of the world’s population use at least one Meta property at least once per day. How many of the other 30% are too poor/young/computer illiterate to be part of an addressable market? Every company has dozens of SaaS products that store their business critical information. Amazon installs Office on each computer, Slack (they were moving away from Chime when I left), and the sales department uses SalesForce - SA’s and Professional Services (former employee). The addressable market of even companies that care about privacy is not a large addressable market. How long will it be before computers become cheap enough that can run even GPT 4 level LLMs that companies will give it to all of their developers?
- innagadadavida 6mo agoThese are all great statistics, but how do you explain ClawdBot explosion. Even in lower income countries like China. So much demand that Apple can’t keep up production of Mac Minis. Why aren’t these folks going towards cloud solutions? Is it cost or is there some consideration for having more control over their data?
- zozbot234 6mo agoClawBot doesn't generally run the model locally, it just talks to remote APIs. No different than any other agentic harness. You could run a local model on the same Mac Mini as your agent, but it wouldn't be very smart and many agentic tasks around computer GUI/browser use, etc. would be out of reach.
- raw_anon_1111 6mo agoAnd people using Clawdbot are still not using local inference for the most part… They aren’t buying high end $2000+ Mac Minis.
- bigyabai 6mo ago> Why aren’t these folks going towards cloud solutions? They are. The majority aren't doing inference on a Mac Mini, but instead using it as a local host for cloud-based inference. You could have the same general experience on a $200 Chromebook or $300 Windows box.
- amelius 6mo ago> Different users. Many people care about privacy and aren’t using Meta products. Yeah but if they can rake in 100x as much by making products for people who don't care about privacy, then why spend time developing stuff for people who care? There is still a small market left, of course, but that market will not have the billions of R&D behind it.
- woopsn 6mo agoIt's largely out of Meta's hands now anyway. The risk here not so much to privacy (it's Apple) but they'll walled garden the model space somehow for sure.
- bigyabai 6mo ago> but they'll walled garden the model space somehow for sure. People have said this since Pytorch was published and it's not any more true now than it was 10 years ago.
- abu_ameena 6mo agoI see it as a long-term tradeoff on user freedom. You pay upfront for a capable hardware, you get your services running locally (you don’t pay subscriptions). Or you buy cheap hardware, you still need the same services “running in some cloud” for $X monthly. X goes up depending on the corporate bottom-line
- raw_anon_1111 6mo agoIn the history of cloud computing, prices have mostly only come down especially as inference becomes a commodity. Realistically, just looking at Mac prices, the cost of a computer with decent local inference would be around $6000 per person. The world is not moving back to on prem.
- esseph 6mo ago> The world is not moving back to on prem. Lol, you should tell my customers (that are moving back on prem) that! You should also tell Microsoft, who just yesterday said they are going back to focusing on local apps.
- raw_anon_1111 6mo agoYour customers are an anecdote, now compare that to the publicly reported numbers from AWS, GCP and Azure where they all say the only thing keeping them from growing more is the chip shortage.
- esseph 6mo agoOh I'm sure they'll continue to have some cloud services, no doubt. But look at VMware for example, even after the insane price increases. Nutanix also seems to be doing quite well. I'm seeing a fair amount of on-prem bare metal k8s too.
- raw_anon_1111 6mo agoAgain - anecdotes is not data. We have data. That would be about as silly as me citing my own experience as proof that “everyone is moving to AWS” when I work for a company that is exclusively an AWS partner consulting company.
- DesiLurker 6mo agoyou are missing a but 'given a choice' disclaimer. Meta is pretty much a monopoly in social space. So is Android. given a choice people will absolutely gravitate towards not-always-snooping device. most people with resources anyway, who matter for the AI adoption. Oh an wait till ad companies start selling your healthcare data and you will see how fast things turn 'given a choice'.
- raw_anon_1111 6mo agoPeople A) don’t have to use Meta and B) do have a choice between not using a mobile phone by an ad tech company.
- nozzlegear 6mo agoPeople don't have a choice between Facebook and not-Facebook-but-still-has-all-of-your-friends-and-family. Abstinence isn't a choice here any more than shutting off your cell phone service is a choice; true in the literal sense, but only if you don't mind being unreachable to everyone who still has a phone.
- raw_anon_1111 6mo agoAnd they do have a choice on proactively giving FB more information than just what it infers
- roadside_picnic 6mo ago> Users don’t care about “privacy”. I worked for a research focused AI startup that had a strict "no external LLM" policy for code touching our core research. You're right that the average consumer doesn't care about privacy, but there are many, many users who do. The average consumer also don't have a desktop with GPU or high end Mac Studio, but that doesn't mean there aren't many people working with AI how do have these things. If we continue to see improvements in running local models, and RAM prices continue to fall as they have in the last month, then suddenly you don't have to worry about token counts any more and can be much more trusting of your agents since they are fully under your control.
- charcircuit 6mo agoThose users are addressed by being able to rent their own exclusive machines to run the model on. There will be some compromise that will be made to get access to the best intelligence available.
- Angostura 6mo agoIt’s not all or nothing there ads trade offs. The fact that Apple still bothers to expend marketing effort on its privacy chops suggests significant numbers of people still do care.
- ilovecake1984 6mo agoUsers here probably means corporations. I still don’t see much use of LLMs in my personal life, other than one thing. Googling stuff in a foreign language.
- api 6mo ago"Users" is a large set of people. Many don't care about privacy, but some do. There's also a difference between where you post random social media stuff vs what you run with something like OpenClaw and give access to your machine.
- Nevermark 6mo agoHave you done A/B tests to see if consumers prefer Facebook with or without privacy? No? What? Oh, you can't? Neither can consumers. Most consumers are very aware of the lack of privacy, the manipulation, and have very cynical feelings about Facebook and similar companies. But it's where their friends and family are. For most people the web is a mine field maze where basic things they want are compromised everywhere. And they are routinely creeped out by ads that reveal they know them far too personally. You are mistaking network capture for preference. Another telling example. Lots of privacy valuing technical people, who would never have a Facebook account, send unencrypted text emails. It is network capture, not preference.
- raw_anon_1111 6mo agoConsumers pro actively tell Facebook their age, sexual preference, race, relationship status, likes and dislikes, they check in to where they are and who they are there with… They are choosing to give Facebook info.
- Nevermark 6mo ago> They are choosing to give Facebook info. Yes, they do. That's is exactly the phenomena my comment addressed. But the way you wrote that implies an improbable motivation or choice framing. Perhaps their real motive/choice is to share with other people on the site. It is called a network effect. If (1) Facebook had been the surveillance/manipulation capital of the world from inception, (2) an equally inviting privacy protecting site took off at the same time, and (3) everyone chose Facebook over E2EE anyway, then sure, we could throw up our hands! Those silly users! The term I have for when people discuss choices involving many-dimensional criteria, as if the choice involved just one or two selected dimensions, is "dimension blindness". It happens in a lot of heated discussions about phone choices too.
- raw_anon_1111 6mo agoWouldn’t the most obvious way for people to protect their privacy while using FB if they cared and still wanted to use FB be not to proactively give them information? You don’t have to share everything I mentioned just to be involved in a group.
- barkerja 6mo agoUser's care about privacy when they understand the threat and impact. The issue is most user's don't understand this, especially when it comes to use of products like Meta where on the surface, everything appears harmless.
- duxup 6mo agoYeah I agree, I fear users don’t care “enough” about privacy that it will matter. :( Care at all sure, but enough to make a difference, the history of the web and recent computing history indicates otherwise.
- testing22321 6mo agoI see all these LLM posts about if a certain model can run locally on certain hardware and I don’t get it. What are you doing with these local models that run at x tokens/sec. Do you have the equivalent of ChatGPT running entirely locally? What do you do with it? Why? I honestly don’t understand the point or use case.
- samuel 6mo agoChat is certainly an option, but the real deal are agents, which have access to way more sensitive information.
- testing22321 6mo agoThanks. What do you do with such an agent? What is the use case?
- dec0dedab0de 6mo agomost of the llm tooling can handle different models. Ollama makes it easy to install and run different models locally. So you can configure aider or vscode or whatever you're using to connect to chatgpt to point to your local models instead. None of them are as good as the big hosted models, but you might be surprised at how capable they are. I like running things locally when I can, and I also like not worrying about accidentally burning through tokens. I think the future is multiple locally run models that call out to hosted models when necessary. I can imagine every device coming with a base model and using loras to learn about the users needs. With companies and maybe even households having their own shared models that do heavier lifting. while companies like openai and anhtropic continue to host the most powerful and expensive options.
- roboror 6mo agoWhat models have you found capable? I was recently recommended Qwen3 Coder Next and I did not find it very successful. I have a good amount of VRAM/RAM so would love to run something locally.
- 6mo ago
- jesse23 6mo agoYes so far do we have a working practice that, with a given local mode, any infra we could use, that provide a good practice that can leverage it for local task?
- thefourthchime 6mo agoMaybe some more distant future. For me, I'm still struggling with the hallucinations and screw-ups that the state-of-the-art models give me.
- mrinterweb 6mo agoI think two recent advances make your statement more true. The new Qwen 3.5 series has shown a relatively high intelligence density, and Google's new turboquant could result in dramatically smaller/efficient models without the normal quantization accuracy tradeoff. I would expect consumer inference ASIC chips will emerge when model developments start plateauing, and "baking" a highly capable and dense model to a chip makes economic sense.
- fauigerzigerk 6mo agoWho will be funding state of the art local models going forward? AI models are never done or good enough. They will have to be trained on new data and eventually with new model architectures. It will remain an expensive exercise. I could be wrong because I'm not following this too closely, but the open weights future of both Llama and Qwen looks tenuous to me. Yes, there are others, but I don't understand the business model.
- mgaunard 6mo agoThese local models are far behind the capabilities of latest Gemini Pro, Claude Opus or GPT. Why waste time with subpar AI?
- Lucasoato 6mo agoThey will eventually catch up, that’s the hope to avoid a techno feudalism in which too much power is in too few hands.
- abu_ameena 6mo agoYes, but you don’t always want the power/expense of these models for the task at hand. A hammer is good enough to push a nail inside a wall. Save the nail gun for when you are building a house.
- sbassi 6mo agoIt's a trade off.
- anon373839 6mo agoThey’re not far behind, unless you mean for “vibe coding”. And for probably 85% of queries that people use LLMs for, you can’t even really perceive the difference between frontier and local.
- sowbug 6mo agoI am concerned that local models will never benefit from the training on live requests that is surely improving cloud-only models. This might be the cost of privacy, and it might be worth paying, unless cloud models reach an inflection point that make local models archaic.
- port11 6mo agoThere’s been some success training models on top of differential privacy. I imagine that with live requests it would be quite challenging but not impossible, assuming you could somehow sanitize all sorts of private data that people throw at these prompts.
- throwawayq3423 6mo agoTechnologists make the same mistake over and over in thinking the better technology will win. vhs vs betamax, etc. Actual consumers not only don't care, they will not even be aware of the difference.