16 ms·

Local AI needs to be the norm

• deleted 5mo ago
  [deleted]
• sgt 5mo ago
  I guess Google got that memo!
• hypfer 5mo ago
  Same as local compute. Welcome back to 2014. Let us now continue yelling at the cloud.
• williamtrask 5mo ago
  I wonder if a popularization moment for local AI will ultimately be the pin-prick that pops the AI bubble. Like the deepseek or openclaw moments but bigger/next.
  • gdulli 5mo ago
    That's like wondering if enough people discovering local media streaming will disrupt commercial streaming services. It's not going to happen. Most people are not ambitious and will let themselves be controlled by the services of least resistance. And you can't take comfort in knowing that you, personally, will remain in control of your own computing. The majority will let the range and direction of their thoughts and output be determined by the will of the tech giant whose AI they adopt. And that will shape society.
    • williamtrask 5mo ago
      Yeah... probably right. I do hold out hope that this is mostly a timeframe thing. Like, the library, printing press, etc. all had their moments of centralization. But eventually they federated.
    • HDBaseT 5mo ago
      I like the analogy of streaming services vs local media streaming, although I don't think it holds up when looking at history. Streaming Services are getting worse and more expensive. I don't see a single report suggesting piracy is decreasing, it seemingly is only increasing now. When costs increase, quality decreases people look for alternatives. The advent of faster broadband enabled Napster and MP3 sharing. I think this could have a resurgence if the peices align correctly (a new bitorrent client, a new torrent site, something to break the status quo). How this related to AI, I don't know, although I wouldn't be set on the idea that we will never have local AI as the norm. There is a lot more movement in this space then there is for local streaming imo.
• Galanwe 5mo ago
  I would love for local inference to be possible, but from my experience, Kimi 2.6 is the only model that would be worth it, and its a $10k (M3 Ultra max spec'd - 30s TTFT so kind of slowish) to $30k (RTX6000/700GB+ DDR5) upfront, noise / power consumption aside.
  • mft_ 5mo ago
    You're maybe missing the article's point, which is to use local models appropriately: > “But Local Models Aren’t As Smart” > Correct. > But also so what? > Most app features don’t need a model that can write Shakespeare, explain quantum mechanics, and pass the bar exam. They need a model that can do one of these reliably: summarize, classify, extract, rewrite, or normalize. > And for those tasks, local models can be truly excellent.
    • Galanwe 5mo ago
      This is a bit naive IMHO... I have tried quite a bunch of local models, and the reality is that it's not just a matter of of "it's a small model that should be hostable easily". Its also a matter of whats your acceptable prefill TTFT and decode t/s. All the local models I used, on a _consumer grade_ server (32GB DDR5, AMD Ryzen) have been mostly unusable interactively (no use as coding agent decently possible), and even for things like classification, context size is immediatly an issue. I say that with 6m experience running various local models for classifying and summarizing my RSS feeds. Just offline summarizing ans tagging HN articles published on the front page barely make the queue sustainable and not growing continuously.
      • mft_ 5mo ago
        1) Again, I suspect you're missing the point of the article. The iPhone's on-device LLM is (apparently) ~3 Bn parameters - and runs well/fast enough to be used in the manner described. Of course, the iPhone has its GPU to leverage. 2) It's probably not the time/place to trouble-shoot your "consumer grade server" LLM experience, but if you're running on CPU (you don't mention a GPU) then yeah, your inference speed will be slow. 3) Counterpoint: my consumer-grade Macbook Pro (M1 Max, 64GB) runs Qwen3.6-35B-A3B fast enough to be very usable for regular interactive coding support. (And it would fly with smaller models performing simpler tasks.)
• jjordan 5mo ago
  It feels like we're one technological breakthrough away from all of these data centers going up to be deemed irrelevant.
  • i_love_retros 5mo ago
    What would that breakthrough be?
    • Waterluvian 5mo ago
      Magic math and computer science that allows us to get the same quality response for a fraction of the GPU.
      • deleted 5mo ago
        [deleted]
      • deleted 5mo ago
        [deleted]
      • toufka 5mo ago
        I mean, the most cutting edge of iPhones, iPads and MacBook Pros _today_ are quite capable of running in realtime today’s high-end local LLMs. If you project out that hardware just a couple of years, and the trained models out a couple of years, you end up in a place where it makes so much more sense to run them locally, for all sorts of latency, privacy, efficacy, and domain-specific reasons. Not all that different from the old terminal & mainframe->pc shifts. Finally - hardware has seemingly gotten out ahead of software that most folks use - watching YouTube, listening to music, playing a game or two. There was a time when playing an mp3 or watching a 4k video really taxed all but the nicest systems. Hardware fixed that problem, like it very well could this one.
        • sofixa 5mo ago
          > I mean, the most cutting edge of iPhones, iPads and MacBook Pros _today_ are quite capable of running in realtime today’s high-end local LLMs Definitely not the high end local LLMs. The small ones, yes, absolutely. > If you project out that hardware just a couple of years One of the biggest bottlenecks for LLMs is memory capacity and bandwidth. With the current glut for memory, it's unlikely we'll see lots of advancements in terms of average memory available or its bandwidth on regular (not super high end devices) in the coming years. Alternatively, it's possible we get dedicated SMLs for e.g. phone specific use cases, that are optimised and run well.
• timeattack 5mo ago
  My problem with LLMs (apart from philosophical aspects and economical impact) is that it would be unlikely for any of us to be able to train something functional locally (toy-like LLMs -- sure, but something really useful -- no). Apart from that it requires immense computing power, it also requires a dataset which is for the most part is obtained illegally.
  • cyanydeez 5mo ago
    That sounds like government. So your problem is mostly that you expect to have a collective social effort, but not enough to pay for it as a public good.
  • Ucalegon 5mo ago
    Depends on the domain. There are plenty of different use cases where the data needed for training is available for personal, or non-commercial, use. At that point, it does come down to compute/time to do the training, which if you are willing to wait, consumer grade hardware is perfectly capable of developing useful models.
  • kibwen 5mo ago
    This seems overly pessimistic. I may personally be of modest intelligence, but to acquire the intelligence that I do have, I did not need to train on every book ever written, every Wikipedia article ever written, every blog post ever written, every reference manual ever written, every line of code ever written, and so on. In fact, I didn't train on even 1% of those materials, or even 0.00000000001% of those. The texts themselves were demonstrably not a prerequisite for intelligence. At minimum, given that it only took me about 20 years of casual observation of my surroundings to approximate intelligence, this is proof positive that the only "dataset" you need is a bunch of sensors and the world around you. And yes, of course, the human brain does not start from zero; it had a few million years of evolution to produce a fertile plot for intelligence to take root. But that fundamental architecture is fairly generic, and does not at all seem predicated on any sort of specific training set. You could feasibly evolve it artificially.
    • 5mo ago
• revolvingthrow 5mo ago
  A local Answer Machine is the dream, especially when the internet is decaying and generally on its last legs, but the hardware requirements seem like a huge mountain to climb. Things are progressing tremendously - deepseek v4 flash is very good for what it is - but even that goes beyond any reasonable local setup, which imo is 128 GB ram + 16 GB vram. 4 ram slots on a consumer board craters ram speed, 256 gb macs are too expensive, and even then the inference is ungodly slow. On the other hand… v4 flash model is actual magic compared to what was available 2 years ago. If the rate of improvement stays as is, we’ll get a similar performance in a ~120B model in a year, which is viable (if expensive) for everyman hardware. Possibly you’ll be able to run its equivalent on a ~$1200 laptop by 2028, which for me-in-2020 would sound straight out of a scifi movie. A good harness that lets the model fetch data from other sources like a local wikipedia copy from kiwix could do a lot for factual knowledge, too; there’s only so much you can encode in the model itself, but even a cheapish (pre-curent prices) 2TB drive can hold an immense amount of LLM-accessible data. Big caveat: I don’t see local models for programming or generally demanding agentic tasks being worth it anytime soon. You likely want bleeding edge models for it, and speed is far more important. Chat at 20tok/s is fine; working on even a small codebase at 20tok/s, especially on a noticeably weaker model, is just a waste of time. Maybe it’s a PEBKAC but I have no idea how people make any meaningful use out of qwen 3.6.
  • zozbot234 5mo ago
    > and even then the inference is ungodly slow. This is the wrong way of putting it. Local inference with SOTA models is all about slowing down compute for the sake of fitting on bespoke repurposed hardware. You don't need to go fast if you have the whole machine to yourself 24/7. Cloud AI vendors can't match that kind of economics.
• agentifysh 5mo ago
  Until the hardware is economical and powerful enough, local AI that can compete with frontier models today is still far off. If we could even get something like GPT 5.5 running locally that would be quite useful.
• vegabook 5mo ago
  >> years ago I launched "The Brutalist Report" proceeds to brutalise the reader with an 88-point headline font.
• artursapek 5mo ago
  I'm someone who is trying to build a subscription-based business to cover underlying LLM costs, and very hopeful I can one day just sell a permanent license to the software instead with customers using local LLMs to power it.
• TheJCDenton 5mo ago
  For the mainstream audience, the sentiment around local ai today is the same that they had around open source a few decades ago. For a few products, some paid solutions were so much more advanced that open source were very often completely overlooked. Why bother ? And the like. Then we had captive SaaS and other plateforms and now it's obviously wrong for most of us. The dependency we have with anthropic and openai for coding for instance is insane. Most accept it because either they don't care, or they just hope chinese will never stop open weights. The business model of open weights is very new, include some power play between countries and labs, and move an absurd amount of money without any concrete oversight from most people. It's a very dangerous gamble. Today incredible value is available for nearly everyone. But it may stop without any warning, for reason outside our control.
  • oytis 5mo ago
    What is the business model of open weight AI? I don't think there is any. At best it can serve as an advertisement for the more advanced models you sell. The huge difference to open source is that you can't just train an LLM with free time and motivation. You need lots of data and a lot of compute. I sure want to be wrong on that, I definitely like the open-weight version of the future more
    • worldsayshi 5mo ago
      It should be feasible to crowd fund training runs right?
      • dmd 5mo ago
        A training run costs somewhere in the neighborhood of a billion dollars. That’s a thousand millions. How many crowdfunded projects do you know that have raised even one percent of that? Who’s going to be in charge of collecting that scale of money? Perhaps some sort of company formed for the benefit of humanity, which will promise to be a non-profit? Some sort of “Open” AI? Oh, wait.
        • 5mo ago
• shmerl 5mo ago
  Depending on some remote AI provider is a major lock-in pitfall. But it's exactly what those AI providers want you to do.
• cubefox 5mo ago
  Local AI is a bit like wind parks. Everyone is in favor, except if they are in your own backyard. There was recently a huge outcry when Chrome shipped a local 4 GB AI model: https://news.ycombinator.com/item?id=48019219 https://news.ycombinator.com/item?id=48019219 I have to conclude that people would like to have powerful local AI but it should at the same time only be a tiny model. In which case it wouldn't be powerful.
• barrkel 5mo ago
  Local models are extraordinarily expensive if you're not maximizing throughput, and you're not going to be maximizing it. Local models need to be resident in expensive RAM, the kind that has fat pipes to compute. And if you have a local app, how do you take a dependency on whatever random model is installed? Does it support your tool calling complexity? Does it have multimodal input? Does it support system messages in the middle of the conversation or not? Is it dumb enough to need reminders all the time? Spend enough time building against local models and you'll see they're jagged in performance. You need to tune context size, trade off system message complexity with progressive disclosure. You simply can't rely on intelligence. A bunch of work goes into the harness. Meanwhile, third party inference is getting the benefits of scale. You only need to rent a timeslice of memory and compute. It's consistent and everybody gets the same experience. And yes, it needs paying for, but the economics are just better.
  • bheadmaster 5mo ago
    > And if you have a local app, how do you take a dependency on whatever random model is installed? Why not ship your own model? In the age of Electron apps, 10GB+ apps are not unheard of.
    • _heimdall 5mo ago
      Personally I wouldn't want a couple dozen apps installed all with their own model. It seems easier to have industry specs that define a common interface for local models. I also assume the OS can, or would need to, be involved in proving the models. That may not be a good thing depending on your views of OS vendors, but sharing a single local model does seem more like an OS concern.
      • alex7o 5mo ago
        I mean the openai API is the industry standard for allowing apps to communicate with models, llama-server has it, oMLX has it, ollama has it, vLLM has it, lmstudio as well. I don't think this is such a hard thing to do, but it requires people to set it up.
• vb-8448 5mo ago
  > Use cloud models only when they’re genuinely necessary. The problem is that it's much easier to use the SOTA models (especially if they are subsidized) instead of spending time fixing the knobs with the local one. I just realized this with coding agents, yeah, you probably shouldn't always use latest version at xhigh, but you will end doing it because you do the job in less time, with less "effort" and basically at the same price. I guess we'll see a real effort for local AI only when major vendors will start billing based on actual token usage.
  • Analemma_ 5mo ago
    I'm also just not seeing good performance from local models. Every time a thread about LLMs comes up, there are tons of people in the comments insisting that they're getting just as good results from the latest DeepSeek/qwen/whatever as with Opus, and that just hasn't been my experience at all: open-source models just fall over completely compared to Claude when asked to do anything remotely complicated. I have a sneaking suspicion this is kinda like the situation with Linux in the 90s, where it kinda worked but it reeeeeally wasn't ready for the home user, but you had a lot of people who would insist to your face everything was fine, mostly for ideological reasons.
    • kgeist 5mo ago
      It depends a lot on how you run those models. I think a lot of disagreement is because of that. A lot of people run local models with incredibly small context windows (makes an agentic LLM circle in loops), use very small quants (like 4 bit => huge degradation), don't set the recommended parameters (like top-p/temperature), or download GGUFs with broken chat templates. And then they claim model X is bad :) I'm currently running both Sonnet 4.6 and Qwen 3.6-27b on the same codebase (via OpenCode, the parameters were carefully tuned to have a good quality/context size ratio), and on this project, they both struggle with complex non-trivial tasks, and both work flawlessly otherwise. Sonnet 4.6 understands the intent better if my task is ambiguously formulated, but otherwise the gap is pretty small for coding under a harness.
    • bilbo0s 5mo ago
      This. I’ve begun to suspect that most people are probably running different hardware. Sure, you run the latest deep flash on your brand new M5 128G maybe you get acceptable performance? But honestly, how many people have an extra $9000 laying around these days? Right now, running with acceptable performance is kind of a luxury. I wish the people who always say - “This is great!” - would realize that not everyone has their hardware.
• holtkam2 5mo ago
  I wish I could upvote this twice. We (devs) really REALLY need to consider on-device compute before going to the cloud for LLM inference.
• mattlondon 5mo ago
  Yet there is another post a few rows down where people are losing their shit that Chrome has a local LLM model that uses a couple of GB of space for local-inference. Damned if they do, damned if they don't.
  • dlcarrier 5mo ago
    Maybe don't use gigabytes of bandwidth and storage space, without asking.
    • hparadiz 5mo ago
      Easy. Stop using Chrome.
  • aabhay 5mo ago
    This is a weird take. If its not opt in or you’re shoe horning it into a browser, then that sucks. Nobody is getting enraged that an app for running local LLMs downloads data to do so.
    • avadodin 5mo ago
      Although you can opt out and even disable the download feature when you build them in some cases, most of the local LLM tools are too download–happy by default.
  • ekjhgkejhgk 5mo ago
    You don't understand the difference between "I run a local LLM because I chose to" vs "The browser chose to run a local LLM and I have no say"? You don't understand? Not to mention that the LLM that I choose to run requires a monster machine and is infinitely more capable than whatever google chose to put on their browser? I mean, none of this affects me because I don't use chrome, obviously, but you don't see the difference? Bewildering.
    • StilesCrisis 5mo ago
      Did you opt into WebGPU? QUIC? Canvas 2D? Brotli? Browsers don't work that way.
• eyk19 5mo ago
  Apple stock is going to skyrocket
  • baal80spam 5mo ago
    Maybe. What about NVDA?
• dana321 5mo ago
  "NO AI" needs to be the norm, we should be working on better ways of sharing information and better documentation instead of fighting with computers for substandard results.
• wilg 5mo ago
  Two issues - 1. Local models are likely to be more power-expensive to run (per-"unit-of-intelligence") than remote models, due to datacenter economies of scale. People do not like to engage with this point, but if you have environmental concerns about AI, this is a pretty important one. 2. Using dumb models for simple tasks seems like a good idea, but it ends up being pretty clear pretty quick that you just want the smartest model you can afford for absolutely every task.
  • manc_lad 5mo ago
    I think using the best model for every tasks makes sense when these models are subsidised. when the prices go up (assuming they do) this could trigger a more varied approach. assuming the model doesn't self select for you.
• msteffen 5mo ago
  > One of the current trends in modern software is for developers to slap an API call to OpenAI or Anthropic for features within their app. Well there’s your problem, control needs to go the other way. If you want your app to be AI-enabled, you need to make it easy for AI to control your app. Have you used OpenClaw? It’s awesome!
• Animats 5mo ago
  Question: for software development, how much of an AI do you need for local development? Can it be run locally? Can someone train something that knows a lot about software but lacks comprehensive coverage of history, politics, and popular culture?
  • deleted 5mo ago
    [deleted]
  • dd8601fn 5mo ago
    The ones that are good for more than elaborate auto-complete are pretty hefty, but it can be done. They’re still not Opus behind claude code.
  • mrkeen 5mo ago
    This is a good snapshot of things: https://news.ycombinator.com/item?id=48050751 https://news.ycombinator.com/item?id=48050751 A specialist handrolls a cut-down framework to power a 1 or 2 bit quantised version of a cut-down sort-of-frontier model. It can be yours if you have 128GB or 256GB of RAM.
• holoduke 5mo ago
  We need computers with 128gb or maybe even 192gb of memory before local use make sense. From my own experience 32b LLMs are the absolute minimum for proper tool use and decent output quality. But for local ai you want also vision models and maybe even various LLMs. Plus some memory for the system of course. On my 36gb M3 the 24b Gemma model is nice. But the entire system gets allocated for that thing.
• pronik 5mo ago
  They will be, and that moment is not that far off. We've got the progression in place already: first, large data centers could have performant LLMs, we are now firmly in "a bunch of servers with a couple of H100s each" territory, slowly going into "128 GB VRAM on a MacBook Pro or a Strix Halo". Within the next year, the pattern of "expensive remote LLM for planning, local slow-but-faster-than-human LLM for execution" will become the norm for companies, slowly moving to "using local LLM for everything is good enough". And then we'll have the equilibrium we already have with the "classic cloud": you either self-host or pay for flexibility and speed. The question will be: how much of the current compute capacity craze will local hosting give the kiss of death to and what that means for the market.
  • dakolli 5mo ago
    This is simply delusional, It cost 20-30k a month to run Kimi 2.6. The tokens are sold for $3 per mm. To sell tokens profitably you'd need to be able to run inference at 150 tokens per second for less than $1,000 USD a month. I don't think people realize how expensive it is to host decently capable models and how much their use of capable models is subsidized. You can only squeeze so many parameters on consumer grade hardware(that's actually affordable, two 4090s is not consumer grade and neither is 128gb macbooks, this is incredibly expensive for the average person, and the models you can still run are not "good enough" they are still essentially useless). People are betting their competency on a future where billionaires are forever generous, subsidizing inference at a 10-1 20-1 loss ratio. Guess what, that WILL end and probably soon. This idea that companies can afford to give you access to 2mm in GPUs for 5 hours a day at a rate of $200.00 a month is simply unsustainable. Right now they are trying to get you hooked, DON'T FALL FOR IT. Study, work hard, sweat and you'll reap the benefits. The guy making handmade watches, one a month in Switzerland makes a whole lot more than the guy running a manufacturing line make 50k in China. Just write your own fkin code people. Don't bet your future on having access to some billionaire's thinking machine. Intelligence, knowledge and competency isn't fungible, the llm hype is a lie to convince you that it is.
    • hparadiz 5mo ago
      Posts like this are so funny to me. I'm staring at a mountain of old hardware right now that cost about $20k ten years ago. I have to pay someone now to come haul it away. What makes you think the current new hardware won't end up with the same fate. > Just write your own fkin code people Bro is nostalgic for googling random stack overflow threads for 10 days to figure out a bug the agent fixes in an hour.
• scriptsmith 5mo ago
  I've got some demos of what the new Prompt API in Chrome that uses a local model can do: https://adsm.dev/posts/prompt-api/#what-could-you-build-with-it https://adsm.dev/posts/prompt-api/#what-could-you-build-with... As OP says, it shines in constrained environments where the model is transforming user-owned data. Definitely less useful for anything more open-ended.
  • 2ndorderthought 5mo ago
    Yea I do not recommend treating chromes prompt API as a good example of local LLMs. It's fine and stuff but it's really weak. 8b models from a year ago are better in some ways. And a lot of the recent model drops are meaningfully better.
    • scriptsmith 5mo ago
      It's based on a Gemma 3n model, and yeah it's not the best. But if you have a use case that needs constrained JSON output for example, it's pretty neat. Maybe it would do better with the new Gemma 4 models, which the Chrome devs have been hinting at moving to. And why the API doesn't let you introspect / pick the model, I'm still not sure.
  • dakolli 5mo ago
    So you're running an llm to do data transformation that deterministic processes would be much better suited for and running 1,000 watt power supply to do so. Wild.
  • robot-wrangler 5mo ago
    > I've got some demos of what the new Prompt API can do: > Use surrounding context to rewrite your ad copy: Yup, that's the plan. No local model, no webpage; more, better and cheaper adtech extortion/surveillance for vendors while everyone else pays for the juice and hardware degradation.
• refulgentis 5mo ago
  The shitty thing here is, either everyone's shipping 800 MB at least with their binary, or, you have to rely on the platform vendor anyway. I'm hoping there's enough external pressure that the OS vendors turn it more into a repository than a blessed-model-garden.
  • wrxd 5mo ago
    To be fair the author of the post is using the model Apple provides with the OS so it doesn't have any extra binary size
• daishi55 5mo ago
  > We are building applications that stop working the moment the server crashes or a credit card expires Isn’t this true of any application that accesses anything not running on your computer? This is just describing what it means to add an API call to your app. Nothing to do with AI (?)
  • simonkagedal 5mo ago
    Furthermore, for the example given, it would have made a lot of sense to me to generate those article summaries on the backend. Once and for all, no need to burden each client device (which are going to need to download the content anyway), no need to tie yourself to a specific provider (Apple in this case), can have the same experience everywhere. Of course, the backend could use a local (to itself) model. Not saying it’s _wrong_ either – maybe it doesn’t use a backend of its own (the client downloads content directly from some predefined set of sites), maybe there is functionality to adjust how the summaries work that benefit from doing it on device, etc. Just doesn’t convince me that ”local AI should be the norm”.
• rduffyuk 5mo ago
  agree with the article but the limitation for local llm usefulness is the limited scope from my experiments. eventually context heavy data pipelines require larger models which consumer hardware can't deal with yet. the local model for summary on a page like you describe could be done via code as well, i've found using an llm isn't always the right choice. for example i use ner tagging in my md docs for better indexing and llm search capabilities. this is purely code based and not via an llm. tried with an llm and the results were a lot worse. augmenting tools to make the llm produce better outputs gives better results.
• hackyhacky 5mo ago
  I would like a standardized API for local AI to exist outside of the Apple ecosystem. The Prompt API is Chrome is halfway there. * What is the answer to local AI for native apps on Windows? * What is the answer to local AI for Linux? This is a big opportunity for Linux, given the high quality of open-weight models. I hope some answer emerges before designs fracture and we get a dozen mutually incompatible answers.
  • franze 5mo ago
    i researched that question for apfel https://github.com/Arthur-Ficial/apfel https://github.com/Arthur-Ficial/apfel and standardized API is openai api so thats what i went with
    • hackyhacky 5mo ago
      OpenAI's API is not local AI.
      • zozbot234 5mo ago
        Most local AI servers expose that API.
  • teravor 5mo ago
    > What is the answer to local AI for Linux? run an ai api endpoint on a unix domain socket
• krupan 5mo ago
  If you don't need a lot of smarts, do you even need an LLM? Aren't older machine learning techniques just as good, or like, you know, old-school algorithms?
  • a96 5mo ago
    Yes. For essentially any problem where a complete solution exists that doesn't use an LLM, it will beat any solution that does in size, speed, energy use, reliability and everything else. Naturally, it's actually complicated. But LLM is a considerable weight and risk. Maybe it's worth involving and maybe not.
• krupan 5mo ago
  Here I was hoping that this was some plea for us to get away from proprietary solutions that we have no control over and go back to open source, but no, not that at all.
• debpalash 5mo ago
  [dead]
• wrxd 5mo ago
  The example in the post confirms my theory that for local models to succeed they need to be "good enough", not big enough that they can compete with frontier models. They need to be able to do a small task well and they need to be able to run reasonably on consumer-class devices. Even better if they can run on mobile phones. In my experiments with local LLMs I noticed that while increasing the size of the model is nice the real thing that turns a barely useless model into something useful is the ability to use tools. Giving my models the ability to search the web and fetch web pages did way more to solve hallucinations than getting a bigger model. And it doesn't have a training cutoff. Sure, the bigger model is probably better at using tools but I often find the smaller models to be good enough.
  • Gigachad 5mo ago
    Will there even be a web to search in the future? These days public access blogs are dying and being replaced with hallucinated AI websites. Sites with original research like Reddit and YouTube are being locked up to prevent 3rd party indexing. Knowledge and clean data sets are becoming increasingly valuable, and free community knowledge is drying up. The next big programming language won’t have years of Stack Overflow posts to train on. Maybe we will see some kind of licensing deals where owners of good datasets charge you a fee to let your AI search them.
• qwertmax 5mo ago
  [flagged]
• RataNova 5mo ago
  I mostly agree, though I think local AI will need better UX around failure modes. Cloud models are often used not just because developers are lazy, but because they are more capable and easier to support consistently across devices.
• Guillaume86 5mo ago
  I think we should separate the private AI discussion from the local AI discussion. The pragmatic choice to run big LLMs is one/several big servers online, but that doesn't mean private companies should be the only ones to run them. A self hosted inference solution that offer good tenant isolation guarantees (ideally zero trust) and is easy enough to deploy and maintain (think Plex for AI) would be my choice for privacy. Now to be honest I have done zero research about this and have zero idea how feasible that is, maybe it already exists and there's some discord servers I should join? Edit: I don't need to mention it here but what's incredible is that open models are in the ballpark of the best commercial models so supposedly, the hardest part by far is already solved.
  • FrasiertheLion 5mo ago
    Another option is verifiably private inference with open source models running inside secure enclaves on the cloud (using NVIDIA confidential computing), and the enclave code is open source and verified via remote attestation upon connection, cryptographically proving that the inference provider cannot see any data. Tinfoil: https://tinfoil.sh/ https://tinfoil.sh/ is a good example of this (disclaimer: i'm the cofounder). You can read more about how this works here: https://docs.tinfoil.sh/verification/verification-in-tinfoil https://docs.tinfoil.sh/verification/verification-in-tinfoil >that open models are in the ballpark of the best commercial models This is basically true for certain tasks. As an example, chat interfaces are not well poised to take advantage of higher model intelligence than what the best open source models already provide. But coding harnesses still benefit from greater model intelligence and even more so, the reinforcement learning that tightly interlinks the provider's coding harness (claude-code, codex) with the model's tool calling interfaces is another reason for discrepancy in effectiveness even when controlled for model intelligence. The opencode founder (open source coding harness that supports different model providers) was recently complaining about the challenges making the harness work well with different providers: https://x.com/thdxr/status/2053290393727324313 https://x.com/thdxr/status/2053290393727324313
• ksec 5mo ago
  While I agree that would be the goal, we are too early for that. Just like how speech recognition used to require many server in a Datacenter to process and you send your data over. It is now completely on devices. We are at least 5 years away from that. And DRAM needs a substantial breakthrough in cost reduction.
• 1a527dd5 5mo ago
  Consumer/private needs to be local. Work? I don't want it local at all. I want it all cloud agent.
• throwaway613746 5mo ago
  [dead]
• robot-wrangler 5mo ago
  Entrenched interests are going to do everything to stop local, but there's at least a few technical reasons to believe small and specialized models could be the norm eventually. If that does happen, local will follow. TFA is focused on whether big models are necessary for what users want. There's some evidence they may never actually be reliable enough unless a) mechanistic interpretation matures far enough or b) our multi-agent systems all become multi-model. For (a), advancement in MI might fix problems with big models, but would also mean we can maybe get unified representations, and just slice and dice the useful stuff out of huge models, getting only what we need without the junk. Ability to isolate problems won't really come without bringing the ability to isolate functional subsystems. Only want logic? Only vision? Just cut it out of the big monster and enjoy reduced costs and surface area for problems. For (b), just look at stuff like the evil vector, or the category of hallucinations specific to tool-use. Without a complete solution for helpful/honest/harmless alignment, it seems likely that creativity and rigor (and many other things) are fundamentally at odds. If you start to need many models for everything anyway, why do we need the huge expensive do-everything ones? So specialization also becomes a pressure to shrink everything towards minimal reliable experts
• jmyeet 5mo ago
  I've been looking into options for this and we are getting close. There are two main constraints: memory and memory bandwidth. NVidia segments the market by limiting the amount of memory on GPUs. It currently tops out at 32GB (on a 5090) but it has excellent memory bandwidth (~1.8TB/s). If you want more than the you need to buy an RTX Pro (eg RTX 6000 Pro w/ 96GB for ~$10K) or you get into high high end solutions like H100, H200, etc that have significantly more memory and even higher bandwidth on HBM memory (eg 3.2TB/s+). NVidia has released the DGX Spark w/ 128GB of memory for ~$4k. The problem is the memory bandwidth. It's only 273GB/s, which is less than the M5 Pro (307GB/s) but more than the M5. You can buy a 16" Macbook Pro with an M5 Max and 128GB of memory for $6k and it has a bandwidth of 614GB/s. So the DGX Spark is a joke, really. In case it wasn't clear, Apple is interesting in this space because it has a shared memory architecture so the GPU can use all the memory. Many, myself include, expect there to be no refresh to the 5000 series consumer GPUs this year, which would otherwise happen based on product cycles. So no 5080 Super, for example. And I wouldn't expect a 6090 before 2028 realistically. One thing Apple hasn't done yet is release the M5 Mac Studios, which are widely expected in Q3 this year. They are interesting because, for example, the M3 Ultra has a memory bandwidth of 819GB/s and previously had a max spec of 512GB but that got discontinued (and the 256GB version also got discontinued more recently). So many expect an M5 Max Mac Studio with 1TB/s+ bandwidth and specs up to 256GB or 512GB, probably for ~$10k later this year. You really have to use this hardware almost 24x7 for it to be economical because otherwise H100 computer hours are probably cheaper. But what happens when the next generation of GPUs comes out to the trillions in AI DC investment? It's going to halve its value. That's over $1 trillion in capex that will disappear overnight, effectively. I think Apple is the dark horse here because they have no interest in NVidia's psuedo-monopoly. I'm just waiting for them to realize it. Now CUDA is an issue here still but I think as time goes on it's going to be less of an issue. Memory is still a huge constraint both in terms of price and just general supply because NVidia can justify paying way more for it than you can, probably. It's still sad to see that 128GB (2x64GB) DDR5 kits are almost $2k now and werre $400 a year ago. Expect that to continue until this bubble pops (which IMHO it will) and we're likely in a global recession. So the other issue is models. OpenAI and Anthropic are built on proprietary models. Their entire valuation depends on this moat. I don't think this last so both companies are doomed because open source models are going to be sufficiently good. We can already do some reasonably cool stuff on local hardware that isn't that expensive and even more so once you get to $5-10k hardware. That's going to be so much better in 2 years that I'm hesitant to spend any amount of money now. Plus the code for running these things is getting better. Just in the last month there have been huge speed ups in local LLMs with MTP.
• hyfgfh 5mo ago
  Local LLMs is the only thing viable and probably the only thing it will remain once the hype dies down. A smaller cheaper local model can delivery most the value for coding, while we still use some services for code review and security compliance. Once the VC money runs out and they start to charge the real price, the C-level will have to impose budges or limits. The current pissing contest over who can expend the most tokens is both ridiculous and shortsighted
• prometheus1992 5mo ago
  Agreed, but the way ram prices are going, I don't think we would be able to afford hardware that can run any useful model.
• ChoGGi 5mo ago
  Who can afford local AI?
  • m463 5mo ago
    Who can afford to backup their own photos? who can afford a house?
• Salgat 5mo ago
  Local models are much less energy efficient right?
  • HDBaseT 5mo ago
    It's a good question, although I think hard to quantify. If you are simply measuring Watt Cost per Token, you are missing the mark drastically. You have to measure quality output per Watt. It sounds reasonably difficult to benchmark this, maybe I'm wrong though.
• lbrauer 5mo ago
  [flagged]
• anArbitraryOne 5mo ago
  Just let me turn it off to preserve battery life
• FrasiertheLion 5mo ago
  Overall I'm bullish on standardized local APIs that ship with the browser or platform. Far more tractable than expecting end users to stand up their own local model instances, though r/LocalLLaMA is a fantastic community to follow if you want to go that route. A useful framing over “local vs cloud AI” can be split along two axes: does the task touch private data, and does it need frontier intelligence? You can use frontier models for developing the software (doesn’t touch data), but open-source models running locally for ops: maintenance, debugging and monitoring (touches data). If you need to fall back to frontier intelligence at some point for a particularly hard to resolve problem, you can still rely on local models for pre-transforming and filtering input in a way that's privacy-preserving or satisfies some constraint before it’s sent off to the cloud for processing. OpenAI's privacy filter is a good example of a model that can be used to mask PII and secrets and that can run locally: https://openai.com/index/introducing-openai-privacy-filter/ https://openai.com/index/introducing-openai-privacy-filter/, before sending any data externally for processing. Another framing for local vs frontier closed which the article mentions is whether the task saturates model capability. With certain tasks like PDF processing or voice or summarization, adding more intelligence isn't necessarily useful. Arguably we've approached that point for chat interfaces already with frontier open-source models. But for coding and ops through well structured tool use inside a coding capable harness, we're still a ways away. Tangentially, a contrarian take here is that AI can actually enable more privacy preserving software if you’re so inclined. You can just build personalized software and it lowers the barrier to entry and the effort required to self host. SaaS complexity often comes from scaling and supporting features for all types of customers, and if you're building software for personal use, you don't need all that additional complexity. Additionally, foundational and infra software that is harder to vibecode with AI is often already open source.
  • hiroto_lemon 5mo ago
    [flagged]
• everlier 5mo ago
  There was never a better time to run LLMs locally. It's just a few commands from zero till a fully working LLM homelab. ``` harbor pull unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_XL # Open WebUI -> llama.cpp + SearXNG for Web RAG + OpenTerminal as sandbox harbor up searxng webui llamacpp openterminal ``` That's it, it's already better than Claude's or ChatGPT's app.
• manlymuppet 5mo ago
  People are trying to “make the best software”, though. I think the Quixotic accelerationists of AI are more or less a vocal minority of the people who make software, and the choice of online APIs over local systems is largely a choice made for users, rather than developer’s laziness. You can do more and better with private AI today than with local models. There is no getting around that. Even if local AIs get better, being on the cutting edge of LLM performance is often a very worthy investment. Most people won’t settle for a product if it’s not the very best and incredibly convenient. That’s a high bar, and local AI often doesn’t meet those standards. HN’s insistence on treating all users like they are open-source, privacy-first, self-hosted Linux fanatics is painfully corny.
  • jdub 5mo ago
    > Most people won’t settle for a product if it’s not the very best and incredibly convenient. ... uh?
    • manlymuppet 5mo ago
      That is, excluding Microsoft users.
• TechSquidTV 5mo ago
  Local AI will catch up. Unless we can't get our hands on hardware anymore, which is a legitimate concern I have.
• deivid 5mo ago
  Sounds great, but if you din't cave to apple/google (eg: graphene, lineage), models are not built-in. Every app needs to ship their own models, and they are not tiny. Is there a solution for this? I'm currently just making users download onnx models if they want a feature, but it's not smooth UX
• mercurialsolo 5mo ago
  Not your weights not your brain. Owning your own action and decision model is super important as these models emulate more of our decisions, thinking and learning. Built claudectl - a local brain for coding agents https://github.com/mercurialsolo/claudectl https://github.com/mercurialsolo/claudectl
• rarisma 5mo ago
  I think with turbo quant forks eventually being merged, its becoming more feasible on mid tier consumer h/w Dont quite think its ready yet.
• Amber-chen 5mo ago
  [flagged]
• antidamage 5mo ago
  The roadblock to this is you seem to have to build it yourself. I've noted that none of the current cloud models are very good at building a replacement for themselves, and there's significant work that needs to be done to make a local LLM reliable in any way. I haven't found a single standalone package that makes setting them up easy. Sure, I can run Hermes Agent and a model, but getting the self-reflection loop in and all of the other stuff the need to actually be good? I'm still at it, trying to get anything to work reliably and factually.
  • DonsDiscountGas 5mo ago
    Could be an opportunity for a business? Except nobody ever wants to pay for software
• shouvik12 5mo ago
  [flagged]
• 0xbadcafebee 5mo ago
  Here's some things you can do right now with local models on a consumer device: - text-to-speech - speech-to-text - dictionary - encyclopedia - help troubleshooting errors - generate common recipes and nutritional facts - proofread emails, blog posts - search a large trove of documents, find information, summarize it (RAG) - manipulate your terminal/browser/etc - analyze a picture or video - generate a picture or video - generate PDFs, documents, etc (code exec) - simple programming - financial analysis/planning - math and science analysis - find simple first aid/medical information - "rubber ducking" but the duck talks back A quarter of those don't need more than a gig of RAM, the rest benefit from more RAM. Technically you don't even need a GPU, it just makes it faster. I do half that stuff on my laptop with local models every day. That said, it really doesn't need to be local. I like the idea that I can do all that stuff offline if I'm traveling, but I usually have cell service, and the total tokens is pretty cheap (like $2/month for all my non-coding AI use).
  • satvikpendem 5mo ago
    Please add double new lines as your formatting for the bullet point list makes it all one paragraph.
    • fennecfoxy 5mo ago
      Tbf I've always hated that about HN formatting as it's not very clear at all that that's how it works. If there's a newline in my comment, why not retain it? Whyyyyy?!
      • xigoi 5mo ago
        Because of the 6 people who write HN comments in Vim with hard wrapping turned on.
        • AlienRobot 5mo ago
          That isn't really a problem because the comment should just be posted hard- wrapped. It looks better that way.
• eldenring 5mo ago
  This article makes 0 sense. Its not up to billing or computer systems or ease of use or anything else that matters. The question is will the scaling laws, which in the asymptote are likely the laws of physics, hold up in converting energy to smarter models. Its not really up to anyone, the labs or developers, to choose if local or remote models will be the norm.
• nate 5mo ago
  I've been fooling with the Apple Foundation model for AlliHat, so you can chat with it from a Safari sidebar instead of just Claude. It's passible for some basic things like summarizing a page. But it really reminds me of Claude from like 3 years ago. I was trying to get it to generate synonyms for me and it would only generate about 10 with some duplicates. And when I asked for more, it said it would be a waste of resources to generate more. It has some kind of "act responsible" thing that Claude seemed to have. I also asked it to help me come up with synonyms for the game Pimantle, and it decided Pimantle was related to the adult industry and no matter how many times I said "it's just a game" or "I think you've misunderstood", it was stuck on not helping me with anything related to adult websites. And recommended I should play Wordle instead. All of this being said, it seems Claude gave up this "constitution" it used to train on? I remember trying to get it to help me code some video editing tools, and it was convinced I was pirating videos and so wouldn't help me anymore in that session.
• try-working 5mo ago
  I'm building a protocol and router runtime for hybrid local/cloud AI. The goal is that you would assign roles to models based on tasks, capabilities and observed performance. The router would then take care of model selection in the background. It's tricky though. Probably have another two weeks before I can release the runtime. I have a preview up at https://role-model.dev/ https://role-model.dev/ You can follow me on Twitter if you want updates (see profile)
• supermdguy 5mo ago
  Interesting to see this after the recent post about Chrome’s on-device model using up 4gb of storage, which frustrated a lot of people [1]. I agree local models are great, and it’s cool that Apple has models built in now. But I feel like it basically has to be an OS level feature or users are going to get upset. I’d certainly rather have a small utility call out to OpenAI than download its own model. [1]: https://news.ycombinator.com/item?id=48019219 https://news.ycombinator.com/item?id=48019219
  • appreciatorBus 5mo ago
    The way I interpret the drama over the Chrome model is that for a large chunk of users, perhaps the majority, Chrome is the OS, and this 4GB model will be their OS Level feature for local AI.
• adamtaylor_13 5mo ago
  Cool, well let me know when Opus 4.5 level performance is available locally, at speeds that serve everyday use, and 100% I'm right there with you. Until then, I'm going to keep sending my JSON to the server farm in Virginia because it's the only place that can serve me a model that actually works for my uses.
  • binyu 5mo ago
    DeepSeek V4 with 1 million token context window is pretty powerful, although still not there. There's hope that Opus 4.5 level performance locally is not that far away.
    • tuananh 5mo ago
      From what I read, ds v4 is very close with opus 4.6 performance.
      • DeathArrow 5mo ago
        The full model is, not the quantized versions.
        • tuananh 5mo ago
          yeah that goes without saying. how can openweight, quantized version beat SOTA :)
          • array_key_first 5mo ago
            Well it depends on the task. For agentic coding, more is more, but for tasks that normal consumers use them for there really is a ceiling. OCR, text to speech, that type of thing doesn't really improve when going to a SOTA model, so you'd just be wasting your money. I think local LLMs have more value than software engineers give them credit for.
            • 5mo ago
• tuananh 5mo ago
  local llm doesn't need to match SOTA performance in order to be useful.
• j45 5mo ago
  It’s easier to say 32 gb ram needs to be the norm to start getting movement on this
• DoctorOetker 5mo ago
  One advantage of local AI is continual learning. When I say 'moat' I don't mean moat specific to a company vis-a-vis other companies, but 'moat' specific to the set of inference providers vis-a-vis self-hosted local inference. The moat consists primarily of being able to batch inference requests. If we pretend people weren't interested in long context-lengths, there would be a moat for inference providers. who can batch many requests so that streaming the model weights (regardless if from system RAM to GPU RAM; or from GPU RAM to GPU cache SRAM) can be amortized over multiple requests. However people do want longer memory than the native context length. One approach is continual learning (basically continue training by using the past conversation as extra corpus material; interspersed with training on continuations from the frozen model, so it doesn't drift or catastrophically forget knowledge / politeness / ...). However this is very expensive for inference providers, since they would have to multiply model weight storage with the number of users U=N. For a single user the memory cost of continual learning is much less since they only need to support a single user, and are returned some of the memory cost through elimination of KV-caches, and returned higher quality answers compared to subquadratic approximations of quadratic attention. An advantage of continual learning is that the conversation / code base / context is continuously rebaked into model weights, and so doesn't need KV caches! It doesn't need imperfect approximations to quadratic attention, it attends through working knowledge being updated. Nothing prevents local LLM users from implementing this and benefiting from the dropped requirements of KV caches and enjoying true quadratic attention implicitly over the whole codebase, or many overlapping projects indeed. The only remaining moat of inference providers vis-a-vis continual learning local LLM's is the batching advantage, plus the gradient update costs for continual learning minus the KV storage and compute costs, minus the performance loss due to inexact approximations to quadratic attention. This points towards a stronger incentive for local hosting than currently realized (none of the popular local LLM tools currently support continual learning, once this genie is out of the bottle it will be a permanent decrease of the inference provider moat, the cost of which can't be expressed merely in hardware or energy costs, since it is difficult to quantify the financial loss of inexact approximations to quadratic attention, the financial loss due to limited effective context length and the concomitant loss in quality of the result)
  • DrScientist 5mo ago
• knlam 5mo ago
  you know what is the hard part about local ai? Supporting it cross platform. The OP get it easy by playing in Apple ecosystem but when you need to support local AI to both iOS/Android the approach is completely different. Even get the users to download the smallest models can be a challenge
• imnes 5mo ago
  I'm going through a similar exercise right now in an app I'm building. No server dependencies, for features that have traditionally used server side APIs, moving those capabilities onto the device. And also utilizing the on-board AI features provided by Android and iOS. So far it's been a very positive experience, and the capabilities provided on these devices have been more than capable for my needs. Working on providing apps that don't have ongoing operation costs of running server side infrastructure, so I can offer them as "pay once, run it forever" instead of ongoing subscription costs for the user.
• diwank 5mo ago
  in order for us to get there, i think we need a standardized api at the os layer for local models so that the os could optimize, batch and safely allocate resources. something like an analog of chrome's local model "prompt" api but provided and managed by the os itself. the user can choose which model they want to primarily use and so on but all of the heavy lifting and continuous batching is done automatically by the os
• karmasimida 5mo ago
  How? Memory price is sky high, that is the choke hold the monopoly will not let go
• manyatoms 5mo ago
  It just depends how quickly models become "good enough" that we don't care about SOTA
  • julianlam 5mo ago
    Arguably, some of the things HN readers ask for can be capably completed by a local open weight model for free.
• duchenne 5mo ago
  Cloud models can use batch processing which is significantly more efficient. A local model has basically a batch of one which takes as much time to process as a batch of 100 because the gpu is memory bound and spend most of its time loading the model from vram to the gpu cache while the gpu cores are idle. With a batch of 100 the model loading time and compute time are roughly similar. So local Models have a first 100x lower efficiency. Secondly, local models are idle most of the time waiting for the user to write a prompt, so the efficiency gap is probably more around 1000x.
  • r0b05 5mo ago
    It's an interesting point but local gpu efficiency is not something I think about when I'm being rate limited or when my subscription costs keep rising.
    • fleventynine 5mo ago
      I think folks in this thread are underestimating how expensive it is to serve a SoTA model at 100 tokens a second. In addition to the $500k in capital costs, you also have significant electricity costs. This stuff is expensive because supply is much lower than demand. If everyone was to run their own hardware with a batch size of 1, we'd have 100x more demand for inference hardware and electricity than we do now, and people would be even more frustrated. Efficiency is everything, and we need all the economies of scale we can get to meet demand.
      • zozbot234 5mo ago
        But that's why you shouldn't expect local models to provide quick real-time answers, at least not with the same smarts as SOTA models running in the cloud. Slow batched inference (if possible - RAM capacity can obviously be a challenge with typical models and end-user hardware) can be a lot more effective.
        • fleventynine 5mo ago
          My point is that it is WAY more efficient if we put the world's DRAM supply into a shared inference pool instead of stranding it in local machines where it won't have as high of batch size or utilization. The cost of not being efficient is even higher DRAM costs than we have now, given supply and demand.
• haltonlabsops 5mo ago
  [flagged]
• gkcnlr 5mo ago
  It seems like everybody is focused on "LLM"s, a.k.a Large Language Models. One interesting addition to that is fine-tuned- small parameter, distilled, context-dependent small language models that: 1- Do a particular task with great capability (due to its constrained, limited scope) 2- Do it in such a way, it integrates gracefully in your workflow without ever requiring you to know you are using an LM. There is a difference between outsourcing your workflow to AI and actually utilizing it. Check this: https://www.distillabs.ai/blog/we-benchmarked-12-small-language-models-across-8-tasks-to-find-the-best-base-model-for-fine-tuning/ https://www.distillabs.ai/blog/we-benchmarked-12-small-langu...
  • fennecfoxy 5mo ago
    Eh I think the small model thing is kind of a no-go. Reason being is that many workloads for AI are dynamically mixed, where training from multiple subjects comes into play and you just can't know exactly what mix will be required for each task ahead of time. I was hoping loras would do this for us as well but they don't really seem to have worked out for llms (compared to in the image/video diffusion space). Perhaps some future model will have some sort of "core" that can load/unload portions of itself dynamically at runtime. Like go for a very horizontal architecture/hundreds of MoE and unload/load those paths/weights once a parent value meets or exceeds some minimum, hmmm.
• rmunn 5mo ago
  For image generation, this has already happened. To what degree, I can't tell, as I don't do image generation much so I don't have numbers on Midjourney subscriptions or any other image-AI-as-a-service sites. But civitai.com has become a place where people share their models, based off of Stable Diffusion or other similar bases, with various fine-tunings to achieve desired results. You name it, you can find a model for it at Civitai, and people doing some very creative things with them. (And also a lot of the obvious things, but it's the Internet, what did you expect?) I haven't seen a text-based model sharing site spring up yet (perhaps they already have and I don't know about it yet). Civitai, being focused on image-generation, has the obvious advantage that it's easy to show off impressive results from the model on the front page of the website, and judging what someone's home-grown fine-tuned LLM will produce is a lot harder. But at some point I expect a Civitai equivalent site for text models, especially code-based ones, to become popular. That will seriously undercut Anthropic, OpenAI, et al, and will probably force them to find a price equilibrium. Because once you're competing with "I spend $2,500 up front on a powerful video card, download an open-source model for free, and then I get pretty much everything I need for free" (additional power cost of running that video card isn't nothing, but probably not noticeable in your power bill compared to what you're already using)... then suddenly $200/month means your customers are thinking "after one year I would have been better off with the homegrown solution". The only way they'll continue to pay $200/month is if Claude/GPT/Gemini/whoever is truly head-and-shoulders above the "pay upfront once for hardware then use it for free afterwards" models available. And that's going to be doable, perhaps, but tough.
  • peab 5mo ago
    Civit ai is like 99% porn though. Most production usage of image gen is google or open ai as they are by far the best
    • rmunn 5mo ago
      As I said, a lot of the obvious things. And if you're scrolling through the front page without being logged in (i.e., so the default "no mature content" filter is on), there's some really creative stuff being done. I personally like the looks of the RPGv5 (or is the guy up to v6 by now? I forget) model, and plan to use it eventually to create custom portraits of characters in my tabletop roleplay campaigns. (Not running any right now, due to having basically zero free time at the moment, but eventually my current situation will change and I'll have the occasional weekend open again). But for a site sharing code-generation models, it's a very different scenario. I'm curious to see what will happen in that space.
• QuadrupleA 5mo ago
  This is just emotional rhetoric. Pretty much any app in the last 20 years has depended on a server somewhere, or a cloud provider. Like an AI provider, they can go down, they can turn off if you don't pay your bill, etc. And local inference requires fairly beefy hardware, that is FAR from ubiquitous across today's userbases. Local models are also still far dumber than what frontier labs can serve. Weird that this is getting such a tidal wave of upvotes.
• hackermanai 5mo ago
  > “But Local Models Aren’t As Smart” This is what makes me continuously doubt and rewrite the local-first approach to inline chat in my editor. Next edit/ code complete makes more sense due to latency advantage. But chat is hard. It's fast and feels good to run locally, but output quality is just not ChatGPT etal.
• QuadrupleA 5mo ago
  Not sure how excited I feel about visiting your website and having it auto-download a 8GB model with GPT-3.5 level hallucinations, and then probably crash because I only have 6GB of VRAM. My dad won't be able to use it, or anyone else without a bleeding edge device. On a powerful enough "neural engine" device the battery will be drained quickly, while the heatsink burns a hole in my lap.
  • dgb23 5mo ago
    Local could also mean self hosted. The obvious optimization for the case presented would be to generate all the summaries on a server instead of in the client. Then the totally used compute would scale with the number of articles instead of number of users.
• vivzkestrel 5mo ago
  - can we get suggestions from people on what would the equivalent for android - and for the web / javascript / svelte applications? - suggestions for local OCR for bulk images?
  • kajman 5mo ago
    I hope there's no web equivalent for a while. I usually hate app lock-in, but any hasty API for this is going to be a DoS or fingerprinting nightmare.
• nezhar 5mo ago
  For me, building with open weights models sounds like the right approach — you are able to switch providers, and you can control where the server is running. You don't have any guarantees in terms of data, that's true, you rely on the provider. But this is similar to a database or other services where you don't have the knowledge or resources to run them yourself. Hardware cost is an additional factor here. If on the other hand your idea works out and the model fits the use case, you can always decide to move to a dedicated infrastructure later.
  • nezhar 5mo ago
    [dead]
• tzm 5mo ago
  People want local AI, but only if UX is good. Tooling/harness quality may matter as much as model quality. I think the future will probably be a hybrid of: 1. local AI for simple, private, everyday tasks 2. online AI for very hard or long tasks
  • anemoknee 5mo ago
    The Clippy app someone made and posted here a while back is the perfect average person LLM interface; https://felixrieseberg.github.io/clippy/ https://felixrieseberg.github.io/clippy/
    • all2 5mo ago
      This is so good. Wiring in small models for a variety of tasks would make this absolutely sing.
  • Gud 5mo ago
    The UI is already great. I can’t wait to run my models locally. The sooner I can do my shit without some American mega corp gulping down all my data, the better.
    • nicce 5mo ago
      I fear that easier it gets to run models locally, more expensive all the hardware gets. So at the same time it gets further and further. You should have bought the hardware yesterday.
      • worldsayshi 5mo ago
        The more expensive it gets, the higher the incentive for more competition in the hardware space.
        • nicce 5mo ago
          The thing is that it is something which takes so long time. E.g. why Taiwan is still so important?
• alfiedotwtf 5mo ago
  This would be nice, but unfortunately the norm at the moment is - release a rushed model that doesn’t work with llama.cpp, but if it does, make sure that the chat template is broken. And even if it did have a perfect chat template, let the model loop endlessly rewriting the same file with same content for hours on end. It would be nice if model makers could at minimum embrace test harnesses, and stretch goal if they’re going to change underlying formats then at least land compatible readers in the big engines (e.g. llama.cpp and vllm)
• hydra-f 5mo ago
  Unless there's a breakthrough or a transition to diffusion models, it's hard to imagine them becoming an affordable commodity Small models are still in their infancy, and there's still much to sort out about and around them, as well
• hona_mind 5mo ago
  [flagged]
• h05sz487b 5mo ago
  I really want this to be true. For me getting all models to run to the best of my hardwares ability and the cli tool to also make best use of the model is still a headache. I had coding models not being able to do a search and replace depending on the tool through which they were called, visible <thinking> elements in my message flow, agents doing a task, failing at the linter, then reverting everything again so the linter is happy and presenting the result as a "good compromise". Right now it feels like we have all the pieces but nobody integrating all that into an amazing experience.
• ninjahawk1 5mo ago
  In my opinion, this is similar to the earlier internet and computers. Few households or individuals had access to state of the art computers, it was primarily research or more well-off individuals. Most random people didn’t really know what it was and certainly didn’t use one. Now today, AI is very expensive and not readily accessible to most people without paying a good amount. The early internet became now you can just get a free phone from phone companies so long as you get their extras. Then you get a ton of subscriptions and ad-ons, but you don’t have to spend money, could just use youtube with ads etc. Local AI would similarly shift this dynamic to paying for access to plug-in’s and tools for your local AI to be able to use. Like how the subscription model works right now. With local model advancements, such as specifically Qwen 3.6 35B A3B, this future is becoming more likely by the year IMO.
• RyanZhuuuu 5mo ago
  I’m skeptical that local AI will work well with today’s technology. Running capable models consumes too many resources on end-user devices.
• testfrequency 5mo ago
  Local AI is definitely going to be the future as these models continue to advance at the rapid pace they already are. This is why I believe OAI and Anthropic I’ve been so aggressive at offering services outside of their pure models like Claude Design. This is what will be competitive and keeping people subscribed.
• october8140 5mo ago
  They will never let us have enough RAM every again. RAM will be kept behind locked doors in the name of national security and only trusted corporations will be aloud to run AIs and "safely" run them in the cloud and sell them to us.
  • j3th9n 5mo ago
    I’ll make my own RAM, with the help of AI.
  • yuppiepuppie 5mo ago
    Is this a conspiracy?
• imrozim 5mo ago
  I use Claude api for my startup and the billings and rate limiting hurt. But local models cant do what i needed yet. Wish they could.
• ElenaDaibunny 5mo ago
  [dead]
• Aleesha_hacker 5mo ago
  To what extent is this strategy currently feasible for windows of android development? I am interested in how portable local-first AI is across platforms, but it seems promising on Apple devices.
• thesuperevil 5mo ago
  [flagged]
• z3t4 5mo ago
  We are experimenting with local LLM and opencode at work and the quality is not as good as Claude code et.al but it's not far off and local speed is actually faster. We got 3 of Nvidias latest AI GPU's which was not cheep. It's not good enough to train our own models, but we can run the biggest open models with some tweaking.
• davmar7878 5mo ago
  [dead]
• ramon156 5mo ago
  GLM 5.1 is very impressive, I wouldn't be surprised if we get to a point where it can live in ~48Gb and have a reliable speed/quality
• system2 5mo ago
  [flagged]
• xiaosong001 5mo ago
  [flagged]
• Tepix 5mo ago
  I'm pretty sure that AI assistants will become widespread. I consider it to be very careless to entrust your emails, your chats, your calendar, your notes, your calls, your pictures, your contacts, your location history, your waking hours, your files, your TODO list, i.e. stuff including your health data to the for-profit AI companies. The temptation to earn money with your data is just too great, plus the risk of the data being stolen and sold illegally. Local AI should be the default. For everone who can't do local AI, we need confidential compute. Yes, it has been hacked before. But it's making it a lot harder.
  • pjerem 5mo ago
    > I consider it to be very careless to entrust your emails, your chats, your calendar, your notes, your calls, your pictures, your contacts, your location history, your waking hours, your files, your TODO list, i.e. stuff including your health data to the for-profit AI companies. Still, we all do it with Google. (I don't do it anymore but i did it for mostly two decades so I include myself)
    • jesterson 5mo ago
      > Still, we all do it with Google We don't. And never did.
• mgrund 5mo ago
  I really really want to like local AI, but I highly doubt it will see wide adoption for a long time. The additional up-front cost for hardware designed to run an LLM in addition to normal workload is unlikely to be accepted by most consumers. The scale will be very constrained (like Apples on-device models which are small, heavily quantized, and have a small 4K token context window). It’s also terrible for battery life. AI as it is implemented today is simply just computationally expensive and unless you put in dedicated hardware (like the ANE) for only this purpose - a large cost driver - I don’t really see it getting large scale adoption. Companies will probably need a server-backed solution as fallback if they want reasonable user experience, so why even invest in diverse hardware support.
• reshef316 5mo ago
  not saying i disagree with the general statement, but there need to be options, not everyone has a machine capable of doing the same type of lifting required to properly run a local version. so what, if my machine is older i'll be locked out? restricted? forced to pay?
• gregjw 5mo ago
  Is there a place to learn more about Local AI specifically and maybe even more specifically about models for bespoke purposes or curating them yourself for more specific uses? Feels like theres a lot of fat you can trim off because you don't need generic use, but I don't understand where to even begin there.
  • StevenWaterman 5mo ago
    /r/localllama is one of the most useful places
• almogodel 5mo ago
  Remember nodes and graphs? A comfy user interface allows pretty incredible wiring among models local ai is like eurorack. The current graph skews heavily towards a a pair of small dense models collaborating with the large heavyweights selectively. It’s Qwen 3.6 27B with Gemma 4 31B, both unquantized, bf16/fp16, with phi 14b, nemotron cascade 2, and then those large heavyweights, r1 and subsequent deepseek models including speciale, gpt oss 120b, glm, min max,kimi, command r, mistrals, ever body, up in one graph, all them llm nodes patched and interconnected. Slow, resource intense, better than non local ai. I used Matteo’s graphllm for inspiration, and comfy ui (and st), and used the models to roll a new imgui node/graph model compositor. Now what?!
  • gpugreg 5mo ago
    > Slow, resource intense, better than non local ai Why should connecting small models to big models result in higher output quality than just running the big models without the small models?
    • CamperBob2 5mo ago
      A hardware analogy: an amplifier might have an open-loop gain of a hundred million or more, but if you actually try to use it without some negative feedback, it will only give you one of two possible output levels. And/or a whole lot of noise.
• tomelders 5mo ago
  I do think local models are the future, but there's still the question of cost to be answered. Even if there's some slew of effincency improvements that mean an LLM can run locally on consumer level hardware on an affordable budget (and that's a big "if"), there's still the cost of training the modles to consider. Assuming we end up in a future where people pay to run multiple smaller models on their machines for specific tasks (e.g. A summariser model, a python coding model, or however fine grained/macro you want to go), the people training those models will need to turn a profit. So how much will that cost? And how often will consumers have to pay? Models have a very short self life. Say you have a dedicated python coding model - that needs re-training every time there's a significant update to the language itself, any popular packages, related technologies (e.g. servers, cloud infra etc). So how often will users need to "upgrade" to the lastest version? It's going to be "frequently". And it still needs the language stuff on top of that. Users aren't going to interact with a python coding model by writing python. They're going to use natural language. So the model needs all that stuff. And they're going to give it problems to solve. What if you asked the model "Write me a Bezier curve function". It needs to know about bezier curves, which have nothing to do with Python. So where do these LLM providers draw the line on what makes it into the training data and what doesn't? And if an LLM doesn't know what a Bezier curve is, that's not going to stop it from just hallucinating an answer. If a significat proportion of prompts resulted in a response that said "Sorry, I don't know what you're talking about", then people will just stop using it. The utility of these things will be quickly overshadowed by the frustrations. The way these frontier models have been introduced and promoted has set unrealistic expectations, and there's no putting the genie back in the bottle.
  • rufasterisco 5mo ago
    > the question of cost to be answered. Commoditizing complements. If Anthropic/OpenAI/etc is eating your lunch, make it work with cheap local LLMs , you can beat them on price by having local inference you don't pay (nor need data centers for), and try to keep your (user/data) moat. The more Anth/OAI disrupt, the more likely this is to happen. If they don't disrupt enough (.ie: grow as an ecosystem to defend against incentives to commoditize), then yes, those incentives are removed, but they also leave money on the table, which they need. Not only at business level, but also geopolitical (to a lesser extent? or not since lots of open weight models comes form China?).
• stuaxo 5mo ago
  Harnessed seem to be a big part of what makes stuff good or not. I tried Cline and couldn't get it working well and part of this was that at the time it expected OpenAIs output format.
• dgb23 5mo ago
  I‘m surpised at the presented dichotomy between JSON formatting and what the Apple SDK provides to parse output into structs. Based on what I understand about how the former works, I would assume that the latter has the same properties and failure modes.
• continueops_com 5mo ago
  Opus 1M context window and lighting fast response time is hard to compete with, even if you run a local A100 the local models are just not as good as tool calling, long running tasks and non-hallucinations
  • twoodfin 5mo ago
    It was hard for an Apple ][ to compete with an IBM mainframe at enterprise data processing, but the power of personal ownership & commodity economics was disruptive enough that 30 years later 99%+ of enterprise data processing was taking place on descendants of the original personal computers.
• khoury 5mo ago
  Agree with the sentiment, but: "We are building applications that stop working the moment the server crashes or a credit card expires." This has been the case for way longer than openAI and Anthropic has been around with services like AWS, Cloudflare, etc.
• jillesvangurp 5mo ago
  I get the sentiment for self hosting. But there are a few counter arguments: - Self hosting is expensive. It involves expensive machines with GPUs that cost hundreds per month if you use cloud based ones. You might need multiple of those. And you need people to mind those machines and they are even more expensive per month. - If you run stuff on your laptop, it consumes a lot of resources and energy. I have qwen running on my laptop. Even minimal usage turns my laptop in a radiator. Nice as a demo, but I can't have it this hot all the time. It would run out of battery, and it's probably not great for longevity of components in the laptop. - Models are evolving quickly and the self hosted smaller ones aren't as good when it comes to things like tool usage, reasoning, etc. Being able to switch tot he latest model is valuable. - It's easier to get your use case working with one of the top models than with one of the smaller self hosted ones. - If you get the wrong hardware, it might not be able to run the latest models very soon. - Self hosting models is mostly a cost optimization. It only becomes relevant if you hit a certain scale. - You have alternatives in the form of hosted models via a wide range of service providers. Some of those are EU based and offer all the things you'd be looking for if you are offering your services there. Including legal requirements. - Reinventing what these companies do in house is technically challenging and possibly more expensive than self hosting models because now you need a lot of engineering capacity dedicated to that. And legal. And all the rest. If, like most companies/people, you are at the experimenting stage, the cheapest and fastest is just getting an API key from an API provider of your choice. You can take it from there if your experiment actually works. And then it's mostly about optimizing cost. If your API usage goes to the thousands per month or worse, it becomes a cost/quality trade off.
• teiferer 5mo ago
  Every reply here forgets/overlooks the main reason for why this is not going to happen: The astronomical AI data center investments currently underway. Those place are not just for training. They are for inference too and the way all those investments are expected to eventually pay off. The whole AI sector of our industry depends on running models in these places.
  • zozbot234 5mo ago
    These astronomical AI data centers will be used for high-value inference with smarter models that really are too large for running locally. The investments will be fine once they pivot to that use. Currently available open models are not in that range.
    • teiferer 5mo ago
      I don't buy that that will be a useful distinction. First of all, no AI model will say "I'm too smart for this question, I suggest you use a cheaper one so I don't make unnecessary money for my owner" or "I'm too dumb, so instead of hallucinating I'll suggest you go to the cloud and ask my smarter sibling". Second, there is no incentive in the market for tooling to evolve that way. There will be the illusion that some models will do that, similar to today (or maybe some harnesses rather) but nobody will willinglylet money sit on the table. These data centers are not being built to solve world hunger. They are built to ultimately hook you on more realistic fake bs youtube videos so you feel good while getting even more ads injected into your life.
      • jononor 5mo ago
        Dynamic routing is the usual name for the piece that orchestrates which LLM will be used, based on query complexity. There in an open source implementation as part of the vLLM project (and probably others), it is a field of active research in several universities and labs. It is also suspected that the frontier LLM providers might already be doing something like it behind the covers.
• maxdo 5mo ago
  The start of the argument is already broken . Ok , slapping api is bad , so you push api that mimics to your provider, install some Chinese llm that will never obey any lawsuit in your country , install 500 packages to do so , every of them has a potential risk a security issue . How is that better ? Oh yeah , it feels independent and not lazy , sure
• cl0ckt0wer 5mo ago
  If they do then hardware costs will explode even more
• KurSix 5mo ago
  [dead]
• skillsora 5mo ago
  [flagged]
• moveax3 5mo ago
  [dead]
• throawayonthe 5mo ago
  it's not going to happen with LLMs unless ram + storage gets several orders of magnitude cheaper like, yesterday informatics aren't magic, you'll never be able to compress """knowledge""" into a small model in a way equivalent to the 1.5 TB model
  • acidhousemcnab 5mo ago
    This will happen, but reconfiguring the infrastructure of the entire planet to train LLMs and run them over networks might be the "bubble", the megalomania.
  • kilroy123 5mo ago
    I agree. But I also think the future is some kind of hybrid approach where agents run locally, what they can, and then call out to the cloud for what they can't.
• acidhousemcnab 5mo ago
  We need better GUI and OS integrations with sandboxed local LLMs, before this is thrust on everyone and rolled out as the default in commercial OSes. Here in Berlin, I was functionally surrounded and hounded out of a local meetup, due to confrontation over the naive pushing of OS-level and network access agentic AI, done in the mode of mystical powers and artistic possibilities, which due to recent experiences, comes off as string-pulling, to produce a threat or danger that then must be observed and kept tabs on, according to Goodhart's Law.
• worthless-trash 5mo ago
  How long till we have distributed AI, where we can have different people run/understand different parts of problems and pass off work to different nodes across the internet.
• andychiare 5mo ago
  > “AI everywhere” is not the goal. Useful software is the goal. Great observation! Often the excitement of novelty makes us lose sight of the real goal
• cryo32 5mo ago
  I think no AI needs to be the norm. Even if we have enough RAM to run it locally, the dependency stack we have on hardware, training and geopolitics is too much of a risk to take on. If something breaks, like supply chain, or the model is found to have particular bias or exploits baked in, we're fucked.
• harrouet 5mo ago
  Running LLMs locally is one way to realize the level of hardware and infrastructure that frontier AI companies are running. Makes me wonder about future strategies. As one commenter mentioned, 2x Mac Studio M3 Max with 512GB can run frontier models and it costs $30k (with RDMA). Apply an efficiency ratio for being in a datacenter, and you understand why OpenAI and the likes spend north of $10k _per customer_ of CAPEX. Add to that the electricity costs and you've got a very shaky business model. I for one would like to thank the VC for subsidizing my tokens. With that said, the VCs are not crazy and probably factored in an annual cost decrease of computing power. But how do you make sure that we won't run local LLMs when the HW becomes affordable -- if ever ? The answer has always been the same in our industry: vendor lock-in. They are getting the users now at a loss, hoping for future captive revenues. So, be careful when your code maintenance requires the full context that yielded that code, and that this context is in [Claude Code|Codex|Cursor].
• 8cvor6j844qw_d6 5mo ago
  Any recommendations to run a local model on a Raspberry Pi 5 16 GB?
• mitchsayre 5mo ago
  I now fully believe that the models will soon be compact enough to work even on older mobile devices. I work on lightweight text-to-speech models. After training on distilled datasets the models sound basically the same as any closed-source speech API model, they just need a ton of data to train on. Other researchers are seeing similar gains with other types of models and its only a matter of time before one drops thats commercially viable. Once this happens, the innovative apps and games will begin shipping AI as a feature that drives the user experience forward, rather than the thing you price the entire product around.
• deleted 5mo ago
  [deleted]
• AuditMind 5mo ago
  It's almost here. Look at the new Qwen 3.6 models. Solid stuff there. It runs by now on 8GB Vram, so a Legion 5 for about 1500$ could be a good workhorse.
• Akuehne 5mo ago
  I feel like lots of people here are just commenting on the headline. This isn't about the local models you're running on your old gaming rig, or the tesla p40 rig you build for local llm's. This is about code leveraging the local resources where the code is running for it's AI needs. Rather than making an API call to an external AI service, the code leverages the AI capabilities built into the hardware it runs on. With modern Apple, Intel, and AMD silicon all shipping dedicated AI acceleration, this is the where IMO the focus should be heading. How many Flops or whatever can your phone do? I bet it's enough to paint the walls of your living room, or draw a pretty good pelican on a bike.
  • andybak 5mo ago
    > draw a pretty good pelican on a bike. You mean the famously hard task? The one picked because it stretches frontier models to their limits?
    • daveguy 5mo ago
      https://simonwillison.net/2026/Apr/22/qwen36-27b/ https://simonwillison.net/2026/Apr/22/qwen36-27b/ Maybe this is an example of training overfit. But it won't be too long before local models chew through the "famously hard tasks". Except possibly ARC-AGI. That's one benchmark that is still developing with capabilities. And every time a new ARC-AGI benchmark is released it make the SOTA LLMs look pathetic. Because there is very little understanding or transferability with LLMs. But in terms of benchmark-able micro tasks, the local LLMs are improving.
    • quantummagic 5mo ago
      In fairness, that isn't due to a lack of compute.
    • munk-a 5mo ago
      It was a famously hard task. It was an ingenious idea for an unexpected task that falls outside of the bounds of predictable normal input but is still readily comprehended by the public. Unfortunately, as soon as it's a famously hard task trainers know they need to succeed at it and it loses a lot of the power to detect correctness.
• deleted 5mo ago
  [deleted]
• theuniverseson 5mo ago
  [flagged]
• noashavit 5mo ago
  Relying on external APIs network failure points and unavoidable latency from the round trips. There is also the AI API rate limits that come into play. We might find that for critical workflows, local compute is the only reliable architecture.
• bluGill 5mo ago
  WRONG, this completely ignores the most important issue and so is completely wrong. The important issue is where is the data stored. And there are far to many advantages to having your data in the cloud: you can access it from whatever device you happen to have, and it isn't lost if you lose the device. This also outsources your backups to the cloud which is probably doing a much better job than you would (maybe no on hacker news, but nearly everyone else) - the cloud has earned a bad reputation for backups, but it is still much better than most people would be. Once you accept the data is going to be elsewhere it doesn't matter if the compute is elsewhere or not. The data is the important part. What needs to be the norm is more self-hosting your own data. Companies should not be outsourcing this by default - even where you outsource some of it, you need to watch your contracts and ensure the ownership is yours - not shared. Once your data is yours on your own cloud accessible servers we can start asking can we run our AI models in the same data center as we already have our data in. I don't need my AI model to run on my phone, it can run on the server in my basement which has a lot more power available (my phone has a better GPU but I can't afford the battery power to run AI on my phone)
  • mohamedkoubaa 5mo ago
    I want a way to backup my data fully encrypted somewhere and have custody of the keys - but importantly, the data should all be decrypted locally where all my apps can use the data without any network
    • chasd00 5mo ago
      tar -czf - /path/to/folder | gpg -c -o folder.tar.gz.gpg then scp/POST that somewhere /s..kinda
  • pcthrowaway 5mo ago
    > What needs to be the norm is more self-hosting your own data I assumed self-hosted AI would fall under local AI for the purposes of this article. Does the author really need to spell it out?
• abhishekhsingh 5mo ago
  [dead]
• tristor 5mo ago
  The biggest challenge I have with local models right now (and I use them extensively) is search integration and tool calling. The thing that Claude and ChatGPT get right for most general purpose use cases which is hard to do with a local model is the model deciding when to search vs use its built-in training, and having strong search tooling, as well as tool calling for additional data sources via MCP. If you can incorporate the right data into the context window, local models are more than good enough for general purpose usage as they stand today. Qwen 3.5, Gemma 4, even gpt-oss-120b are solid at reasonable quants if they have the right data. The moment we see standardized and batteries-included pathways to integrate search, ideally at no additional cost, in things like LM Studio combined with better tool calling in the local models, you'll quickly see local model performance catch up.
• senko 5mo ago
  I love this line: > Stop shipping distributed systems when you meant to ship a feature. But not in the contex the author meant. Many people don't realize that when you have a frontend, a backend (several instances, for failover/scaling), a (separate) database, maybe some object store -- you have a distributed system. A recent article[0] touched on that, although most HN commenters[1] latched on the "go" part. But there's something to avoiding rube goldberg machines where we don't need them. [0] https://blainsmith.com/articles/just-fucking-use-go/ https://blainsmith.com/articles/just-fucking-use-go/ [1] https://news.ycombinator.com/item?id=48062997 https://news.ycombinator.com/item?id=48062997
• henry_kang 5mo ago
  [dead]
• sinansaka 5mo ago
  I'm betting my startup on it. The subsidised model subscription will start to dry out and providers will lean heavier into locking down how they want their models to be used (Anhropic has been paving the way already). The only way forward is open weight models. If you are working on any LLM powered product be careful betting on utilising user subscriptions.
  • sanderjd 5mo ago
    What is your startup?
  • 0xbadcafebee 5mo ago
    Maybe you know something I don't, but it seems the standard will continue to be a large number of companies hosting and reselling LLMs as both subscription plans and pay-as-you-go. It's virtually identical to the mobile market: the economics of the business require a large regular infusion of cash, and limits are used to prevent a minority of users from making the service unusable/unprofitable. A few giants are the most expensive but offer the most features, and cheap providers offer less for less. All of this will happen because people constantly want "more": more bandwidth, more quality, etc. Capitalism rewards this constant growth/advancement with constantly increasing bills. Anthropic is going to go out of business by probably Q1 2027 due to not paying their bills. OpenAI will become a new Oracle, serving a luxury product for enterprises and governments. Google and Microsoft will keep doing what Google and Microsoft do. Chinese vendors will capture a significant amount of business over the next 10 years by running the models in non-Chinese DCs, with demand coming from their much lower prices. 95% of regular users will be paying for open model subscriptions, even if their local machine can run the model, because the providers will be offering features that are hard to impossible to replicate locally.
• JamesSwift 5mo ago
  I think moving straight to local models is missing the required next step of open/self-hostable models which is certain to be the "AI future" end-state. Then local models become an optimization on top of that. I just dont want us to put all this effort in to on-device computation when we need to get to "SOTA-equivalent" self-hosted computation faster.
• ge96 5mo ago
  I'm looking into it since it I'm going to be sending personal info/thoughts would like to keep it local. I have a 4070 running the TheBloke 7B mistral via llama cpp. I still am not using llms daily though other than Google searches.
• butz 5mo ago
  Really silly, when you buy "AI PC" with "AI CPU" and still run any "GenAI" related stuff in the cloud.
• Slix 5mo ago
  Chrome did this, and there was a huge outcry. Even though local AI is much better for privacy.
• unnouinceput 5mo ago
  Quote 1: "We need to return to a habit of building software where our local devices do the work." Quote 2: "I can only speak on the tooling available within the Apple ecosystem since that’s what I focused initial development efforts on." Oh, the irony. I will use your tooling when is available on Android with F-droid, that's when, at least, be decoupled from big companies grip.
• chakintosh 5mo ago
  I'm literally working on an iOS app right now that needs to infer some input fields from free text typed by the user. Now to take into consideration typos, unstructured text (pricing, dates .. etc), I was pondering a cloud LLM or a basic local parser or even a local on-device LLM (ANE for 15+ devices and a different on-device LLM for the older models) For the different on-device LLM, I literally went to HuggingFace and filtered by the smallest available models that can do the job, and Granite-4.0-h-1b works just fine, it corrects typos, infers dates, currencies all fields I need. And it got me thinking how my first reflex was to rely on a cloud LLM which is waaay overkill for my need. Granted, an on-device LLM will need to be loaded on the devices on install or downloaded after the fact (which adds latency when the user needs it for the first time) but still, it's a better tradeoff than a cloud LLM. I decided on a basic parser, and so far it seems to work fine. granted, it struggles with some words, but I just need to finetune it to have as much coverage as possible in terms of typos without triggering false positives. A lot of developers have that reflex too and go along with it and then just pass the API costs to the customer. I could have gone that route too but turned out I don't even need an LLM for my usecase.
  • deleted 5mo ago
    [deleted]
  • coevcan 5mo ago
    Apple includes a local LLM on all recent iPhones, https://developer.apple.com/documentation/foundationmodels https://developer.apple.com/documentation/foundationmodels. Seems like a bad idea to force your users to download a 3GB LLM just to parse a text field.
    • chakintosh 5mo ago
      Yeah but I need broader coverage on older phones. No I'm not going for a 3rd party LLM. Foundation Models for iPhone 15 and newer, and a parser for the older ones. Currently training a Word Tagger in Create ML
• wolvoleo 5mo ago
  > Most app features don’t need a model that can write Shakespeare, explain quantum mechanics, and pass the bar exam. They need a model that can do one of these reliably: summarize, classify, extract, rewrite, or normalize. > And for those tasks, local models can be truly excellent. 100% true and I use them for this. But the open-source models seem to be drying up unfortunately. There never was much incentive for the big players to train a model and give it away for free, it was mostly virtue signalling and advertising for their knowhow. The AI "race" seems to have entered a new phase that's more on clamping down costs and making money and this doesn't fit in well. I hope good local models will still appear but the days that there was a new groundbreaking model for download every couple of weeks is over :'(
• kandros 5mo ago
  We need more tools like QMD that beautifully download and use local models under the hood https://github.com/tobi/qmd https://github.com/tobi/qmd
• osjxjsjxjs 5mo ago
  No AI needs to be the norm. Again.
• leoc 5mo ago
  (I am not an expert on anything.) One happy circumstance here is that while the RAM cartel is chasing Big AI's money today, in the medium term its self-interest probably makes it a supporter of local AI. A new, compelling reason to have 128GiB, 256GiB or more of VRAM on all your devices? You can be sure that the dollar signs are glowing in their eyes already. The less efficient use of VRAM by personal devies (any given device's VRAM will be mostly idle much of the time) tends to make it more attractive, all else being equal (though of course it isn't) compared to the centralised systems run by engineers and accountants striving all day to maximise ROI; and in any case, since the short-run supply constraints on RAM go away in the longer term, the RAM manufacturers will be able to supply both. My guess is that you can probably also also explain Apple's AI strategy (sit tight and wait for Moore's Law to make local AI more viable) and maybe even nVidia's (lay the groundwork for a gradual switch from selling shovels to the army to selling shovels at Home Depot over time, at least as a Plan B) in similar terms.
  • dTal 5mo ago
    Just because we'll have to pay for the hardware, doesn't mean we'll have meaningful control. Look at what happened with phones - weak and limited slaves to the mothership, secured against pesky users with powerful encryption, yet costing more than a vastly superior laptop; quasi-mandatory platforms for highly addictive experiences, centered around the flow of information. And now with LLMs we can create even more fabulously addictive experiences, even more finely tuned information flows, even more treacherous servants. I very much doubt that we'll be allowed full control of it all. Every effort will be spent to centralize power, and every effort will be spent to extract as much cash as possible from us for the privilege.
    • fsflover 5mo ago
      > Look at what happened with phones - weak and limited slaves to the mothership, secured against pesky users with powerful encryption Not all phones are like this. GNU/Linux phones obeying users exist too.
    • array_key_first 5mo ago
      Phones are such a travesty because they're so incredibly overpowered. I think there's a lot of people out there where their iPhone has more compute than their laptop or desktop, but it can't do 1/10th the amount of stuff. What a waste!
• deweywsu 5mo ago
  How is having local AI going to produce a result that's any better than using OpenAI or Anthropic? Isn't what we really need programmers who rely on themselves more than AI so they avoid technical debt accumulation?
  • jononor 5mo ago
    Having local AI as a credible threat will keep them on their toes. Which will benefit consumers a lot.
• ki_sum_ai 5mo ago
  [flagged]
• katzito 5mo ago
  Most people are lazy (which is (mostly) good) and don't care (which is (mostly) not good), as Gmail has proven since 2004 (according to Google AI). Still waiting for those analog AI chips that were supposed to make it lightning fast using minimal energy...
  • nsvd2 5mo ago
    Assuming you're talking about Taalas, they have a live demo for inference on their HC1 chip.
    • katzito 5mo ago
      Taalas HC1 is digital. Was thinking more along those lines: https://mythic.ai/ https://mythic.ai/
• maxothex 5mo ago
  [flagged]
• plexescor 5mo ago
  Yea i agree to this. Especially considering that now even igpus can get respectable scores, like my iris xe 80eu 16gb ram @ 2133mhz gets like 6-8 Tokens per second in gemma-4-E4B model
• runfreeapps 5mo ago
  Any project that requires a local model should always be the way to go on first attempts and if the functionality is acceptable should stay with local models. Token burn is a serious problem and will ultimately lead developers to ask one question "Do I really need Opus xyz?" For most requirements of standard applications the answer is no. So using open-source llm models that are integrating in practical use-cases to create a value-add not for 'hey look I have AI in my app, sign up please.' Open source models are competing well and is the way to go for the majority of projects and mindsets do have to change and I see them changing this way rapidly. You don't have to host your open-source llm locally but host it with a 3rd party, it is cost-effective and the token burn is not a barrier.
• selectedambient 5mo ago
  Agree. We ought to be measuring the minimum viability of lesser parameter, local models for specific tasks. You don't need opus 4.7 or sonnet 4.6 to accomplish some of these basic, yet tedious tasks, i.e. the news aggregator you demonstrated. Thinking about things like, how many parameters does it take to manipulate a pdf in every way possible with accurate results? Likely, a reason there isn't a coordinated push toward people running local models is the fact that your data couldn't be mined, manipulated, and abused; obviously outside pure capability of some of the frontier models (which truthfully some of which aren't even very good). While I think we may see more things like Apple's models, like you mentioned being run locally, I think we all know at the end of the day they're phoning home in some way (which if that is fine for you, fine). Again though, and you touch on this in the article, highly specified tasks that have a certain amount of redundancy built in are very suitable for these local models right now, without relying on enormous weights and token usage. I have been working on a VERY SMALL local-first ai lab myself. nothing crazy, a text editor, a claw, and some lightweight models I started playing with. Absolutely looking for contributions as well.
  • selectedambient 5mo ago
    didn't want to lead with it but if interested: https://mithraeums.github.io/ https://mithraeums.github.io/
• latentframe 5mo ago
  A lot of AI aspects probably don’t need to be permanent cloud services as local hardware improves part of the industry may change from renting intelligence to on-device computing.
• PeterStuer 5mo ago
  I use a 4090 and 96GB ram to run local models slowly (atm Qwen-code-next at 7 tps) with their full context window. I keep this up just for testing and practicing fallback should I lose access to Claude and GPT.
• grig0r 5mo ago
  This doesn't make sense for consumer apps if it chugs a ton of RAM. No student will want to use local AI apps if their Macbook Air's battery dies in 2 hours.
• butterNaN 5mo ago
  The ability to build my own local AI is exactly what I want to learn. Are there any good resources to learn this?
• unixhero 5mo ago
  How do we know that Qwen is not up to anything nefarious?
• giancarlostoro 5mo ago
  We could have been there if the big AI companies didnt create a RAM crisis. I will be buying the next iteration of the Mac Studio, I have been doing local inference on my Macbook Pro and just small models, I cant imagine how much better things will be on the Mac Studio.
• MarkPeterson 5mo ago
  [flagged]
• JakobSmith 5mo ago
  [flagged]
• shailendra_sis 5mo ago
  Yes, local ai is the future. More important is democratizing the ai for the common masses.