6 ms·

We’re not there yet, but the obvious endgame of the present bubble insanity is open models running on local hardware and devices are “good enough” for most use

• herval 5mo ago
  this is sorta like saying that being able to run your blog on your laptop will completely implode the cloud business
  • Kinrany 5mo ago
    More like implode proprietary blog hosting platforms and replace them with commodity VMs that can be used for blog hosting, among other things
  • grumpymuppet 5mo ago
    It's a little different because cloud and blogs didn't actively get in the way of your home compute. To wit, the various cost spikes for hardware. People -- WANT -- this technology on their home devices and (apparently?) the providers of this tech don't seem to be running a profit so they probably don't want the maintenance tail on their side either. I think it's a bit different. Inevitable that this becomes a household-run thing? Not likely.
  • asimovDev 5mo ago
    Wouldn't arcade cabinets vs home video game consoles be a more apt comparison?
  • malmz 5mo ago
    Running an LLM locally is theoretically viable. Running your blog on your laptop is never viable (unless you hook it up like a server). One just requires compute while the other a stable network.
    • Scoundreller 5mo ago
      tbh, my home network is pretty close to the stability of my host these days… But my downtimes are a bit self-inflicted: changing ISPs which I can personally workaround but harder for a blog where one expects uptime.
  • cduzz 5mo ago
    This is actually what happens. I run my word processing software on my apple 2 (a total joke of a computer) instead of running it on the WANG. I run my book keeping software on visicalc instead of the IBM. I run my simulation software on my IBM PC (I even paid for the 8087!) instead of the VAX. Moore's law has, at least so far, allowed the pioneers with toy computers to grow their toys big enough to solve "big boy" problems after some time has allowed the toy computers to be faster and the pioneers have scaled their crappy home-grown solution to solve their 60% of the problem that was originally solved by some enormous complex system. Eventually the toy infrastructure gets expensive and solves 90-120% of the "big iron" problem space, but it also grows to cost as much as the big iron solution, but then a new generation of toy software and toy systems emerges to disrupt the "big iron" systems. See also http://www.catb.org/jargon/html/W/wheel-of-reincarnation.html http://www.catb.org/jargon/html/W/wheel-of-reincarnation.htm...
    • ethbr1 5mo ago
      Under appreciated requirement for this to work in post-cloud times: open source If a vendor can SaaS a solution, then enterprise is generally happy (they don't want to have to hire folks for maintenance), and that completely locks out any ability to run locally. Between enterprise's ambivalence and the obvious financial incentive to vendors, you get SaaS-only products.
    • manoDev 5mo ago
      You're right Moore's law has been holding up, but will hit a hard limit on process node size, so all scaling will be based on multiple cores. OTH, computing per watt spent has been plateauing. If the future bottlenecks are energy and cooling, that will require infrastructure-scale solutions. My bet is this is going to be real AI company moat. https://www.riq.net.br/pub/computing-scaling/ https://www.riq.net.br/pub/computing-scaling/
  • observationist 5mo ago
    It's a huge difference. If you had AI sufficiently good running locally on a phone, you could devise workflows for things like basic digital hygiene, technical assistance, and tedious tasks like inbox management, image sorting, device updates, and so on. Privacy and security gets a big boost past some local competence threshold, and we're nearly there. Make the local AI competent enough to do good image generation and editing, realtime voice and music generation, handle agentic tasks with a framework like Hermes, and you can take your AI places to do tasks in contexts that are inaccessible to or inappropriate for cloud. Frontier big platform models will be the best, but there's a level of "good enough" for local uses that we're already seeing flourish, and "good enough" for the average joe is almost here.
    • zozbot234 5mo ago
      Phones and laptops are terrible devices for local AI, way too constrained by bad thermals and small batteries. MiniPC's (many of them using mobile hardware) don't have that particular issue, and can easily run on a 24/7 basis.
      • trollbridge 5mo ago
        Phones are also a terrible place to run a radio, but there's a huge amount of benefit in figuring out how to do so.
        • observationist 5mo ago
          That level of local AI is also more or less what you need for competent autonomous robots, too. If your household robots are orchestrated from your phone, the local security and cloud convenience converge on a single device. No extra servers, etc, reduced cost, all that - local AI is a massive market amplifier.
          • yard2010 5mo ago
            Let me speculate - we are going in the weird direction of no private property unless you're an overlord that rents his property to peasants. I like to call it the revenge of communism. See how the market behaves in the llm space - it's more viable to share infrastructure than to own it. Imagine the private car revolution in the US was a bus revolution.
  • asdfsa32 5mo ago
    The primary feature of a blog or any website is that it is available around the clock, that is the primary feature of cloud: around on the clock computer and network that scales on demand. The primary feature of "AI" is to process information and reason with a natural language interface at speed, the primary feature of AI bigboys is to provide the machinery that runs the "models". See the difference?
    • gowld 5mo ago
      You severely underestimate how little the fraction of the performance and human labor of a frontier AI is in "the model". Hosting a blog 24x7 on a laptop is trivial, except for hyperscaling to the front page of HN and Reddit.
      • asdfsa32 5mo ago
        Yeah, exactly, hosting on a laptop is trivial except for when it is not. However, I am using an AI on a mac mini just fine, Qwen 3.6 27B at Q6. Works just as good as STOA models for most things.
  • emsign 5mo ago
    You have to consider that the enshittification factor is much higher now than in the cloud-for-free age.
• IdiotSavage 5mo ago
  I find that hard to believe. The AI companies will want to control what's possible and find new things to do that "need" their services. Otherwise it would be like Intel and Microsoft had decided in the year 2000 that computers are "good enough" now and we would have explored what's possible with that hardware ever since.
  • benterix 5mo ago
    > The AI companies will want to control what's possible and find new things to do that "need" their services. That's correct. The problem is they have smart people, tons of money, and several years to figure that out, and the best thing they can come up is a coding agent.
    • lazide 5mo ago
      That isn’t the best thing they’ve come up with. It’s a marquee product that is fit for public consumption, however. The ‘best’ things are; - fuzzy pattern matching algorithms for traffic analysis, human and other image target recognition. - targeting algorithms that identify ‘suspicious’ individuals in large volumes of metadata. - fraud analysis - antagonistic image and video generation, both for fooling other fraud analysis, but also for propaganda, screwing with other actors, etc. - directed high speed content generation (text, pictures, video) to spam the ‘algorithm’ and allow near realtime identification of additional buttons to push for given target audiences. - massive marketing/ad manipulation. Those budget line items (and the suppliers) really want to stay off the radar however, as it makes their life harder.
      • benterix 5mo ago
        But you're mentioning several things that predate the current LLM craze and belong to the ML domain. These mostly benefit from GPUs but often have much lower hardware requirements. I'm talking specifically about the moat of LLM providers.
        • lazide 5mo ago
• benterix 5mo ago
  I wouldn't say "completely implode", too much money was poured int it, but it's clear we're heading in that direction. You get a model that is "good enough", plus privacy, plus savings in the long term. Paradoxically, the better results we get from general harness of coding agents, the less moat Claude and co. get. It's unbelievably how fast some open models outpaced frontier models of just a few months ago.
  • brightball 5mo ago
    I keep intending to find time to try them. What are you seeing the best results with?
• PLenz 5mo ago
  This. OpenAI and Anthropic are ultimately compute infrastructure plays and not really AI. Everyone will have models, they'll have the ability to run them. This is why the GPU shortage is in their favor.
  • butokai 5mo ago
    Do you think there will still be an incentive to release weights in that scenario? Everyone will have models only if there continue to be companies releasing weights.
    • PLenz 5mo ago
      Companies won't but I suspect this is a role that something else open source-y will fill that niche. Maybe orgs like wikimedia or internet archive, maybe some hackers just making things, maybe nation states that want to disrupt other players. Also model training will get better and better both on the algo and the hardware side. You can easily see a world where you might be able to train a good enough model on a home lab in a few days.
      • rmoriz 5mo ago
        But you will need training data. Like a whole Internet search engine or massive data scraping. That‘s a thing that will not change with better algorithms, hardware or cheaper energy.
        • PLenz 5mo ago
          Data is the only moat but they'll be starting in the same place the current set of players statyed out just a few years ago. I suspect that the delta between what is publicly available (if not legally publicly available! see scihub) and what open ai and anthropic have is relatively small.
  • ryandvm 5mo ago
    And like Google and Meta, these companies are going to morph into advertising giants. Advertising is an economic black hole and it eats everything that comes close.
• cbdevidal 5mo ago
  Happened to me. CoPilot changing prices prompted me to cancel my CoPilot subscription and install a local coding model running entirely in VRAM. Will call Claude APIs when I get really stuck, but I should be able to handle 80% of my needs with a dumber local model. For a long time, too. Programming languages rarely change much, techniques rarely change, so I should be able to use said model for I hope at least five years; and if at any time they optimize local models to cram even more intelligence into the same amount of VRAM, I can upgrade to that. I like this path.
  • Aurornis 5mo ago
    > Will call Claude APIs when I get really stuck, but I should be able to handle 80% of my needs with a dumber local model. I experiment with all of the local models I can fit into 32GB of VRAM and I have subscriptions to multiple SOTA providers. The difference between them is very large, unfortunately. The local models can handle small tasks and refactoring mostly okay, but doing anything challenging with them becomes a waste of time. Unfortunately the waste isn’t immediately obvious because they will come back with something that looks like it works, but then on closer examination I need to throw it out and reset them in a usable direction.
  • deleted 5mo ago
    [deleted]
• sreekanth850 5mo ago
  Curious when NVIDIA monopoly will ends. China will sure release something that can runs on commodity hardware. I wish they will soon.
• dboreham 5mo ago
  Not saying this isn't the case, but my Anthropic subscription costs me less than the electricity would to power such a home inference system.
  • dragandj 5mo ago
    What happens when Anthropic decides that the free hay time is over, and it's milking time?
• ryandvm 5mo ago
  I disagree. We are currently in a weird period where these frontier AI companies are losing tons of money even on the subscription-based AI models. It's just too compute intensive and there's no way most people are going to be buying the kind of hardware required to run $20 worth of inference every day. Sadly - it's going to be ads. Advertising is going to get in there and enshittify the whole thing because as always, advertising income is too easy and too plentiful for any company to resist. Right now the models are fairly agnostic, but we are a hair-breadth away from ChatGPT responding with, "the right tool for this job is a circular saw - something like the Milwaulkee M18, which happens to be on sale at Home Depot this weekend."
  • Scoundreller 5mo ago
    Ads are usually the workaround where you don’t deliver enough value to get people to subscribe or payments are unavailable for some reason. It makes sense to show some ads and get some money at low volume (like a faraway reader wanting to read a story in your local newspaper) but taking money from regular users directly will pay much more. Newspapers are happy to cannibalize 99% of their ad revenue with a paywall if that 1% subscribes because that’s how much more money you make from someone paying $10-$20/month vs ads. But yeah, if people use it as a buying recommendation engine, that’s where the money is on ads/referrals but a lot of AI use has little/no connection to buying intent touchpoints.
    • hylaride 5mo ago
      Newspapers had no choice after craigslist and later Google/Facebook took all their classified revenue. LLMs may or may not be able to cover their costs with it. We'll see - I suspect product placement as recommendations will become a thing as it won't take as much GPU to give a "recommendation" on "the best widget for X". I firmly expect it to become enshittified the same way google and amazon search has. And that's if LLMs don't become commodified.
      • enoint 5mo ago
        For agentic services, how would you be able to tell that you’ve been product-placed?
• techpression 5mo ago
  Gamers Nexus has a good video on this, but if NVIDIA exits the consumer market, and honestly why would they stay when they can charge up to a 100x for the same wafer space for enterprise, AMD would likely do the same. Only Apple really makes consumer hardware suitable for running things locally then, and maybe some weird Qualcomm ARM chip for Windows. It will be hard running things locally if nobody is supplying the hardware.
• mv4 5mo ago
  You just described the absolute nightmare scenario for the newly minted trillion-dollar companies whose only hope is for enterprises and SMB to move all their business processes to the cloud, with employees competing at token maxxing.
• fooker 5mo ago
  If you are willing to spend about 2000 on GPUs, we are almost there. In my opinion, the bottleneck is the package management layer and not the model capabilities and performance. I have been an avid Linux user for decades, and if I find it confusing and painful, something is missing.
• Npovview 5mo ago
  More likely we will have a compute device like NAS or something which will run one good model locally for all the house members just like we have one wifi router in every house. Nvidia can invest in building such a device as well as the models and make money on the hardware.
• materielle 5mo ago
  One thing I don’t quite understand: Wouldn’t it be in Amazon’s interest to run open models and sell time slots at around the cost of running them? My only guess for why they don’t is that AI labs are currently selling their models at a huge loss, so this isn’t worth Amazon spending low-margin compute on compared to other higher margin products. What I’m getting at, is maybe we won’t even need to run the models locally for the current status quo to implode. After today’s AI labs run out of free-money runway and actually have to sell their models at a price above running them, there will be the incentive for anyone with compute to just undercut by selling open-models-as-a-service at commodity prices.
  • philipkglass 5mo ago
    AWS Bedrock offers a mixture of proprietary and open-weights models (DeepSeek, Nemotron, gpt-oss, etc.): https://docs.aws.amazon.com/bedrock/latest/userguide/model-cards.html https://docs.aws.amazon.com/bedrock/latest/userguide/model-c...
• billfor 5mo ago
  It might just shift who is buying memory, from large corporations to billions of individuals.
• ranger_danger 5mo ago
  Given the current performance requirements for "good enough for most people", I just don't see that happening any time soon. Most users (potential or actual) are not on a desktop and don't have a beefy discrete GPU. There are "NPU" ASIC chips like what is being put in the new raspberry pi's but their performance and compatibility is not what you might think it is. To get GPU-like performance the ASIC would have to be closer to the size of a real GPU, and at that point why bother. And many devices just don't have the room.