6 ms·
"I am not sure how many people will run AI models locally. It still seems like a niche application to me. However, it will make decent machines to play video ga
by infecto 5mo ago
"I am not sure how many people will run AI models locally. It still seems like a niche application to me. However, it will make decent machines to play video games."
I don't know who will be the winner but with some of the recent releases from gemma it seems more probable that you may run some models locally if only from a cost perspective, not even considering business security. Not sure how this type of architecture would make for good gaming though, puts into question the whole statement.
"Ranked in the top 2% of scientists globally (Stanford/Elsevier 2025) and among GitHub's top 1000 developers" - side note but this guy puts this everywhere, gives me probably the inverse of what he is marketing for.
- iLoveOncall 5mo ago> "Ranked in the top 2% of scientists globally (Stanford/Elsevier 2025) and among GitHub's top 1000 developers" - side note but this guy puts this everywhere, gives me probably the inverse of what he is marketing for. Lol yeah seriously, that stinks "I ask AI to generate a huge amount of bullshit and upload it to pad irrelevant stats". Absolute loser.
- deleted 5mo ago[deleted]
- netsharc 5mo agoI found his website, https://www.lemire.me/en/ https://www.lemire.me/en/ , and the "2%" brag is the very first sentence, geez. Being the top x% is what OnlyFans girls brag about, professor... And it's not exactly brain surgery, is it? https://www.youtube.com/watch?v=THNPmhBl-8I https://www.youtube.com/watch?v=THNPmhBl-8I
- Zetaphor 5mo ago> Daniel Lemire’s blog is one of the top 50 most popular blogs on Hacker News, the standard tech news aggregation site. Citation needed
- nkurz 5mo agoI agree that it sends the wrong symbol, but actually Daniel is great. He cares tremendously about doing work that is actually real-world useful. I've co-written a few papers with him, and he's really hard working and open to outside suggestions. The danger is that if you send him comments, he'll eventually manage to rope you into writing a new and improved version. Seriously, if you are a non-academic computer scientist with a good idea that you want to publish, he'd be incredibly open to working with you. As to why he now has this on his blog? I also cringe when I read it. I presume someone told him he should self-promote more, and this is his lame attempt to do so. He's almost certainly the most cited person in his department, but it's entirely possible that none of his colleagues actually know this. Cut him some slack. Self-promotion is not his strength. He's a nerd's nerd, and not a marketer. I'll mention to him that his attempt here might be backfiring when I'm next in contact with him.
- infecto 5mo agoI cringe calling it out but it just stood out as it was plastered everywhere and I actually have never seen his links before.
- iLoveOncall 5mo ago> As to why he now has this on his blog? He doesn't just have it on his blog, he has it EVERYWHERE. Sometimes 2 or 3 times on the same page.
- hgoel 5mo agoI kind of get it in the sense that every academic has to make themselves somewhat comfortable with self-promotion even if they don't like it. It's an important part of getting funding, but putting a blurb like that everywhere just hurts his credibility I think.
- SkiFire13 5mo agoThat lines looks very cringe indeed, but the guy has some crazy good blogposts on SIMD stuff.
- dgacmu 5mo agoHe's not a loser; he's done some really fun work that many people use daily. I've used his range mapping trick in multiple projects/papers. It's elegant. It sounds like he's gotten bad advise about how to market himself /or/ this is being marketed to people who have bigger checks to write and whom he believes will be responsive to this kind of marketing. As an academic, it rubs me very wrong - I think it's detrimental to the field when we get into h-index stacking contests or citation count comparisons. But I don't know what incentives he's responding to, which seems important for putting this stuff in context. (as an aside, it turns out that polars + fastexcel is about 10x faster than pandas + openpyxl for searching that dataset, if anyone else is curious what he was actually talking about. :)
- SwtCyber 5mo agoI think the local-model use case is going to become less niche pretty quickly if the models keep getting smaller and more capable. Even if most people do not care about privacy or offline use, the cost argument is pretty strong
- unmole 5mo ago> you may run some models locally if only from a cost perspective I have a hard time believing running a model on a laptop will be cheaper than running it in a datacenter. Why wouldn't economies of scale apply here as with every other computation?
- dgellow 5mo agoA laptop is really a pretty bad form factor to run LLMs. Worst cooling, more expensive memory that you cannot replace, resell value depreciating fast. It’s fine for tinkering, small scale research, and demos but it’s definitely niche. The vision NVIDIA is selling is pure marketing IMHO
- wazdra 5mo agoThis is assuming that you'll be priced the fraction of computing that you consumed. But you are actually paying for their infrastructure, for the R&D (and also the computation that went into training the model) etc. It is not clear that, for your own small computations, this kind of costs are needed, but you will still pay your share in the investment the provider made so that they could serve everyone's computation needs.
- wjnc 5mo agoIn that analogy bigtech AI is currently investing in cleaner air for all of us? We _could_ breath it through their hose, but might as well breath it outside.
- hungryhobbit 5mo agoBut, currently ... you're not. AI companies are operating at a loss, and are being subsidized by their investors. Local may or may not be cheaper than remote now, depending on the details, but the factors you describe won't affect the math nearly as much as they will once that subsidization ends.
- dannyw 5mo agoNot for API pricing. The latest models are not subsidised API wise anymore. Qwen3.6 is practically indistinguishable to Sonnet 4.6 at least in my personal experience. And sonnet 4.6 is not that cheap.
- sandworm101 5mo agoLots of people are already running AI locally. They are the people buying up all the consumer-grade nvidea gpus. What are they doing with them? Well, the same things people with home media or email servers are doing: stuff they dont want to share with the general public.
- Zetaphor 5mo agoI want to reduce my dependency on companies like Google, OpenAI, and Anthropic. Aside from the concerns of data sharing I'm also not a fan of how they run their operations, for example Anthropic now using xAI's Colossus data center which is poisoning a marginalized community, or OpenAI getting in bed with the military. Not everything I want to use an LLM for requires "PhD level intelligence", and increasingly I'm finding more uses that involve sharing my personal data. Yesterday my local model helped me when looking for a doctor who is in-network for my insurance. I threw it a screenshot from the providers search results and it looked up reviews for all of them.
- sandworm101 5mo agoMy local AI is currently upscaling an old british comedy from sub-dvd quality to 1k. (It is not availible other than on DVD.) It looks like it will take about a week for my pair of 5060s to chew through the task.
- eszed 5mo agoWhich show?
- sandworm101 5mo agoChelmsford 123 I own the DVDs so I'm OK upscaling/editing my own copies for my own use. But if I ran the task on an ai service I would no doubt trigger copyright issues.
- pratnala 5mo agoWhich model are you running?
- root-parent 5mo ago"I am not sure how many people will run AI models locally. It still seems like a niche application to me. However, it will make decent machines to play video games..." This is the 2026 edition of Ken Olsen: "There is no reason anyone would want a computer in their home"
- joering2 5mo agoor "640K ought to be enough for anybody."
- throw1234567891 5mo agoThere’s no public proof this has ever been said, and if it was, if it was not taken out of context.
- shermantanktop 5mo agohttps://quoteinvestigator.com/2011/09/08/640k-enough/ https://quoteinvestigator.com/2011/09/08/640k-enough/ Nobody ever said that, at least not as an assertion or prediction. The actual instances of similar language are from multiple people describing their earlier thoughts before they learned it wasn’t true.
- DonHopkins 5mo agoI have that many browser tabs.
- AaronAPU 5mo agoThat’s too strong of an assertion. Local models aren’t deterministically equivalent in capabilities to foundation models. Home computers are turing complete; just like a mainframe. They are just slower. Often not slower enough to matter.
- sandworm101 5mo agoMost people are ok with slower. An AI that lets you edit a family picture, in say 30 seconds, locally is preferable to one that is instantaneous but requires you to submit that picture to examination/storage/training/sale in someone else's AI ecosystem. If i want to crop my ex out of family photos, i should not have to first give that photo to Microsoft. If want an LLM to write a book report for me, i dont want it also alerting my school. And if i write a memo for a client, and i want an LLM to check the spelling, i dont want that memo leaked either.
- voidfunc 5mo ago> "Ranked in the top 2% of scientists globally (Stanford/Elsevier 2025) and among GitHub's top 1000 developers" This made me laugh. I can only image how insufferable this person is to deal with.
- flatline 5mo agoThe HN crowd is, by and large, not the target audience for his self promotion. I guarantee there is one and this is more or less effective.
- bespokedevelopr 5mo agoThe security aspect is the main driver why I’m seeing so many businesses investing in local hardware. They know the models aren’t as good (caveat that they also can’t run Chinese models) and that’s ok. Places that really care about security and data governance already aren’t on the bleeding edge. They wait for the nice stable lts version, they lock down dev machines in frustrating ways and have lots of IT admin layers. But they also want to taste the sweet fruit of AI so the only way to do this that a CISO will approve is on local air gapped hardware. It’s a niche but still a billion dollar niche.
- thewebguyd 5mo agoMicrosoft is working on this with their new execution containers (https://github.com/microsoft/mxc https://github.com/microsoft/mxc)
- cyanydeez 5mo ago128GB seems the sweet spot for local models. I can program and install most GitHub projects with opencode and QWEN 32b with mtp. anyone whose addicted to token theoughput is losing the operational knowledge and offline capabilities. if you arent moving to the AMD 395 or MACs then youre hitching aride on the expensive calory ride
- throw1234567891 5mo agoIf you could buy a 256GB you’d be claiming that 256GB is a sweet spot. But I agree with you. Crack-tokens are not the future.
- cyanydeez 5mo agono, the fact that MACs and x86 and soon ARM are all going to have 128GB models in every sector, yeah, sure. But watching everyone flounder because claude goes down or forcing you on API costs. I'm programming things that'd take me days with a PC that, without OpenAI's VRAM shenagans, would cost you $2k. It's more than just 'this is what I could do' it's definitely about 'this is what anyone could do with a new PC purchase'.
- throw1234567891 5mo agoYou must be unaware that System76 was already selling 192GB machines, mac studios used to be 512GB max. The only reason why we don’t have them anymore is that we are in RAM shortage.
- cyanydeez 5mo agoI'm aware you can have more. the term "SWEET SPOT" references a area that anyone/everyone can get to and isn't some magical expensive unicorn. You're doing what the IT industry has been addicted to for decades: number goes up.
- throw1234567891 5mo ago
- smcleod 5mo agoQwen 3.6 is far ahead of Gemma for most (but not all) things. I've deployed it out across a number of M5 MacBooks and it's genuinely useful for many tasks. It won't replace an Opus or current gen Sonnet sized model but it's still amazingly good for its size and probably as good as or just a bit before Sonnet 4 era. Far more reliable for tool calling, coding, agentic tasks and faster than the Gemma models especially with MTP.
- Pxtl 5mo agoI've got a Qwen 3.5 running on a 12GB 3060 and it's dumb as a stump but still smart enough to get some useful work done. Since it's my daily driver desktop I havent jumped to 3.6 since last time I did I quickly ran out of vram and locked the desktop environment. But yeah, the Qwen line is pretty impressive on commodity hardware.
- derefr 5mo agoI must be using LLMs very differently than y'all, because I can't think of a single thing I would rely on an LLM that's "dumb as a stump" to do for me. To me, LLMs are for asking research questions + exploring design spaces + pointing at codebases to investigate bugs. And those all benefit from the model being as "smart" (in terms of both fluid intelligence and burned-in knowledge) as possible. I'm guessing there exist problems where "intelligence past a certain point" doesn't matter, so these medium-sized models can match the performance of the bigger models. But what problems might those be?
- Pxtl 5mo agoThings that are tedious but simple but I'm unfamiliar with. "Go add a gh action to compile and deploy this thing and run its tests" is one I've found it's good at. Yes I know how to make a gh pipeline but it's always a hassle to remember what goes where. Cranking out unit tests is okay. It's good at summarizing things so it's not half bad at writing jsdoc/xmldoc comments.
- zozbot234 5mo agoQwen 3.6 is a toy compared to DeepSeek V4 Flash or Pro. These models can now run on Apple Silicon hardware with as little as 32GB RAM for the Flash (with 2-bit quant, which is still quite capable) using SSD offloading, with just-about-reasonable performance for interactive use, and far better performance on longer contexts than Qwen (due to the more efficient KV cache/attention mechanisms in DeepSeek). Very significant improvements may be viable for unattended inference via large-scale batches, which can reuse sparse experts and thereby mask some of the latency involved - this is quite unique to DeepSeek, again due to its efficient KV cache.
- unstatusthequo 5mo agoI hope a family-level AI appliance is a thing later. Local non-cloud assistant that lives in the house, families interact via voice or phones or whatever. Knows the contextual family stuff you need, etc.
- Pxtl 5mo agoWe didn't get people buying family-level file servers for the family photo gallery and documents at any real scale, so i doubt we'll see similar for AI especially when the cost is that much higher for GPUs vs an SBC machine.
- JMiao 5mo agobecause nas hardware and software suck and everything else was a poorly executed subscription product...i think one was called helm, another was by early twitter alumni. imagine a home device that manages and maintains itself and is a joy to interact with.
- jayd16 5mo agoMaybe they just mean from a "it can run a lot of DLSS" perspective.
- falsemyrmidon 5mo ago> this guy puts this everywhere, gives me probably the inverse of what he is marketing for. Do you think he's in mensa too?
- jb1991 5mo agoHe’s just a braggart. When you see something like this in somebody’s personal bio on social media, it’s basically a banner that means “take everything I say in the context of me promoting myself.”
- GeekyBear 5mo ago> However, it will make decent machines to play video games." Where you will need games to be rewritten for ARM to get full performance, just like on Apple's M series chips.
- epolanski 5mo agoDeepSeek Flash v4 is the leading local AI on 128GB machines, and DS4 is still in preview (training not finished), no? Especially on Dwarfstar.
- strictnein 5mo agoI also don't get why this twitter user is linked here, versus all the news articles about this new hardware that have been everywhere over the past number of days.
- latch 5mo agoI also dislike his self-promotion, but his work _is_ well know and, as far as I know, well looked upon. I think he has more expertise and knowledge in this area than most (including what you'd find in the news).
- strictnein 5mo agoAhh, thanks, that explains things a little more. I wasn't familiar with the author, and his tweet just read like one of those people on Linkedin who regurgitates knowledge and passes it off as their own insight.
- bredren 5mo ago> not even considering business security. I suspect personal privacy and need to run AI workflows to handle the litany of administration tasks of a household will be what result in regular need for local AI. Apple is already out front with this on a personal, individual level, but they are not obviously headed toward multiuser/family-level ~biz admin with a persistent server running local LLM.
- deleted 5mo ago[deleted]