4 ms·
I'm curious what the mental calculus was that a $5k laptop would competitively benchmark against SOTA models for the next 5 years was. Somewhat comically, the
by Workaccount2 10mo ago
I'm curious what the mental calculus was that a $5k laptop would competitively benchmark against SOTA models for the next 5 years was.
Somewhat comically, the author seems to have made it about 2 days. Out of 1,825. I think the real story is the folly of fixating your eyes on shiny new hardware and searching for justifications. I'm too ashamed to admit how many times I've done that dance...
Local models are purely for fun, hobby, and extreme privacy paranoia. If you really want privacy beyond a ToS guarantee, just lease a server (I know they can still be spying on that, but it's a threshold.)
- ekjhgkejhgk 10mo agoI agree with everything you said, and yet I cannot help but respect a person who wants to do it himself. It reminds me of the hacker culture of the 80s and 90s.
- slicktux 10mo agoAgreed, Everyone seems to shun the DIY hacker now a days; saying things like “I’ll just pay for it”. It’s not about just NOT paying for it but doing it yourself and learning how to do it so that you can pass the knowledge on and someone else can do it.
- davidw 10mo agoI loathe the idea of being beholden to large corporations for what may be a key part of this job in the future.
- Eupolemos 10mo agoAnd we all know that enshittyfication is coming.
- ekjhgkejhgk 10mo agoExactly. Google doesn't show you what it knows is the most appropriate answer, it shows you a compromise between the most appropriate answer and the one that makes them the most money. Same thing will happen with these tools, just a matter of time.
- ryandrake 10mo agoAnd, it's not just about "pay for it" vs. "don't pay for it". It's about needing to pay for it monthly or it goes away. I hate subscriptions. They sneak their way into your life, little by little. $4.99/mo here. $9.99/mo there. $24.99/yr elsewhere. And then at some point, in a moment of clarity, you wake up and look at your monthly expenses and notice you're paying a fortune just to exist in your life as you are existing. I'm not going to pay monthly for X service when similar Y thing can be purchased once (or ideally open source downloaded), self-hosted, and it's your setup forever.
- ekjhgkejhgk 10mo ago> or ideally open source downloaded Ideally Free software downloaded. Even more ideally copyleft Free software downloaded.
- satvikpendem 10mo ago> I'm curious what the mental calculus was that a $5k laptop would competitively benchmark against SOTA models for the next 5 years was. Well, the hardware remains the same but local models get better and more efficient, so I don't think there is much difference between paying 5k for online models over 5 years vs getting a laptop (and well, you'll need a laptop anyway, so why not just get a good enough one to run local models in the first place?).
- brulard 10mo agoIf you have inference running on this new 128GB RAM Mac, wouldn't you still need another separate machine to do the manual work (like running IDE, browsers, toolchains, builders/bundlers etc.)? I can not imagine you will have any meaningful RAM available after LLM models are running.
- satvikpendem 10mo agoNo? First of all you can limit how much of the unified RAM goes into VRAM, and second, many applications don't need that much RAM. Even if you put 108 GB to VRAM and 16 to applications, you'll be fine.
- brulard 10mo agoHow about the rest of the resources? CPU/GPU? Would your work not be affected by inference running?
- satvikpendem 10mo agoAI doesn't really use much CPU. In a simple answer, no your work would not be affected.
- Workaccount2 10mo agoEven if intelligence scaling stays equal, you'll lose out on speed. A sota model pumping 200 tk/s is going to be impossible to ignore with a 4 year old laptop choking itself at 3 tk/s. Even still, right now is when the first gen of pure LLM focused design chipsets are getting into data centers.
- smcleod 10mo agoMy 2023 Macbook Pro (M2 Max) is coming up to 3 years old and I can run models locally that are arguably "better" than what was considered SOTA about 1.5 years ago. This is of course not an exact comparison but it's close enough to give some perspective.
- menaerus 10mo agoOpenAI released GPT-4o in May 2024, and Anthropic released Claude 3.5 Sonnet in June 2024. I haven't tried the local models as much but I'd find it difficult to believe that they would outperform the 2024 models from OpenAI or Anthropic. The only major algorithmic shift was done towards the RLVR and I believe it was already being applied during the 2023-2024.
- Aurornis 10mo agoI don't know about that. Even trying Devstral 2 locally feels less competent than the SOTA models from mid-2024. It's impressive to see what I can run locally, but they're just not at the level of anything from the GPT-4 era in my experience.
- wyldfire 10mo agoIs that really the case? This summer there was "Frontier AI performance becomes accessible on consumer hardware within a year" [1] which makes me think it's a mistake to discount the open weights models. [1] https://epoch.ai/data-insights/consumer-gpu-model-gap https://epoch.ai/data-insights/consumer-gpu-model-gap
- hu3 10mo agoOpen weight models are neat. But for SOTA performance you need specialized hardware. Even for Open Weight models. 40k in consumer hardware is never going to compete with 40k of AI specialized GPUs/servers. Your link starts with: > "Using a single top-of-the-line gaming GPU like NVIDIA’s RTX 5090 (under $2500), anyone can locally run models matching the absolute frontier of LLM performance from just 6 to 12 months ago." I highly doubt a RTX 5090 can run anything that competes with Sonnet 3.5 which was released June, 2024.
- Lapel2742 10mo ago> I highly doubt a RTX 5090 can run anything that competes with Sonnet 3.5 which was released June, 2024. I don't know about the capabilities of a 5090 but you probably can run a Devstral-2 [1] model locally on a Mac with good performance. Even the small Devstral-2 model (24b) seems to easily beat Sonnet 3.5 [2]. My impression is that local models have made huge progress. Coding aside I'm also impressed by the Ministral models (3b, 8b and 14b) Mistral AI released a a couple of weeks ago. The Granite 4.0 models by IBM also seem capable in this context. [1] https://mistral.ai/news/devstral-2-vibe-cli https://mistral.ai/news/devstral-2-vibe-cli [2] https://www.anthropic.com/news/swe-bench-sonnet https://www.anthropic.com/news/swe-bench-sonnet
- cmrdporcupine 10mo agoThing is you can pay basically fractions of cents a query to e.g. DeepSeek Platform or DeepInfra or Z.Ai or whatever and have them run the same open models for far cheaper and faster than you could ever build out at home. It's neat to play with, but not practical. The only story that I can see that makes sense for running at home is if you're going to fine tune a model by taking an open weight model and <hand waving> doing things to it and running that. Even then I believe there's places (hugging face?) that will host and run your updated model for cheaper than you could run it yourself.
- ekianjo 10mo agoThat's the kind of attitude that removes power from the end user. If everything becomes SAAS you don't control anything anymore.
- thefourthchime 10mo agoI completely agree. I can't even imagine using a local model when I can barely tolerate a model one tick behind SOTA for coding.
- littlestymaar 10mo ago> Local models are purely for fun, hobby, and extreme privacy paranoia I always find it funny when the same people who were adamant that GPT-4 was game-changer level of intelligence are now dismissing local models that are both way more competent and much faster than GPT-4 was.
- ashirviskas 10mo agoMoon lander computers were also game changers. Does not mean I should be impressed by the compute of a 30 year old calcualator that is 100x more powerful/efficient in 2025 when we have stuff a few orders of magnitude better. For simple compute, its usefulness curve is a log scale. 10x faster may only be 2x more useful. For LLMs (and human intelligence) its more quadratic, if not inverse log (140IQ human can do maths that you cannot do with 2x 70IQ humans. And I know, IQ is not a good/real metric, but you get the point)
- littlestymaar 10mo ago30-years old calculators are still good enough for basic arithmetic and in fact even in 2025 people have one emulated on their phone that isn't more powerful than the original, and people still use them routinely. If Claude 3 Sonnet was good enough to be your daily driver last year, surely something that is as powerful is good enough to be your daily driver today. It's not like the amount of work you must do to get paid doubled over the past year or anything. Some people just feel the need to live always on the edge for no particular reason.
- ashirviskas 10mo agoClaude 3 Sonnet was good enough for many things, but not as universal as 4.5 Opus. It is immeasureably more useful to me. I agree with all of your other points though.
- unethical_ban 10mo agoThere is a cultural component to privacy, and insulting people who want a technical assurance that their queries are under their own control is a way to work against privacy.