7 ms·
This isn't a good analysis, and it's because it keeps rounding everything up. He rounds up the cost of electricity by 10%. He has a range of power use, takes th
by bastawhiz 5mo ago
This isn't a good analysis, and it's because it keeps rounding everything up. He rounds up the cost of electricity by 10%. He has a range of power use, takes the high end (which is 2x the low end) and multiplies it by the inflated electricity cost.
But then they talk about using a newly purchased Mac to do the inference, running at full capacity, 24/7. Why would you do that? Apple silicon is fast but the author points out: you're only getting 10-40 tokens per second. It's not bad, but it's not meant for this!
It's comparing apples to oranges. Yeah, data centers don't pay residential electricity rates. Data centers use chips that are power efficient. Data centers use chips that aren't designed to be a Mac.
Apple silicon works out pretty good if you're not burning tokens 24/7/365 and you're not buying hardware specifically to do it. I use my Mac Studio a few times a week for things that I need it for, but I can run ollama on it over the tailnet "for free". The economics work when I'm not trying to make my Mac Studio behave like a H100 cluster with liquid cooling. Which should come as no surprise to anyone: more tokens per watt on hardware that's multi tenant with cheap electricity will pretty much always win.
- datadrivenangel 5mo agoRounding everything down in the most optimistic setting got me to $0.40 per million tokens, and openrouter has the same model at $.38/mtok.
- 650REDHAIR 5mo agoI’ll keep my data local over a $.02/mtok difference.
- quietsegfault 5mo agoIt’s more than just data locality. OpenRouter is faster, no? I have an M4 pro, and anything but the smallest dumbest models are unusably slow for interactive use. I personally haven’t yet found a good use case for offline/non-interactive LLM work locally.
- datadrivenangel 5mo agoYeah. The speed is the biggest issue. The intelligence of open models is good enough for serious work (though still worse than the frontier models), but the cloud models are often 3-7 times faster, and you can get more parallelization and so get speeds on the order of hundreds of tokens per second, which makes things fast!
- freeopinion 5mo agoEven extremely slow LLMs can generate Part B faster than I can audit Part A. So the LLM can generate Part A while I look over my email. Then it can worry over Part B while I look over Part A. It can worry over Part C while I have my 10:30 group meet. And it can worry over Part D while I do whatever other silly, time-wasting thing all humans do in almost all organizations. Then I still haven't reviewed Part B, yet, so the extremely slow AI is waiting on me. Maybe someday I'll be good enough to need faster AI so I can rewrite something like Bun in a few days. Right now, slow and local fits my use case very well.
- quietsegfault 5mo agoI don’t think it matters if you’re “good enough” or not. Much of AI development is iterative. If you context switch between A from project 1 to B from project 2 back to check A, then maybe C while B finishes up, you will lose the flow state that AI assistance can enable with speed for those who are not fluent coders. Sure, I can wait hours for my local model to finish, or I can spend basically as much and get the answer right away There’s a lot of exciting stuff with local LLMs despite the speed, but for me I don’t have the discipline and working memory to jump from project to project.
- bcjdjsndon 5mo ago> Even extremely slow LLMs can generate Part B faster than I can audit Part A. Depends what part a and b are...because workloads are different
- threatofrain 5mo ago
- formerly_proven 5mo agoWhat is it with AI SaaS naming themselves "openxyz" when there is 0% open about them?
- em500 5mo agoThey learnt from ooenai that naming yourself open-xyz doesn't actually require opening anything.
- debugnik 5mo agoIt's the next co-opted buzzword after "democratize".
- throwaway2037 5mo agoYeah, anytime that I see "People's" or "Democratic" in the name of a nation, I grow suspicious. It is rarely a well-functioning democracy.
- brikym 5mo agoIt's how marketing works. If something is a problem they have to loudly claim to have fixed it. Look around the economy and you'll see lots of it. "Healthy" (high sugar) muesli bars, clean-diesel, surveillance wrapped up as keeping us safe. The modus operandi of marketing is to change minds about self evident things otherwise what is the point?
- nativeit 5mo agoBut once all that is done you still own a Mac in one case, and you don’t in the other, correct?
- odo1242 5mo agoYea this; it’s the same reason why mortgaging is cheaper than renting
- ericpauley 5mo agoThis is far from a universal truth: https://www.nytimes.com/interactive/2024/upshot/buy-rent-calculator.html https://www.nytimes.com/interactive/2024/upshot/buy-rent-cal... Real estate is only a clearly good investment if you ignore opportunity cost.
- seanmcdirmid 5mo agoYou also need to pay close attention to rent vs purchase ratios. A lot of cities are cheap to rent but expensive to buy (eg beijing 10 years ago).
- mantas 5mo agoKey word being „ago“.
- deaux 5mo agoSuch cities still exist and have been in such a state for decades. They can change but that's meaningless as they can also change the other way around.
- seanmcdirmid 5mo agoI’m covering my bases because Chinese real estate has been volatile recently and I’m not sure where the market is at now. It could be that renting is still way cheaper than buying, I just don’t have any direct experience to back that up. If I bought while I was living in Beijing I would probably be underwater with my investment right now, renting for 9 years was the right call and my rent was pretty affordable anyways.
- novok 5mo agoAlso many have power even cheaper or even free unused surplus power with solar. I don't do local inference other than hobby & learning reasons because electricity is so expensive where I am at.
- dist-epoch 5mo agousing it 24/7 brings the average cost down, not up. the less you use local LLM, the less sense it makes since you paid a lot for hardware you don't use
- groundzeros2015 5mo agoThe hardware has multiple uses for the same cost. The pay-per-use server does not.
- bastawhiz 5mo agoThe author isn't pricing in the multiple uses. You either compare it apples to apples or you don't. If you're using the machine for general purpose computing on top of inference then the amortized hardware costs are pointless to measure. This is exactly what I said.
- groundzeros2015 5mo agoOk you can resell it at the end
- bastawhiz 5mo agoThat's the point: why would you buy a device that's specifically not optimized to be used for 24/7 inference? It's expensive hardware that's not designed to be used in that situation! The power use for inference isn't especially good and you're not getting even a fraction of the benefit from the hardware that you're paying for.
- dist-epoch 5mo ago> why would you buy a device that's specifically not optimized to be used for 24/7 inference because it costs $1k-$2k instead of $10k-30k+ for optimized devices
- bastawhiz 5mo ago
- cyanydeez 5mo agonothing about the current data center craze looks efficient.
- bastawhiz 5mo agoWhether you think building data centers or not is a good idea it's inarguable that the per-token efficiency (power, hardware, etc) is FAR higher in a data center. That's literally what it's designed for.
- cyanydeez 5mo agoim talking per value. look at the efgiency of chinese open source models; then look at SOTA sucking gigawatts, then the proposals. America is basically proposing AI using the equivalent bloatware of Windows 11.
- bastawhiz 5mo agoI run two 49B parameter models on a pair of used A100s full time and it sucks down 250 watts at peak utilization. That's not gigawatts, and it's completely within the realm of comparison to what the author is describing.
- trollbridge 5mo agoProbably because lots of data centres are being built (or half-built) which are sitting idle.
- faitswulff 5mo agoThe article makes no sense. I can't use OpenRouter as a general purpose computing device. Why are we comparing a whole computer to a single purpose SaaS?
- tuwtuwtuwtuw 5mo agoI think it's because there are a lot of people writing articles about the benefits of running local models. I think it's fair to say that there are daily threads on HN singing the praises or local inference. I also see people buying new hardware where the main trigger is ability to run local models.
- FuckButtons 5mo agoBut the people who want to do local inference are putting some amount of value on privacy that’s not captured by the raw monetary value so just comparing the price is somewhat beside the point, it’s also true that, if you have eg a Mac and you use that as your main computing device then you would have spent money on it anyway, so you can’t even really compare its value to spend on something that’s not general purpose.
- datadrivenangel 5mo agoMy overall opinion is that the smart thing is not to upgrade to the maximum memory for AI purposes. It's worth quantifying how much extra we pay for privacy.
- tuwtuwtuwtuw 5mo agoI replied to a comment asking why the article exists. As for privacy, I'm sure there are many people that are not so interested in that aspect.
- apf6 5mo agoThat's a lot of assumptions. I think there are also people buying new hardware specifically for this purpose, and their motivation to do it is thinking it will be cheaper in the long run. Privacy is not necessarily the motivation.
- statestreet123 5mo agoRounded up, yes, and oddly inefficient for someone obsessed with inefficiency. One could buy a brand new 64gb M5 macbook for well over 4k. Another could buy a scratched up but functioning M1 Max 64gb off of ebay for a little over 1k—and somehow get the same 10-20 t/s with 31b that the author does with an M5. Or better yet, have a frontier model do the planning and judging, and have a local MOE model execute at 50 t/s. All of this achievable by a former English major with too much free time.
- novok 5mo agoI have an M1 Pro, and a M4 & M5 max to play with at work and the speed difference is very significant between all 3 machines, the M1 Pro is far slower, and the M5 is significantly faster than the M4. And a windows 3090 beats all of them but eats twice the amount of power per token. This is all running the same 24GB memory friendly model with LM studio.
- llm_nerd 5mo agoYour post makes sense if you bought the hardware for other reasons, and maybe run models occasionally as a novelty. That isn't the case for many, though, and there is a whole social media space where people are hyping up the latest homebrew options for running models, believing it frees them from the yoke of big AI. Millions of people are buying big $ maxed-out hardware like the Mac Studios or DGX specifically to run LLMs. Someone rationally running the numbers is a good thing.
- atq2119 5mo agoLet's not get ahead of ourselves. Millions, really? I can believe there are a lot of enthusiasts doing this, but "millions" needs a citation.
- filleduchaos 5mo agoThis is HN; it has probably never occurred to half the people here that the average person even in first world countries doesn't even have the financial capacity to make an impulse five-figure USD purchase, even if on credit.
- llm_nerd 5mo agoNo one said anything about this being an "impulse" purchase -- it would usually be perceived as a "career investment" purchase, given that many feel they need to race to be with "it" or be left behind -- nor does "five figure" have anything to do with this -- DGX options are available at $3000 -- there's a certain irony that you are posting a comment that is basically "people in my country/circle can't, therefore no one can", while dismissing my comment that many people can. Millions of people are paying thousands of dollars a year to buy a slightly upgraded entertainment package in their car. There are 60 million or so millionaires alone, including 6 million+ in China. There are a lot of people with a lot of wealth on the planet. A lot. Millions...it isn't that unfounded, friend. So doing this "this is HN" snide jerk act, and then basically projecting your lot on the planet is...I don't know if you intended it, but it's rather amazing.
- 5mo ago
- espadrine 5mo ago[dead]
- giancarlostoro 5mo agoHonestly, I don't even see my Macbook Pro costing me anywhere near as much as using any of these AI services, but maybe I'm just not seeing a significant increase in my power bill to notice? I am the power user who uses Claude Max pretty much all the time to prototype ideas, and build things I actually use, and has given me a lot of value, I work full time and have a family to raise and care for, my free coding time is mostly limited to ideas. Now I can draft a plan with detail, review the code, run the code, test it, and use software custom tailored to my needs.
- PunchyHamster 5mo ago> Yeah, data centers don't pay residential electricity rates. There are 2 caveats here: Some places have higher prices for industrial than residential power as residential one might be subsidied by govt. And DC also pay for cooling, which residential will only effectively pay if they have AC and is hot outside. So power rates are some multiply of industrial pricing.
- bastawhiz 5mo agoGenerally you don't build a data center in a place that doesn't sell you electricity for cheap
- PunchyHamster 5mo agoStill for purpose of calculation you still need to calculate the price of both power and removing heat created by that power.
- make3 5mo agoThe real reason this comparison makes no sense is that only a vanishingly small fraction of people seriously using ai to code would seriously use a model so far from the top models (including open source ones). He should compare his MacBook to Open Router on Kimi 2.6 1.1T or GLM 5.1 (754B), at bfloat16 precision, which he can't ofc. But it furthers his point that things like open router are a better idea, which is not surprising.
- avidphantasm 5mo agoNot sure where 40 tokens per second is coming from. I’ve seen 95-100 tokens per second on M5 Max 128GB running Gemma 4 31B. I’ve done experiments where it is faster than Claude Opus 4.5 for the same prompts.
- dhiraj_bhakta_ 5mo agocan you provide your configurations pls ?
- avidphantasm 5mo agoIt's actually a bit faster than that now it seems, about 112 tok/sec. Configuration: Gemma 4 31B Instruct Q6K Context size 40960 LM Studio 0.4.13+1 Metal llama.cpp v2.14.0 LM Studio MLX (Apple M5) v1.6.0 Here are my results: prompt eval time = 32545.36 ms / 5625 tokens ( 5.79 ms per token, 172.84 tokens per second) eval time = 20227.99 ms / 310 tokens ( 65.25 ms per token, 15.33 tokens per second) total time = 52773.35 ms / 5935 tokens This was for interacting with a local MCP service, running a tool that returns a ~20KB text file to the agent to add to the chat context. I'm seeing about the same number of tokens/second on an M2 Ultra that I have access to (also with 128GB of memory). This is surely apples-to-oranges to the OP results (and I don't spend a great deal of time benchmarking these things, so my methodology might be lacking), but it's interesting seeing okay performance for a top open model. For most use, however, I find Gemma 4 26B A4B (Q6K) to be good enough (esp. for MCP calling) and much much faster (~1,200 tokens/second).
- throwaway2037 5mo ago> M5 Max 128GB Wild. That must be like a 5,000 USD laptop.
- ikidd 5mo agoActually, figuring it on generating tokens 24/7 is the best case scenario. if you figure it at 8 hours a day of actual use, you still have the fixed cost of the hardware being the highest portion of the budget, but now you generate 1/3 the tokens so you triple that cost per token.
- econ 5mo agoBoss, I make 16.50 per hour, say 15, I work 36 hours, say 35, say 500 per week, say 4 weeks per month, that's only about 2000! Don't you agree I need a raise?
- outside1234 5mo agoWe also have no idea what it actually costs Anthropic. This could be wildly subsidized and actually Apple Silicon is more cost effective.