6 ms·
Online, R1 costs what, $2/MTok? This rig does >4 tok/s, which is ~15-20 ktok/hr, or $0.04/hr when purchased through a provider. You're probably spending $0.20
by isoprophlex 2y ago
Online, R1 costs what, $2/MTok?
This rig does >4 tok/s, which is ~15-20 ktok/hr, or $0.04/hr when purchased through a provider.
You're probably spending $0.20/hr on power (1 kW) alone.
Cool achievement, but to me it doesn't make a lot of sense (besides privacy...)
- rightbyte 2y ago> Cool achievement, but to me it doesn't make a lot of sense (besides privacy...) I would argue that is enough and that this is awesome. It was a long time ago I wanted to do a tech hack like this much.
- isoprophlex 2y agoWell thinking about it a bit more, it would be so cool if you could A) somehow continuously interact with the running model, ambient-computing style. Say have the thing observe you as you work, letting it store memories. B) allowing it to process those memories when it chooses to/whenever it's not getting any external input/when it is "sleeping" and C) (this is probably very difficult) have it change it's own weights somehow due to whatever it does in A+B. THAT, in a privacy friendly self-hosted package, i'd pay serious money for
- Willingham 2y agoI imagine it could solve crimes if it watched millions of hours of security footage…scary thought. Possibly it could arrest us before we even commit a crime through prediction like that black mirror episode.
- falcor84 2y agoOh, you're thinking of "Hated in the Nation"? More relevant would possibly be "Minority Report" (set in 2054) and Hulu's "Class of '09", in which the FBI starts deploying a crime prediction AI in their version of 2025. Quite scary. As the meme has it, it seems that we're getting ready to create the Torment Nexus from classic sci-fi novel Don't Create The Torment Nexus.
- codetrotter 2y ago> doesn't make a lot of sense (besides privacy...) Privacy is worth very much though.
- infecto 2y agoDefinitely but when you can run this in places like Azure with tight contracts it makes little sense except for the ultra paranoid.
- gloflo 2y agoConsidering the power of three letter agencies in the USA and the complete unhingedness of the new administration, I would not trust anything to a contract.
- rsanek 2y agocan we even trust the hardware?
- sambull 2y agono, just being speculative... about that.
- WiSaGaN 2y agoWell you don't need to worry unless you are already on the list.
- HeatrayEnjoyer 2y agoThese days getting on a list may require as little as "is trans" or "has immigrant parents."
- lyu07282 2y agoor "your competitor donated to Musk/Trump campaign"
- topbanana 2y agoThe point is running locally, not efficiently
- api 2y agoHow is it that cloud LLMs can be so much cheaper? Especially given that local compute, RAM, and storage are often orders of magnitude cheaper than cloud. Is it possible that this is an AI bubble subsidy where we are actually getting it below cost? Of course for conventional compute cloud markup is ludicrous, so maybe this is just cloud economy of scale with a much smaller markup.
- agieocean 2y agoIsn't that just because they can get massive discounts on hardware buying in bulk (for lack of a proper term) + absorb losses?
- raducu 2y agoAll that, but also because they have those GPUs with crazy amounts of RAM and crazy bandwidth? So the TPS is that much higher, but in terms of power, I guess those boards run in the same ballpark of power used by consumer GPUs?
- realusername 2y agoIt's cheaper because you are unlikely to run your local AI at top capacity 24/7 so you have unused capacity which you are paying for.
- NeutralCrane 2y agoThey are specifically referring to usage of APIs where you just pay by the token, not by compute. In this case, you aren’t paying for capacity at all, just usage.
- davrosthedalek 2y agoThe calculation shows it's cheaper even if you run local AI 24/7
- octacat 2y agoIt is shared between users and better utilized and optimized.
- ekianjo 2y ago> (besides privacy...) that's the whole point of local models
- bloomingkales 2y ago"besides privacy" lol. Yeah, just besides that one little thing. We really are a beaten down society aren't we.
- 7thpower 2y agoThere is something about this comment that is so petty that I had to re-read it. Nice dunk, I guess.
- brookst 2y agoPrivacy is a relatively new concept, and the idea that individuals are entitled to complete privacy is a very new and radical concept. I am as pro-privacy as they come, but let’s not pretend that government and corporate surveillance is some wild new thing that just appeared. Read Horace’s Satires for insight into how non-private private correspondence often was in Ancient Rome.
- atq2119 2y agoIt's a bit of both. Village societies don't have a lot of privacy. But they also don't make it possible for powerful individuals to datamine personal information of millions. Most of us have more privacy than 200 years ago in some ways, and much less privacy in other ways.
- Aurornis 2y agoMost people value privacy, but they’re practical about it. The odds of a cloud server leaking my information is non-zero, but it’s very small. A government entity could theoretically get to it, but they would be bored to tears because I have nothing of interest to them. So practically speaking, the threat surface of cloud hosting is an acceptable tradeoff for the speed and ease of use. Running things at home is fun, but the hosted solutions are so much faster when you actually want to get work done. If you’re doing some secret sensitive work or have contract obligations then I could understand running it locally. For most people, trying to secure your LLM interactions from the government isn’t a priority because the government isn’t even interested. Legally, the government could come and take your home server too. People like to have fantasies about destroying the server during a raid or encrypting things, but practically speaking they’ll get to it or lock you up if they want it.
- matja 2y agoHow would it use 1kW? Socket SP3 tops at 280W and the system in the article has a 850W PSU so I'm not sure what I'm missing.
- falcor84 2y agoI assume that the parent just rounded 850W up to 1kW, no?
- isoprophlex 2y agoYeah i was vigorously waving hands. Even at 200W, 10 cents/kWh you'd need to run this a LONG time to break even
- jpc0 2y agoYou could absolutely install 2kw of solar for probably around 2-4k and then at worst it turns your daytime usage into 0$. I also would be surprised if this was pulling 1kw in reality, I would want to see an actual measurement of what it is realistically pulling at the wall. I believe it was an 850w PSU on the spec sheet?
- dboreham 2y agoQuick note that solar power doesn't have zero cost.
- madduci 2y agoAnd in winter, depending on the region, it might generate 0kW
- isoprophlex 2y agoOr, in my case, currently 32.9 W.
- SV_BubbleTime 2y agoDon’t worry, you can charge an iPhone!
- bee_rider 2y agoIt could have zero marginal cost, right? In particular, if you over-provisioned your solar installation already anyway, most of the time it should be producing more energy than you need.
- _w1tm 2y agoAt that point you’re still paying the opportunity cost by losing out on selling your surplus.
- sieabahlpark 2y ago
- WiSaGaN 2y agoI think the main point of local model is privacy set aside hobby and tinkering.
- rufus_foreman 2y agoPrivacy, for me, is a necessary feature for something like this. And I think your math is off, $0.20 per kWh at 1 kW is is $145 a month. I pay $0.06 per kWh. I've got what, 7 or 8 computers running right now and my electric bill for that and everything else is around $100 a month, at least until I start using AC. I don't think the power usage of something like this would be significant enough for me to even shut it off when I wasn't using it. Anyway, we'll find out, just ordered the motherboard.
- rodonn 2y agoDepends on where you live. The average in San Francisco is $0.29 per kWh.
- deleted 2y ago[deleted]
- michpoch 2y ago> I pay $0.06 per kWh That is like, insanely cheap. In Europe I'd expect prices between $0.15 - 0.25 per kWh. $0.06 sounds like you live next to some solar farm or large hydro installation? Is that a total price, with transfer?
- rufus_foreman 2y agoSo one thing, it's the winter rate, summer rate is higher when people run their AC and the energy company has to use higher cost sources. Second, it's a tiered rate, first x amount of kWh is a higher rate, then once you reach that amount it's a lower rate. But I'm already above the tier cutoff every month no matter what, so marginal winter rate is around $0.06.
- onlyrealcuzzo 2y ago> You're probably spending $0.20/hr on power (1 kW) alone. For those that aren't following - means you're spending ~$10/MTok on power alone (compared to $2/MTok hosted).
- carbonbioxide 2y agoI think the privacy should be the whole point. There's always a price to pay. I'm optimistic that soon you'll be able to get better speeds with less hardware.
- spaceport 2y agoThe system idles at 60w and running hits 260w.
- magic_hamster 2y agoThis gets you the (arguably) most powerful AI in the world running completely privately, under your control, in around $2000. There are many use cases for when you wouldn't want to send your prompts and data to a 3rd party. A lot of businesses have a data export policy where you are just not allowed to use company data anywhere but internal services. This is actually insanely useful.
- jaggs 2y agoI think you may be underestimating future enshitification? (e.g. it's going to be trivially easy for the cloud suppliers to cram ads into all the chat responses at will).