5 ms·
Rounding everything down in the most optimistic setting got me to $0.40 per million tokens, and openrouter has the same model at $.38/mtok.
by datadrivenangel 5mo ago
Rounding everything down in the most optimistic setting got me to $0.40 per million tokens, and openrouter has the same model at $.38/mtok.
- 650REDHAIR 5mo agoI’ll keep my data local over a $.02/mtok difference.
- quietsegfault 5mo agoIt’s more than just data locality. OpenRouter is faster, no? I have an M4 pro, and anything but the smallest dumbest models are unusably slow for interactive use. I personally haven’t yet found a good use case for offline/non-interactive LLM work locally.
- datadrivenangel 5mo agoYeah. The speed is the biggest issue. The intelligence of open models is good enough for serious work (though still worse than the frontier models), but the cloud models are often 3-7 times faster, and you can get more parallelization and so get speeds on the order of hundreds of tokens per second, which makes things fast!
- freeopinion 5mo agoEven extremely slow LLMs can generate Part B faster than I can audit Part A. So the LLM can generate Part A while I look over my email. Then it can worry over Part B while I look over Part A. It can worry over Part C while I have my 10:30 group meet. And it can worry over Part D while I do whatever other silly, time-wasting thing all humans do in almost all organizations. Then I still haven't reviewed Part B, yet, so the extremely slow AI is waiting on me. Maybe someday I'll be good enough to need faster AI so I can rewrite something like Bun in a few days. Right now, slow and local fits my use case very well.
- quietsegfault 5mo agoI don’t think it matters if you’re “good enough” or not. Much of AI development is iterative. If you context switch between A from project 1 to B from project 2 back to check A, then maybe C while B finishes up, you will lose the flow state that AI assistance can enable with speed for those who are not fluent coders. Sure, I can wait hours for my local model to finish, or I can spend basically as much and get the answer right away There’s a lot of exciting stuff with local LLMs despite the speed, but for me I don’t have the discipline and working memory to jump from project to project.
- bcjdjsndon 5mo ago> Even extremely slow LLMs can generate Part B faster than I can audit Part A. Depends what part a and b are...because workloads are different
- threatofrain 5mo agoAnd continuing the argument of "more than just...", if you stopped inferencing on your Mac you still have a generally nice computer. The difference between rent vs buy.
- novok 5mo agoI played with classifying and summarizing my entire email history (per email) with small models, but that only took about 12h of GPU time at most. Using a coding agent cli wrapper in that case is far slower because of all the spin up cost and the system prompt they inject even if you want to turn it all off. If I used an actual direct API it probably would've been much faster, but I'm doing it for hobby / fun reasons. You also get to fiddle with a lot more params.
- PAndreew 5mo agoI’m running a local Whisper + Gemma 4 pipeline with a cheap USB mic to extract health related data and potential todos from ambient speech. It doesn’t have to be fast doesn’t have to be 100% correct because if it captures at least a few bits of interesting information that would otherwise go unnoticed it’s still a win.
- 650REDHAIR 5mo agoI run whisper through openwebui to gemma4 moe and use kokoro TTS back to me. I use a 5060ti 16gb and a minipc. I tunnel in via Tailscale and access it with my phone or laptop from anywhere. It’s pretty good and will only get better as I optimize.
- formerly_proven 5mo agoWhat is it with AI SaaS naming themselves "openxyz" when there is 0% open about them?
- em500 5mo agoThey learnt from ooenai that naming yourself open-xyz doesn't actually require opening anything.
- debugnik 5mo agoIt's the next co-opted buzzword after "democratize".
- throwaway2037 5mo agoYeah, anytime that I see "People's" or "Democratic" in the name of a nation, I grow suspicious. It is rarely a well-functioning democracy.
- brikym 5mo agoIt's how marketing works. If something is a problem they have to loudly claim to have fixed it. Look around the economy and you'll see lots of it. "Healthy" (high sugar) muesli bars, clean-diesel, surveillance wrapped up as keeping us safe. The modus operandi of marketing is to change minds about self evident things otherwise what is the point?
- nativeit 5mo agoBut once all that is done you still own a Mac in one case, and you don’t in the other, correct?
- odo1242 5mo agoYea this; it’s the same reason why mortgaging is cheaper than renting
- ericpauley 5mo agoThis is far from a universal truth: https://www.nytimes.com/interactive/2024/upshot/buy-rent-calculator.html https://www.nytimes.com/interactive/2024/upshot/buy-rent-cal... Real estate is only a clearly good investment if you ignore opportunity cost.
- seanmcdirmid 5mo agoYou also need to pay close attention to rent vs purchase ratios. A lot of cities are cheap to rent but expensive to buy (eg beijing 10 years ago).
- mantas 5mo agoKey word being „ago“.
- deaux 5mo agoSuch cities still exist and have been in such a state for decades. They can change but that's meaningless as they can also change the other way around.
- seanmcdirmid 5mo agoI’m covering my bases because Chinese real estate has been volatile recently and I’m not sure where the market is at now. It could be that renting is still way cheaper than buying, I just don’t have any direct experience to back that up. If I bought while I was living in Beijing I would probably be underwater with my investment right now, renting for 9 years was the right call and my rent was pretty affordable anyways.
- novok 5mo agoAlso many have power even cheaper or even free unused surplus power with solar. I don't do local inference other than hobby & learning reasons because electricity is so expensive where I am at.