3 ms·
For a lot of local workloads, sub-1 tok/s is useless in foreground and perfectly acceptable in background. If the choice is “this crashes” vs “this finishes ove
by vanyaland 6mo ago
For a lot of local workloads, sub-1 tok/s is useless in foreground and perfectly acceptable in background. If the choice is “this crashes” vs “this finishes overnight,” that’s still a meaningful capability jump.
- joelthelion 6mo agoHow much are you going to spend on electricity though? Is this really going to be more cost-effective than just using openrouter?
- austinthetaco 6mo agoThere are many other reasons someone might want to run a model locally outside of cost savings, ownership of data flow and use in locations without internet to name a couple.
- hadlock 6mo agoIf my options are run Opus 4.6 in the cloud for $200/mo or run Opus 4.6 locally for $275, I am absolutely going to self-host 100% of the time. Sending all that data to the cloud presents tremendous legal risk for companies. There's currently no retention rules about privately hosted AI.