3 ms·

Or get a 3090 24 GB off eBay for 700€ and run mistral, devstral, qwen3 and others indefinitely. I'm a big fan of two-tier systems where the 90% case is taken o

Or get a 3090 24 GB off eBay for 700€ and run mistral, devstral, qwen3 and others indefinitely. I'm a big fan of two-tier systems where the 90% case is taken on by something cheap, and renting the premium tier. Works for cars, server hosting, dining ...

scuderiaseb 1y ago
Would be cheaper in the long run maybe but heat, noise, electricity cost and the fact that the models are not as good as sonnet 4 and opus 4 are some things to consider.
- justlikereddit 1y ago
  For testing always-on concepts and long-running agent experiments a local model will be superior. For investigating concepts, can AI for this at all? Subscriptions will be a better approach. Local models are underutilized in comparison to the promise they have. Imagine browser-agnostic universal AdBlock and that's just the surface.
JKCalhoun 1y ago
I haven't paid for the nicer models — now wondering if you could use less capable models to get some code up and running, then pay for better models just to "polish up" the code, get it to "final".
- deleted 1y ago
  [deleted]
scosman 1y ago
Slower tokens, maintenance time (installing new models/updating tools), lower SWE bench scores. For me it’s not worth it - optimize for speed. If I worked on a codebase I couldn’t send to the hosted tools I’d go this route in a heartbeat.
dist-epoch 1y ago
I did the math, it just doesn't check out, especially if you take into account time to generate and electricity. Even assuming you generate 100 tok/second, 24 hr, that's only 8 mil tokens. About $5 for Gemini 2.5 Flash or $50 for 2.5 Pro. But you will not be able to run it 24/7 locally - sleep, thinking between prompts, etc Even at continuous usage (24 hr/day), it's still cheaper to just pay Google for Gemini 2.5 Flash tokens for half a year. And this is NOT taking into account that mistral/devstral/qwen3 are vastly inferior models. How can Google be so cheap? TPUs, batching multiple requests, economy of scale, loss-leader, ... Running stuff locally makes sense for fun, as backup for loss of connectivity, if you want to try uncensored models, but it doesn't make sense economically.
iLoveOncall 1y ago
I tried running those locally via Ollama on my 3080 (latency doesn't matter) and the performance is abysmal. They're almost random token generators, literally unusable. My hallucination rate was near 100% for basic tasks like data extraction from markdown.
- dist-epoch 1y ago
  Have you tried Qwen-32b-8bit? It's pretty solid. Anything lower, yeah, it's kind of garbage.
maccard 1y ago
How do you connect that 3090 to a MacBook? How does that work when I’m not tethered to my desk? I think if you’re comparing it to Claude max it’s ok, but the payback period on GitHub copilot for example is almost 7 years.
- benterix 1y ago
  > How do you connect that 3090 to a MacBook? This is probably the easiest problem to solve, isn't it? Most tools offer a n API endpoint setup that you can use locally and also expose on a local network easily.

scuderiaseb 1y ago

Would be cheaper in the long run maybe but heat, noise, electricity cost and the fact that the models are not as good as sonnet 4 and opus 4 are some things to consider.

justlikereddit 1y ago
For testing always-on concepts and long-running agent experiments a local model will be superior. For investigating concepts, can AI for this at all? Subscriptions will be a better approach. Local models are underutilized in comparison to the promise they have. Imagine browser-agnostic universal AdBlock and that's just the surface.

JKCalhoun 1y ago

I haven't paid for the nicer models — now wondering if you could use less capable models to get some code up and running, then pay for better models just to "polish up" the code, get it to "final".

deleted 1y ago
[deleted]

scosman 1y ago

Slower tokens, maintenance time (installing new models/updating tools), lower SWE bench scores. For me it’s not worth it - optimize for speed. If I worked on a codebase I couldn’t send to the hosted tools I’d go this route in a heartbeat.

dist-epoch 1y ago

I did the math, it just doesn't check out, especially if you take into account time to generate and electricity. Even assuming you generate 100 tok/second, 24 hr, that's only 8 mil tokens. About $5 for Gemini 2.5 Flash or $50 for 2.5 Pro. But you will not be able to run it 24/7 locally - sleep, thinking between prompts, etc Even at continuous usage (24 hr/day), it's still cheaper to just pay Google for Gemini 2.5 Flash tokens for half a year. And this is NOT taking into account that mistral/devstral/qwen3 are vastly inferior models. How can Google be so cheap? TPUs, batching multiple requests, economy of scale, loss-leader, ... Running stuff locally makes sense for fun, as backup for loss of connectivity, if you want to try uncensored models, but it doesn't make sense economically.

iLoveOncall 1y ago

I tried running those locally via Ollama on my 3080 (latency doesn't matter) and the performance is abysmal. They're almost random token generators, literally unusable. My hallucination rate was near 100% for basic tasks like data extraction from markdown.

dist-epoch 1y ago
Have you tried Qwen-32b-8bit? It's pretty solid. Anything lower, yeah, it's kind of garbage.

maccard 1y ago

How do you connect that 3090 to a MacBook? How does that work when I’m not tethered to my desk? I think if you’re comparing it to Claude max it’s ok, but the payback period on GitHub copilot for example is almost 7 years.

benterix 1y ago
> How do you connect that 3090 to a MacBook? This is probably the easiest problem to solve, isn't it? Most tools offer a n API endpoint setup that you can use locally and also expose on a local network easily.