4 ms·

We need a new word, not "local model" but "my own computers model" CapEx based This distinction is important because some "we support local model" tools have t

We need a new word, not "local model" but "my own computers model" CapEx based This distinction is important because some "we support local model" tools have things like ollama orchestration or use the llama.cpp libraries to connect to models on the same physical machine. That's not my definition of local. Mine is "local network". so call it the "LAN model" until we come up with something better. "Self-host" exists but this usually means more "open-weights" as opposed to clamping the performance of the model. It should be defined as ~sub-$10k, using Steve Jobs megapenny unit. Essentially classify things as how many megapennies of spend a machine is that won't OOM on it. That's what I mean when I say local: running inference for 'free' somewhere on hardware I control that's at most single digit thousands of dollars. And if I was feeling fancy, could potentially fine-tune on the days scale. A modern 5090 build-out with a threadripper, nvme, 256GB RAM, this will run you about 10k +/- 1k. The MLX route is about $6000 out the door after tax (m3-ultra 60 core with 256GB). Lastly it's not just "number of parameters". Not all 32B Q4_K_M models load at the same rate or use the same amount of memory. The internal architecture matters and the active parameter count + quantization is becoming a poorer approximation given the SOTA innovations. What might be needed is some standardized eval benchmark against standardized hardware classes with basic real world tasks like toolcalling, code generation, and document procesing. There's plenty of "good enough" models out there for a large category of every day tasks, now I want to find out what runs the best Take a gen6 thinkpad P14s/macbook pro and a 5090/mac studio, run the benchmark and then we can say something like "time-to-first-token/token-per-second/memory-used/total-time-of-test" and rate this as independent from how accurate the model was.

bigyabai 8mo ago
OOM is a pretty terrible benchmark too, though. You can build a DDR4 machine that "technically" loads 256gb models for maybe $1000 used, but then you've got to account for the compute aspect and that's constrained by a number of different variables. A super-sparse model might run great on that DDR4 machine, whereas a 32b model would cause it to chug. There's just not a good way to visualize the compute needed, with all the nuance that exists. I think that trying to create these abstractions are what leads to people impulse buying resource-constrained hardware and getting frustrated. The autoscalers have a huge advantage in this field that homelabbers will never be able to match.
- FrenchTouch42 8mo ago
  > time-to-first-token/token-per-second/memory-used/total-time-of-test Would it not help with the DDR4 example though if we had more "real world" tests?
  - bigyabai 8mo ago
    Maybe, but even that fourth-order metric is missing key performance details like context length and model size/sparsity. The bigger takeaway (IMO) is that there will never really be hardware that scales like Claude or ChatGPT does. I love local AI, but it stresses the fundamental limits of on-device compute.
deleted 8mo ago
[deleted]
echelon 8mo ago
I don't even need "open weights" to run on hardware I own. I am fine renting an H100 (or whatever), as long as I theoretically have access to and own everything running. I do not want my career to become dependent upon Anthropic. Honestly, the best thing for "open" might be for us to build open pipes and services and models where we can rent cloud. Large models will outpace small models: LLMs, video models, "world" models, etc. I'd even be fine time-sharing a running instance of a large model in a large cloud. As long as all the constituent pieces are open where I could (in theory) distill it, run it myself, spin up my own copy, etc. I do not deny that big models are superior. But I worry about the power the large hyperscalers are getting while we focus on small "open" models that really can't match the big ones. We should focus on competing with large models, not artisanal homebrew stuff that is irrelevant.
- Aurornis 8mo ago
  > I do not want my career to become dependent upon Anthropic As someone who switches between Anthropic and ChatGPT depending on the month and has dabbled with other providers and some local LLMs, I think this fear is unfounded. It's really easy to switch between models. The different models have some differences that you notice over time but the techniques you learn in one place aren't going to lock you into a provider anywhere.
  - airstrike 8mo ago
    right, but ChatGPT might not exist at some point, and if we don't force feed the open inference ecosystem and infrastructure back into the mouths of the AI devourer that is this hype cycle, we'll simply be accepting our inevitable, painful death
    - christkv 8mo ago
      If they die there will be so much hardware released to do other tasks.
      - echelon 8mo ago
christkv 8mo ago
I won't need a heater with that running in my room.
- hedora 8mo ago
  This looks like it’ll run easily on a Strix Halo (180W TDP), and be a little sluggish on previous gen AMDs (80W TDP). I can’t be bothered to check TDPs on 64GB macbooks, but none of these devices really count as space heaters.
  - yencabulator 8mo ago
    For the record, a typical US space heater is 1500 W. We're talking a small fraction of a space heater. (Not that I'd want one in my office in the summertime, but I also wouldn't want the fan noise.)
- wincy 8mo ago
  Haha running OSS-120B on my 5090 with most of the layers in video memory, some in RAM with LM Studio, I was hard pressed to get it to actually use anywhere near the full 600W. Gaming in 4K playing a modern game generates substantially more sustained heat.
zozbot234 8mo ago
You can run plenty of models on a $10K machine or even a lot less than that, it all depends how much you want to wait for results. Streaming weights from SSD storage using mmap() is already a reality when running the largest and sparsest models. You can save even more on memory by limiting KV caching at the cost of extra compute, and there may be ways to push RAM savings even higher simply by tweaking the extent to which model activations are recomputed as needed.
- kristopolous 8mo ago
  Yeah there's a lot of people that advocate for really slow inference on cheap infra. That's something else that should be expressed in this fidelity Because honestly I don't care about 0.2 tps for my use cases although I've spoken with many who are fine with numbers like that. At least the people I've talked to they talk about how if they have a very high confidence score that the model will succeed they don't mind the wait. Essentially a task failure is 1 in 10, I want to monitor and retry. If it's 1 in 1000, then I can walk away. The reality is most people don't have a bearing on what this order of magnitude actually is for a given task. So unless you have high confidence in your confidence score, slow is useless But sometimes you do...
  - zozbot234 8mo ago
    If you launch enough tasks in parallel you aren't going to care that 1 in 10 failed, as long as the other 9 are good. Just rerun the failed job whenever you get around to it, the infra will still be getting plenty of utilization on the rest.
openclawai 8mo ago
For context on what cloud API costs look like when running coding agents: With Claude Sonnet at $3/$15 per 1M tokens, a typical agent loop with ~2K input tokens and ~500 output per call, 5 LLM calls per task, and 20% retry overhead (common with tool use): you're looking at roughly $0.05-0.10 per agent task. At 1K tasks/day that's ~$1.5K-3K/month in API spend. The retry overhead is where the real costs hide. Most cost comparisons assume perfect execution, but tool-calling agents fail parsing, need validation retries, etc. I've seen retry rates push effective costs 40-60% above baseline projections. Local models trading 50x slower inference for $0 marginal cost start looking very attractive for high-volume, latency-tolerant workloads.
- taneq 8mo ago
  At this point isn’t the marginal cost based on power consumption? At 30c/kWh and with a beefy desktop pc pulling up to half a kW, that’s 15c/hr. For true zero marginal cost, maybe get solar panels. :P
  - EGreg 8mo ago
    This is an interesting question actually! Marginal cost includes energy usage but also I burned out a MacBook GPU with vanity-eth last year so wear-and-tear is also a cost.
- pstuart 8mo ago
  Might there be a way to leverage local models just to help minimize the retries -- doing the tool calling handling and giving the agent "perfect execution"? I'm a noob and am asking as wishful thinking.
  - jermaustin1 8mo ago
    > I'm a noob and am asking as wishful thinking. Don't minimize your thoughts! Outside voices and naive questions sometimes provide novel insights that might be dismissed, but someone might listen. I've not done this exactly, but I have setup "chains" that create a fresh context for tool calls so their call chains don't fill the main context. There is no reason why the Tool Calls couldn't be redirected to another LLM endpoint (local for instance). Especially with something like gpt-oss-20b, where I've found executing tools happens at a higher success than claude sonnet via openrouter.
mrklol 8mo ago
I mean if it’s running in your lan, isn’t it local? :D
estimator7292 8mo ago
Local as in localhost
- helly25 8mo ago
  Local! I do not mind the cost honestly. And a bit slower also works. I just use one older mac ultra 2/192G ram and another with an rtx5060/16G and an and r9700/32G. Between those I get my models working fine. That also gives me full privacy. And that is worth way way way more than any cost.

bigyabai 8mo ago

OOM is a pretty terrible benchmark too, though. You can build a DDR4 machine that "technically" loads 256gb models for maybe $1000 used, but then you've got to account for the compute aspect and that's constrained by a number of different variables. A super-sparse model might run great on that DDR4 machine, whereas a 32b model would cause it to chug. There's just not a good way to visualize the compute needed, with all the nuance that exists. I think that trying to create these abstractions are what leads to people impulse buying resource-constrained hardware and getting frustrated. The autoscalers have a huge advantage in this field that homelabbers will never be able to match.

FrenchTouch42 8mo ago
> time-to-first-token/token-per-second/memory-used/total-time-of-test Would it not help with the DDR4 example though if we had more "real world" tests?
- bigyabai 8mo ago
  Maybe, but even that fourth-order metric is missing key performance details like context length and model size/sparsity. The bigger takeaway (IMO) is that there will never really be hardware that scales like Claude or ChatGPT does. I love local AI, but it stresses the fundamental limits of on-device compute.

deleted 8mo ago

[deleted]

echelon 8mo ago

I don't even need "open weights" to run on hardware I own. I am fine renting an H100 (or whatever), as long as I theoretically have access to and own everything running. I do not want my career to become dependent upon Anthropic. Honestly, the best thing for "open" might be for us to build open pipes and services and models where we can rent cloud. Large models will outpace small models: LLMs, video models, "world" models, etc. I'd even be fine time-sharing a running instance of a large model in a large cloud. As long as all the constituent pieces are open where I could (in theory) distill it, run it myself, spin up my own copy, etc. I do not deny that big models are superior. But I worry about the power the large hyperscalers are getting while we focus on small "open" models that really can't match the big ones. We should focus on competing with large models, not artisanal homebrew stuff that is irrelevant.

Aurornis 8mo ago
> I do not want my career to become dependent upon Anthropic As someone who switches between Anthropic and ChatGPT depending on the month and has dabbled with other providers and some local LLMs, I think this fear is unfounded. It's really easy to switch between models. The different models have some differences that you notice over time but the techniques you learn in one place aren't going to lock you into a provider anywhere.
- airstrike 8mo ago
  right, but ChatGPT might not exist at some point, and if we don't force feed the open inference ecosystem and infrastructure back into the mouths of the AI devourer that is this hype cycle, we'll simply be accepting our inevitable, painful death
  - christkv 8mo ago
    If they die there will be so much hardware released to do other tasks.
    - echelon 8mo ago

christkv 8mo ago

I won't need a heater with that running in my room.

hedora 8mo ago
This looks like it’ll run easily on a Strix Halo (180W TDP), and be a little sluggish on previous gen AMDs (80W TDP). I can’t be bothered to check TDPs on 64GB macbooks, but none of these devices really count as space heaters.
- yencabulator 8mo ago
  For the record, a typical US space heater is 1500 W. We're talking a small fraction of a space heater. (Not that I'd want one in my office in the summertime, but I also wouldn't want the fan noise.)
wincy 8mo ago
Haha running OSS-120B on my 5090 with most of the layers in video memory, some in RAM with LM Studio, I was hard pressed to get it to actually use anywhere near the full 600W. Gaming in 4K playing a modern game generates substantially more sustained heat.

zozbot234 8mo ago

You can run plenty of models on a $10K machine or even a lot less than that, it all depends how much you want to wait for results. Streaming weights from SSD storage using mmap() is already a reality when running the largest and sparsest models. You can save even more on memory by limiting KV caching at the cost of extra compute, and there may be ways to push RAM savings even higher simply by tweaking the extent to which model activations are recomputed as needed.

kristopolous 8mo ago
Yeah there's a lot of people that advocate for really slow inference on cheap infra. That's something else that should be expressed in this fidelity Because honestly I don't care about 0.2 tps for my use cases although I've spoken with many who are fine with numbers like that. At least the people I've talked to they talk about how if they have a very high confidence score that the model will succeed they don't mind the wait. Essentially a task failure is 1 in 10, I want to monitor and retry. If it's 1 in 1000, then I can walk away. The reality is most people don't have a bearing on what this order of magnitude actually is for a given task. So unless you have high confidence in your confidence score, slow is useless But sometimes you do...
- zozbot234 8mo ago
  If you launch enough tasks in parallel you aren't going to care that 1 in 10 failed, as long as the other 9 are good. Just rerun the failed job whenever you get around to it, the infra will still be getting plenty of utilization on the rest.

openclawai 8mo ago

For context on what cloud API costs look like when running coding agents: With Claude Sonnet at $3/$15 per 1M tokens, a typical agent loop with ~2K input tokens and ~500 output per call, 5 LLM calls per task, and 20% retry overhead (common with tool use): you're looking at roughly $0.05-0.10 per agent task. At 1K tasks/day that's ~$1.5K-3K/month in API spend. The retry overhead is where the real costs hide. Most cost comparisons assume perfect execution, but tool-calling agents fail parsing, need validation retries, etc. I've seen retry rates push effective costs 40-60% above baseline projections. Local models trading 50x slower inference for $0 marginal cost start looking very attractive for high-volume, latency-tolerant workloads.

taneq 8mo ago
At this point isn’t the marginal cost based on power consumption? At 30c/kWh and with a beefy desktop pc pulling up to half a kW, that’s 15c/hr. For true zero marginal cost, maybe get solar panels. :P
- EGreg 8mo ago
  This is an interesting question actually! Marginal cost includes energy usage but also I burned out a MacBook GPU with vanity-eth last year so wear-and-tear is also a cost.
pstuart 8mo ago
Might there be a way to leverage local models just to help minimize the retries -- doing the tool calling handling and giving the agent "perfect execution"? I'm a noob and am asking as wishful thinking.
- jermaustin1 8mo ago
  > I'm a noob and am asking as wishful thinking. Don't minimize your thoughts! Outside voices and naive questions sometimes provide novel insights that might be dismissed, but someone might listen. I've not done this exactly, but I have setup "chains" that create a fresh context for tool calls so their call chains don't fill the main context. There is no reason why the Tool Calls couldn't be redirected to another LLM endpoint (local for instance). Especially with something like gpt-oss-20b, where I've found executing tools happens at a higher success than claude sonnet via openrouter.

mrklol 8mo ago

I mean if it’s running in your lan, isn’t it local? :D

estimator7292 8mo ago

Local as in localhost

helly25 8mo ago
Local! I do not mind the cost honestly. And a bit slower also works. I just use one older mac ultra 2/192G ram and another with an rtx5060/16G and an and r9700/32G. Between those I get my models working fine. That also gives me full privacy. And that is worth way way way more than any cost.