3 ms·

Incredibly important research. We've reached the point where local LLMs are good enough! It takes less time for local model to take the first action on your tas

Incredibly important research. We've reached the point where local LLMs are good enough! It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. Local models are persistent and 100% predictable unlike any cloud offering. It's better for the power system for the demand to be distributed. During the winter time the GPU also doubles as a 300W in-house heater. Not to mention avoiding personal data collection and re-selling.

jgalt212 23d ago
> It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. This is only true if your local model is already resident in RAM / VRAM.
- jmiskovic 22d ago
  In my case the bad internet also tips the scale.
rpozarickij 22d ago
> During the winter time the GPU also doubles as a 300W in-house heater There could be a service that works in reverse where if someone needs a heater for a few months, they could rent a portable server (e.g. using older repurposed GPUs) with a built-in 5G modem that would run inference on LLM queries. As an incentive perhaps renting itself could be free (or you could earn money?), but you'd still have to pay your electricity bill.
- HPsquared 22d ago
  I remember a company that sold or used bitcoin mining rigs as swimming pool heaters.
  - teamonkey 22d ago
    There was one that tried to sell them as free heating for the elderly.
- stymaar 22d ago
  I saw a company pitching almost exactly this on linkedin earlier this month. Except not as a portable heater but as your house's central heating system.
- hunter2_ 22d ago
  > you'd still have to pay your electricity bill. I should only have to pay 1/3 of what it adds to my bill, given that it's 1/3 as efficient as a heat pump. And that coefficient is to be adjusted as outside temp changes (assuming air source heat pump; water source would have a stable COP).
rckoepke 22d ago
> During the winter time the GPU also doubles as a 300W in-house heater. Just a reminder that heat pumps can consume 300W of electricity to provide 1200W of heat.
- sva_ 22d ago
  Under ideal conditions with outside temperatures* But yes
- stymaar 22d ago
  Of course, but the power dissipated in a data center provide zero heat for your house. (Tangent: I wonder why we haven't seen deployment of organic Rankin cycle generators in AI data centers, the exhaust temperature should be compatible and that could yield a 10-20% energy bill saving).
  - mattmanser 22d ago
    I have no idea what they use, but Stockholm is already using the waste heat from data centres: https://eu-mayors.ec.europa.eu/en/news/stockholm-sweden-heat-recovery-data-centres https://eu-mayors.ec.europa.eu/en/news/stockholm-sweden-heat...
    - stymaar 22d ago
      They use it in the city heating system, which is obviously the optimal way of reusing the heat but most data centers aren't put next to such a heating system. I'm talking about making electricity back from the heat (using a low-temp thermodynamic cycle). It has a low yield (due to the low input temperature) but it's usually economically viable when using heat that would end up in the heavens anyway.
      - adgjlsfhk1 22d ago
        the heat is way too low grade. typical water cooling setups only have ~50C water. that's barely anything
tccole 22d ago
For sure but aren’t there more regulations and building codes around heaters? It’s been close to a decade since I took heat and mass transfer courses but this seems like saying you can use your oven as a heater for your house in the wintertime. Please correct me if I am wrong.
- fc417fc802 22d ago
  Yes it is exactly like that. And I've done that before when my central heat went out so I'm not sure what you're trying to say here?
iovrthoughtthis 22d ago
What devices and models are people running locally?
- stymaar 22d ago
  Qwen 3.6-35B-A3B is surprisingly OK on a medium-range business laptop (Lenovo). A bit slow for agentic coding of course but fine for any chatbot use-case.
- qtqtqt 22d ago
  Any good models that do not need a dedicated GPU?
- m101 22d ago
  A MacBook Pro is probably the best bang for your buck. I have a 64gb m4 max, but you’ll run decent models with half that ram I think.
Aurornis 22d ago
> We've reached the point where local LLMs are good enough! For some tasks, yes. For most of my deeper work they're not even close to my subscriptions. > It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. I have some decent LLM hardware here and I strongly disagree with this. Claude responds quickly. Using Fable or Opus it will deliver a working result faster than my local models because it gets there in fewer tokens. That's just how it is. > During the winter time the GPU also doubles as a 300W in-house heater. This is a curse in the summer. I'm feeling it right now.
01284a7e 22d ago
This is annoying but, can you point me to where to get started with local models? A repo, website, something?
- lowbloodsugar 22d ago
  Ask Claude.ai. Tell it about your hardware. What you plan on doing. Can start with chat. Then ask it how to install Claude remote control on you system. Then it can work while you sleep. Now I have a solid local model that I use instead of Claude.
- thin_carapace 22d ago
  visit canirun.ai to find a suitable model, download .gguf of a version of that model from huggingface, plug .gguf into ollama
applicative 22d ago
Local LLMs make my laptop hot and are wrong about everything.
18Deepnar 22d ago
yea i see the potential of local slms too, it wont replace the frontier llms but for particular aspects they are generally really good, and at a certain point models like qwen 3.8 if people are able to set up proper api providers for it it can be extremely cheap and helpful. I am also working on trying to fix the memory and low context issue of such slms so they can actually be used locally for real use cases and not just small work

jgalt212 23d ago

> It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. This is only true if your local model is already resident in RAM / VRAM.

jmiskovic 22d ago
In my case the bad internet also tips the scale.

rpozarickij 22d ago

> During the winter time the GPU also doubles as a 300W in-house heater There could be a service that works in reverse where if someone needs a heater for a few months, they could rent a portable server (e.g. using older repurposed GPUs) with a built-in 5G modem that would run inference on LLM queries. As an incentive perhaps renting itself could be free (or you could earn money?), but you'd still have to pay your electricity bill.

HPsquared 22d ago
I remember a company that sold or used bitcoin mining rigs as swimming pool heaters.
- teamonkey 22d ago
  There was one that tried to sell them as free heating for the elderly.
stymaar 22d ago
I saw a company pitching almost exactly this on linkedin earlier this month. Except not as a portable heater but as your house's central heating system.
hunter2_ 22d ago
> you'd still have to pay your electricity bill. I should only have to pay 1/3 of what it adds to my bill, given that it's 1/3 as efficient as a heat pump. And that coefficient is to be adjusted as outside temp changes (assuming air source heat pump; water source would have a stable COP).

rckoepke 22d ago

> During the winter time the GPU also doubles as a 300W in-house heater. Just a reminder that heat pumps can consume 300W of electricity to provide 1200W of heat.

sva_ 22d ago
Under ideal conditions with outside temperatures* But yes
stymaar 22d ago
Of course, but the power dissipated in a data center provide zero heat for your house. (Tangent: I wonder why we haven't seen deployment of organic Rankin cycle generators in AI data centers, the exhaust temperature should be compatible and that could yield a 10-20% energy bill saving).
- mattmanser 22d ago
  I have no idea what they use, but Stockholm is already using the waste heat from data centres: https://eu-mayors.ec.europa.eu/en/news/stockholm-sweden-heat-recovery-data-centres https://eu-mayors.ec.europa.eu/en/news/stockholm-sweden-heat...
  - stymaar 22d ago
    They use it in the city heating system, which is obviously the optimal way of reusing the heat but most data centers aren't put next to such a heating system. I'm talking about making electricity back from the heat (using a low-temp thermodynamic cycle). It has a low yield (due to the low input temperature) but it's usually economically viable when using heat that would end up in the heavens anyway.
    - adgjlsfhk1 22d ago
      the heat is way too low grade. typical water cooling setups only have ~50C water. that's barely anything

tccole 22d ago

For sure but aren’t there more regulations and building codes around heaters? It’s been close to a decade since I took heat and mass transfer courses but this seems like saying you can use your oven as a heater for your house in the wintertime. Please correct me if I am wrong.

fc417fc802 22d ago
Yes it is exactly like that. And I've done that before when my central heat went out so I'm not sure what you're trying to say here?

iovrthoughtthis 22d ago

What devices and models are people running locally?

stymaar 22d ago
Qwen 3.6-35B-A3B is surprisingly OK on a medium-range business laptop (Lenovo). A bit slow for agentic coding of course but fine for any chatbot use-case.
qtqtqt 22d ago
Any good models that do not need a dedicated GPU?
m101 22d ago
A MacBook Pro is probably the best bang for your buck. I have a 64gb m4 max, but you’ll run decent models with half that ram I think.

Aurornis 22d ago

> We've reached the point where local LLMs are good enough! For some tasks, yes. For most of my deeper work they're not even close to my subscriptions. > It takes less time for local model to take the first action on your task than it does for Claude to validate your login, put you into queue and start issuing the commands. I have some decent LLM hardware here and I strongly disagree with this. Claude responds quickly. Using Fable or Opus it will deliver a working result faster than my local models because it gets there in fewer tokens. That's just how it is. > During the winter time the GPU also doubles as a 300W in-house heater. This is a curse in the summer. I'm feeling it right now.

01284a7e 22d ago

This is annoying but, can you point me to where to get started with local models? A repo, website, something?

lowbloodsugar 22d ago
Ask Claude.ai. Tell it about your hardware. What you plan on doing. Can start with chat. Then ask it how to install Claude remote control on you system. Then it can work while you sleep. Now I have a solid local model that I use instead of Claude.
thin_carapace 22d ago
visit canirun.ai to find a suitable model, download .gguf of a version of that model from huggingface, plug .gguf into ollama

applicative 22d ago

Local LLMs make my laptop hot and are wrong about everything.

18Deepnar 22d ago

yea i see the potential of local slms too, it wont replace the frontier llms but for particular aspects they are generally really good, and at a certain point models like qwen 3.8 if people are able to set up proper api providers for it it can be extremely cheap and helpful. I am also working on trying to fix the memory and low context issue of such slms so they can actually be used locally for real use cases and not just small work