4 ms·

The funny thing is Claude Cowork has taught me to be patient with response timelines. I’m now figuring I’ll be running locally no later than 2028. (I want to s

The funny thing is Claude Cowork has taught me to be patient with response timelines. I’m now figuring I’ll be running locally no later than 2028. (I want to spend no more than $10k. And I want to run a model comparable to today’s SOTA.)

paytonjjones 4mo ago
Today's SOTA also sounds totally sufficient to me, but I wonder how much our standards will inflate by 2028. Maybe a lot, maybe not at all...very hard to say.
- anon373839 4mo ago
  This seems to vary by person. I get immense value in coding assistance from Qwen 3.6 35B-A3B which is like a frontier model from a year ago. But a lot of people say it’s stupid, useless, a toy, etc. I do work by the “short leash” method and mainly just use the model for brainstorming/planning/design assistance and zipping through the drudgery of boilerplate and executing refactors. I don’t think this tier of model is good for “hey LLM, build me a Github clone” ... but I also don’t see the value in that use anyway.
  - 3836293648 4mo ago
    Could you expand more on what you do with qwen3.6? Because I couldn't get the denser 27B version to do trivial "take this pattern, repeat it over a single file with minimal thought, just slightly beyond what I can do with sed" reliably.
    - kgeist 4mo ago
      How was qwen3.6 launched? The thing is, everyone has their own variant of "qwen3.6 27b" depending on the launch parameters, ranging from "SOTA in its class" to "completely broken"
      - 3836293648 4mo ago
        Yeah, I really should know. But I'm using whatever ollama gave me by default, which probably isn't optimal.
        kgeist 4mo ago
        Ollama uses 4 bit quants and a very short context window by default. It can easily break on anything more complex than a simple chat.
        3836293648 4mo ago
        Oh, the context window I overrode immediately, the default one can't even reach my prompt in opencode, it's pushed out by all the pre- and post-amble. But I should probably try something less compressed next time I bother.
    - anon373839 4mo ago
      Certainly. First of all, I am using OpenCode as the harness. (I have heard there are better harnesses such as little-coder for small open-weights models, but I haven't tried them yet.) Looking over some of my recent sessions, here are some examples: - Asking Qwen to review project docs (requirements, user stories, etc) so that "we" can evaluate an iterate on an API design. Then back-and-forth chat about possible design directions. Then I ask for a rough-sketch plan of the one I'm interested in. I provide some tweaks to the plan and request a final plan in full detail. I switch to build mode and say go; everything is written to spec. - Asking Qwen to write a suite of tests covering X, Y, Z issues with permutations A, B, C per issue. - Asking Qwen to edit the shape of a CNN to insert auxiliary branches for intermediate supervision, and to extract out part of the network as a modular component with parameterized architecture. I have less experience with the dense 27B because it's too slow to use on Apple Silicon. But regardless of which model you try, I would recommend trying a full-fat cloud hosted version of it first, so that you can get a sense of what it's capable of when the inference stack is correctly configured. LLMs are very sensitive to quantization formats, discrepancies in chat templates, etc. That kind of stuff is make-or-break.
  - tharkun__ 4mo ago
    Caveat: I have not been able to try that model locally, so no personal experience. Running this locally at usable speeds would be cost prohibitive for personal coding use for me. But if we can believe you that it's doing what a Claude model was doing a year ago then I'd say: OMG no I really never want to go back to that level of frustration getting an agent to do what I want it to do.
    - mrandish 4mo ago
      > OMG no I really never want to go back to that level of frustration getting an agent to do what I want it to do. While it probably won't matter enough to change your mind, remember that you've gotten better at extracting value from all models than you were a year ago - plus the harnesses and other tools have gotten a lot better too.
  - dyauspitr 4mo ago
    > I don’t think this tier of model is good for “hey LLM, build me a Github clone” ... but I also don’t see the value in that use anyway. What could be more valuable than outputting the exact thing you asked for?
    - kennywinker 4mo ago
      Because the thing you get, from a prompt like that - even with a sota llm like fable - is a Potemkin village.
    - anon373839 4mo ago
      Knowing what to ask for, for one. Nobody can just whip up a specification for a system that satisfies all of the technical/design/business constraints that will turn out to have been relevant, has good usability for the target users, hits the right performance tradeoffs - all out of thin air. If anyone could, THAT would be priceless.
      - alfiedotwtf 4mo ago
        Using the superpowers skill will help fill in the gaps for what’s needed to one-shot (it’s more setup so you can one-shot). And if it gets it wrong, at least you have 80% scaffolding that you can ask to iterate on. Hoping you’ll get 100% you asked for is going to fail. Hell… you can even achieve this with humans!
- stingraycharles 4mo ago
  Looking at how critical we are about today’s models, vs where we were last year, and I don’t expect anyone to be content with Fable-class models in 2028. Expectations seem to be rising at a faster rate than models can improve.
walrus01 4mo ago
For 10k you can buy a used dual socket Intel or amd based rackmount server with a terabyte of ram, and run models on cpu only at a reasonable speed. Same server would have been 4-5k a couple years ago before ram price rise. Or buy one on eBay with 512GB that has half its slots populated and then buy the matching 512GB kit to add.
- Abishek_Muthian 4mo ago
  Which CPU gen are you suggesting, is there any writeup on such setup where <10K (not incl. power bill) cpu only rig is giving usable token speeds on latest SoTA open weights models? In my experience with rig half that cost, entire exercise of running coding models locally has been a huge disappointment. Cost/Value when compared to cloud services is just not there, but I see the merit for those who value privacy over quality of output and want a backup of huge condensed corpus of data within their control. Kudos to OP though, They had clear goals and they achieved it.
  - walrus01 4mo ago
    I think there is a good sized population of people who absolutely don't want to submit everything they do to an off site service, or let their content be used for unknown training purposes, and will tolerate slowness at 1 to 10 tok/s as a tradeoff. Or people who want or need to run an uncensored (abliterated) gguf file to deal with controversial topics that a paid LLM service will refuse to work with or ban you for.
    - sandworm101 4mo ago
      Not just controversial but also regulated areas. Virtually every law firm would be interested on locally-hosted AI at a reasonable price. So too ever medical research lab. Every CGI firm doing work for film/TV. And all the video game developers.
      - swiftcoder 4mo ago
        Do they care about locally-hosted, or only about self-hosted? I'm not really clear why a local box would be any better than running on a private AWS instance in any of these scenarios...
netdevphoenix 4mo ago
> (I want to spend no more than $10k. And I want to run a model comparable to today’s SOTA.) The question is, will you want to run a model comparable to today's (meaning 2026) SOTA in 2028? Humans always want the latest shiny LLM model.
- hypfer 4mo ago
  > I want to run a model comparable to today’s SOTA. It says so in the quoted text, yes.

paytonjjones 4mo ago

Today's SOTA also sounds totally sufficient to me, but I wonder how much our standards will inflate by 2028. Maybe a lot, maybe not at all...very hard to say.

anon373839 4mo ago
This seems to vary by person. I get immense value in coding assistance from Qwen 3.6 35B-A3B which is like a frontier model from a year ago. But a lot of people say it’s stupid, useless, a toy, etc. I do work by the “short leash” method and mainly just use the model for brainstorming/planning/design assistance and zipping through the drudgery of boilerplate and executing refactors. I don’t think this tier of model is good for “hey LLM, build me a Github clone” ... but I also don’t see the value in that use anyway.
- 3836293648 4mo ago
  Could you expand more on what you do with qwen3.6? Because I couldn't get the denser 27B version to do trivial "take this pattern, repeat it over a single file with minimal thought, just slightly beyond what I can do with sed" reliably.
  - kgeist 4mo ago
    How was qwen3.6 launched? The thing is, everyone has their own variant of "qwen3.6 27b" depending on the launch parameters, ranging from "SOTA in its class" to "completely broken"
    - 3836293648 4mo ago
      Yeah, I really should know. But I'm using whatever ollama gave me by default, which probably isn't optimal.
      - kgeist 4mo ago
        Ollama uses 4 bit quants and a very short context window by default. It can easily break on anything more complex than a simple chat.
        3836293648 4mo ago
        Oh, the context window I overrode immediately, the default one can't even reach my prompt in opencode, it's pushed out by all the pre- and post-amble. But I should probably try something less compressed next time I bother.
  - anon373839 4mo ago
    Certainly. First of all, I am using OpenCode as the harness. (I have heard there are better harnesses such as little-coder for small open-weights models, but I haven't tried them yet.) Looking over some of my recent sessions, here are some examples: - Asking Qwen to review project docs (requirements, user stories, etc) so that "we" can evaluate an iterate on an API design. Then back-and-forth chat about possible design directions. Then I ask for a rough-sketch plan of the one I'm interested in. I provide some tweaks to the plan and request a final plan in full detail. I switch to build mode and say go; everything is written to spec. - Asking Qwen to write a suite of tests covering X, Y, Z issues with permutations A, B, C per issue. - Asking Qwen to edit the shape of a CNN to insert auxiliary branches for intermediate supervision, and to extract out part of the network as a modular component with parameterized architecture. I have less experience with the dense 27B because it's too slow to use on Apple Silicon. But regardless of which model you try, I would recommend trying a full-fat cloud hosted version of it first, so that you can get a sense of what it's capable of when the inference stack is correctly configured. LLMs are very sensitive to quantization formats, discrepancies in chat templates, etc. That kind of stuff is make-or-break.
- tharkun__ 4mo ago
  Caveat: I have not been able to try that model locally, so no personal experience. Running this locally at usable speeds would be cost prohibitive for personal coding use for me. But if we can believe you that it's doing what a Claude model was doing a year ago then I'd say: OMG no I really never want to go back to that level of frustration getting an agent to do what I want it to do.
  - mrandish 4mo ago
    > OMG no I really never want to go back to that level of frustration getting an agent to do what I want it to do. While it probably won't matter enough to change your mind, remember that you've gotten better at extracting value from all models than you were a year ago - plus the harnesses and other tools have gotten a lot better too.
- dyauspitr 4mo ago
  > I don’t think this tier of model is good for “hey LLM, build me a Github clone” ... but I also don’t see the value in that use anyway. What could be more valuable than outputting the exact thing you asked for?
  - kennywinker 4mo ago
    Because the thing you get, from a prompt like that - even with a sota llm like fable - is a Potemkin village.
  - anon373839 4mo ago
    Knowing what to ask for, for one. Nobody can just whip up a specification for a system that satisfies all of the technical/design/business constraints that will turn out to have been relevant, has good usability for the target users, hits the right performance tradeoffs - all out of thin air. If anyone could, THAT would be priceless.
    - alfiedotwtf 4mo ago
      Using the superpowers skill will help fill in the gaps for what’s needed to one-shot (it’s more setup so you can one-shot). And if it gets it wrong, at least you have 80% scaffolding that you can ask to iterate on. Hoping you’ll get 100% you asked for is going to fail. Hell… you can even achieve this with humans!
stingraycharles 4mo ago
Looking at how critical we are about today’s models, vs where we were last year, and I don’t expect anyone to be content with Fable-class models in 2028. Expectations seem to be rising at a faster rate than models can improve.

walrus01 4mo ago

For 10k you can buy a used dual socket Intel or amd based rackmount server with a terabyte of ram, and run models on cpu only at a reasonable speed. Same server would have been 4-5k a couple years ago before ram price rise. Or buy one on eBay with 512GB that has half its slots populated and then buy the matching 512GB kit to add.

Abishek_Muthian 4mo ago
Which CPU gen are you suggesting, is there any writeup on such setup where <10K (not incl. power bill) cpu only rig is giving usable token speeds on latest SoTA open weights models? In my experience with rig half that cost, entire exercise of running coding models locally has been a huge disappointment. Cost/Value when compared to cloud services is just not there, but I see the merit for those who value privacy over quality of output and want a backup of huge condensed corpus of data within their control. Kudos to OP though, They had clear goals and they achieved it.
- walrus01 4mo ago
  I think there is a good sized population of people who absolutely don't want to submit everything they do to an off site service, or let their content be used for unknown training purposes, and will tolerate slowness at 1 to 10 tok/s as a tradeoff. Or people who want or need to run an uncensored (abliterated) gguf file to deal with controversial topics that a paid LLM service will refuse to work with or ban you for.
  - sandworm101 4mo ago
    Not just controversial but also regulated areas. Virtually every law firm would be interested on locally-hosted AI at a reasonable price. So too ever medical research lab. Every CGI firm doing work for film/TV. And all the video game developers.
    - swiftcoder 4mo ago
      Do they care about locally-hosted, or only about self-hosted? I'm not really clear why a local box would be any better than running on a private AWS instance in any of these scenarios...

netdevphoenix 4mo ago

> (I want to spend no more than $10k. And I want to run a model comparable to today’s SOTA.) The question is, will you want to run a model comparable to today's (meaning 2026) SOTA in 2028? Humans always want the latest shiny LLM model.

hypfer 4mo ago
> I want to run a model comparable to today’s SOTA. It says so in the quoted text, yes.