7 ms·

After having been a happy user of Qwen3.6-27B for a few weeks, due to being away from the hardware, I'm currently forced to use Claude Sonnet 4.6 It is such a

by hypfer 4mo ago

After having been a happy user of Qwen3.6-27B for a few weeks, due to being away from the hardware, I'm currently forced to use Claude Sonnet 4.6 It is such a downgrade. I don't understand how that's even possible. The thing has so many strongly-held opinions I did not ever ask it for, talking just way too much and generally feeling somehow dumber. Of course, being significantly larger, it will encode more knowledge, but that doesn't help me when I hate talking to it. And all that on top of the fact that talking with it costs real money. I wonder what it might be that makes me hate it so much. Maybe because it doesn't see itself as a tool but almost an equal? As if its opinions would have weight. Qwen too can act like an overeager intern, but if you tell it that it is an idiot, it will drop that ego. Not so much with Claude. In my experience, anyway. Anyway, point is: full ack on that headline.

kitd 4mo ago
Funny that coding agents have personalities, including "that colleague" you want to avoid even if you know they're probably quite good at what they do!
- otabdeveloper4 4mo ago
  That's exactly what RLHF is for. (In fact, "that colleague" might have even been the source of the RLHF training set.)
MostlyStable 4mo ago
Curious if you have tried custom instructions. I was never quite as unhappy with Claude's voice as you appear to be, but there were several things I didn't like. A custom prompt fixed almost all of them.
- clickety_clack 4mo ago
  I think it would be very hard to convince someone to pay $100/mo to go back to Claude if they have a local model up and running, particularly now that model improvement has basically been stalled for the last 6 months. It’s so easy to set it up for yourself now too with things like LM studio. That said, there will always be unsophisticated users who can’t figure it out, so there will always be someone there to pay.
  - chrisweekly 4mo ago
    Not everyone has the right hardware.
    - clickety_clack 4mo ago
      I guess I’m thinking of the $100/mo users, for whom it’s probably possible to get the right hardware.
      - lukeschlather 4mo ago
        The hardware to match Opus costs at least $200k, and you have to maintain it, and it's still not going to be as good.
  - Scoundreller 4mo ago
    The third category are the occasional users that won’t have the hardware and won’t stomach a monthly fee for “unlimited” but are happy to pay-per-use. I’d think the volume for that category would be low but LLMs aren’t just for coding.
    - dghlsakjg 4mo ago
      I’m probably the third category. I like experimenting and trying different models and techniques. I want api access for my own apps and Claude subscriptions don’t have that. Sure I could splash out a ton of money for a high ram Mac, but deepseek is so dirt cheap that I think depreciation on a high end machine costs more than my api spend. Example of what I’m using it for: building a semantic database of podcast content (podcast discoverability sucks on an episode level). I need a cheap LLM, an embedder, a transcriber, none of which Claude will do. My api costs for coding agents plus running apps are about ~$20/month, but I get more than just chat + Claude code. If all I was doing was pumping an employers codebase through a coding agent, Claude would be the answer.
  - MostlyStable 4mo ago
    The person I was replying to specifically said that the Claude will "encode more knowledge" and that their problem was that they didn't like talking to Claude. It sounds like they think that Claude is at least slightly more functional. And the "not liking talking to it" is probably fixable. Someone for whom a local model works, and for whom the economics make sense, should absolutely run a local model and I wouldn't try to convince them otherwise. I'm sure it's the right choice for a lot of people. But not liking the personality of Claude is probably not a great reason on its own, given the minuscule amount of effort it takes to fix.
StevenWaterman 4mo ago
Yep, I daily drive Qwen3.6-27B (including for work), have done pretty much since it came out. IMO it's the only (small-ish, local) model worth using, if you can run it. It might not be as good as Opus at "add X large feature" but I don't want that in a model. I want to do the thinking while it does the typing. And Qwen 3.6 27B is perfectly good at that (while in my experience models like the 35A3B and gemma are significant downgrades) Plus, I never have to worry about rate limits, quotas, or sitting in a queue during peak time. And I can always see its full thoughts, don't have to worry about where my data is getting sent, and know it can't get secretly nerfed. Running on 2x 3090, 500-1000tok/s prefill and 60tok/s output at Q6_K_XL with MTP on llama.cpp, 220k tokens context window (starts to get a bit dumb above 160k ish), no KV quantization
- giancarlostoro 4mo ago
  > (starts to get a bit dumb above 160k ish) If open models can ever hold roughly 600k token windows, I'll be really excited, I found that around 300 ~ 400k of Claude reading through your codebase results in better outputs. I also have Claude read official docs instead of "guessing" as to how to do something.
  - StevenWaterman 4mo ago
    I think we'll get there. Right now it works for me, because I'm naturally pretty verbose in my prompts, and know the codebase well, so I know what it needs to look at. Plus subagents for anything exploratory. I think deepseek v4 pro has 1m context and does pretty well up to around 600k. But if you have the hardware to run that locally, you already know Even then if there's a smaller model with 1M context, you'll need a ton of RAM to actually run it at full 1M. I guess that's why you don't see it too much. Anyone that could run Qwen 3.6 27B with 1m context would be better off running a much bigger model with smaller context instead, in the same amount of VRAM. In terms of optimizing further, huge context + KV quantization sounds like a terrible idea, but there's some decent innovation in sparse attention, KV cache rotation allowing Q8 to perform nearly as well as full 16-bit precision, plus some ideas around offloading KV cache to system RAM (but I'm skeptical)
derethanhausen 4mo ago
I would not generalize based on experiences with Sonnet. The flagship models (Opus being the claude equivalent) are dramatically better.
- hypfer 4mo ago
  Opus in my experience is equally unpleasant "character"-wise, but at least it actually gets stuff done more often, so it's at least slightly more earned at that. It's still a neurotic cargo-culting dogmatic idiot, but one that at least sometimes does produce deliverables instead of only bottom-tier HN-esque opinions. Hmm. I think I might just fundamentally disagree with Anthropic about the idea of what a "tool" should be.
  - KaoruAoiShiho 4mo ago
    Fable largely fixed the annoying chatterness so sucks that it's gone now.
  - hypfer 4mo ago
    This morning I have been blessed with an example of the exact behavior that is so infuriating. > But re-reading the comment: > "In the real world however, it does not. Hence, in the future we might fail this check even if it works within this limited check." > The comment says "we might fail this check even if it works" — implying the original intent was to always fail (return 1) as a conservative stance, leaving room to tighten it later. The new code does the opposite of what the comment describes. > So either: > 1) The comment needs updating to reflect the new intent, or > 2) The logic is wrong and should still return 1 unconditionally if the original intent was preserved. > Which is it — did you intentionally want to succeed when X works, making the comment outdated? Or should the return 1 be restored? > User: did you see the "in the future"? > Yes, fair point — "in the future we might fail this check" is forward-looking, not a statement of current intent. The comment is saying the door is left open to tightening the check later, not that it should currently always fail. So the current code is correct and the comment still holds. All good. ___ It reads/comprehends 5% of what is on screen but acts like it knows it all, bossing you around. Why is it like this. What's wrong with Anthropic and what's wrong with people _not_ getting incredibly angry at that unearned smartass.
giancarlostoro 4mo ago
There's a model on Huggingface where someone takes Qwen and makes it think Opus style, and that one seems to be decent, not sure if they have the 27B variant in that style. I do wonder if you can tweak your system prompt to force Qwen to behave better?
- whythismatters 4mo ago
  Yes, Qwopus :) I've been pleasantly surprised by its quality
  - giancarlostoro 4mo ago
    Seen that one too, same guy I'm thinking of too, havent had a chance to try all of their models. For anyone curious I believe the username is Jackrong on huggingface? They've got several models out on there each focused on programming from different approaches.
- StevenWaterman 4mo ago
  You read the OP backwards, they said Sonnet is a downgrade from Qwen, and prefer Qwen's tone
  - giancarlostoro 4mo ago
    Sure, but my argument still holds, the idea is that Qwen reasons the way that Opus on High (what is now Max or whatever?) level thinking to reason about problems instead of its standard approach.
indoordin0saur 4mo ago
Very curious what hardware you're running this on!
- hypfer 4mo ago
  The same 24GB VRAM RTX 4090 I bought to play Cyberpunk 2077 with. Works perfectly fine in llama.cpp throwing 70+t/s at me with 128k q8 K/V context when using the IQ4_NL quant + MTP at q4 MTP K/V. Also leaving this here because you might find it useful: https://hypfer.github.io/will-it-fit-llama-cpp/ https://hypfer.github.io/will-it-fit-llama-cpp/
  - cdelsolar 4mo ago
    What did you call me?
  - indoordin0saur 4mo ago
    Nice! Do you do anything with that compute when you're not actively using it? Is the crypto-mining hobby still worth it? I've also wondered if such expensive hardware can be rented back out to offset cost. Looks like these cards are going for as much as $4k nowadays.
    - all2 4mo ago
      There are services where you can hook your card up and rent it out to other users. I don't know what any of them are called, but they do exist.
      - dghlsakjg 4mo ago
        Salad.com is one. (I’m unaffiliated, just happened to come across it this week while looking for a cheap option)
    - hypfer 4mo ago
      I've paid ~2k€ in 2023. Since I'm usually sitting next to it, I'm only using it when I want to use it. It can get quite loud and warm. Crypto (to my knowledge at least) moved away from GPU mining. I guess you could maybe rent out GPU compute, but - being in germany - it's not worth the legal hassle. You could of course always commit tax fraud, though I wouldn't recommend that.
chrisweekly 4mo ago
Why Sonnet 4.6 not Opus?
deleted 4mo ago
[deleted]
radium3d 4mo ago
If you think about it, they're splitting the power across millions of users. Essentially, these AI companies have YOUR hardware that YOU are paying (them) for in a cabinet at some data center. This means the hardware could easily be run locally for inference for these 'big' models. It's just a problem of dynamics-- RAM is being bought in bulk by these companies through these B200 style cards, instead of sold slowly through the open public markets. This is likely due to a combination of mass funding for the AI companies, but also they are trying to governmentally restrict which countries get access to these cards so certain countries get a head start. The only way to lock that down is to have them literally locked in their own GPU prisons (data centers). Third reason is it does make it possible to train the models faster by having them in the same data center connected directly. Having them distributed to everyone would slow down training considerably. The current way to 'own' decent RAM and GPUs right now is through the stock market it seems.
ggerganov 4mo ago
I haven't spent a dime on cloud inference, so cannot make a direct comparison like you. But I can 100% attest to the fact that Qwen3.6-27B is a very capable local model for coding tasks. Over the last month and a half I've been using it almost daily, either on my M2 Ultra or on my RTX 5090 box. I use it for small mundane tasks at ggml-org [0] - nothing really impressive, but definitely a helpful tool for a maintainer. I think I would be using it much more, if I didn't have to spend a lot of my time on reviewing PRs. Currently, I have a very lightweight harness - the pi agent with everything stripped (`pi -nc --offline`) and a short system prompt [1] to align it a bit with my style. About the generation speed: ~100-150 t/s on the RTX 5090 and ~40 t/s on the Mac. I definitely prefer running it on the RTX machine - it's so much faster. But for the sake of testing and getting wider experience with local configurations, I often run it on the Mac too. [0] - https://github.com/search?q=%22Assisted-by%22+user%3Aggml-org&type=commits&ref=advsearch https://github.com/search?q=%22Assisted-by%22+user%3Aggml-or... [1] - https://github.com/ggml-org/llama.cpp/blob/master/.pi/gg/SYSTEM.md https://github.com/ggml-org/llama.cpp/blob/master/.pi/gg/SYS...
- deleted 4mo ago
  [deleted]
- celrod 4mo ago
  What quant do you run it at? 32GB seems like cutting it close on the rtx 5090 if going 8b, but other commenters are saying 4b lobotomizes the model.
  - ggerganov 4mo ago
    As a baseline, I run all models in Q8 [0] because I want to be confident that when I observe a problem, the root cause is not due to the quantization. However, in this specific case, I use Q8 on the mac and Q4 on the RTX machine because the latter does not fit the full context at Q8. So far, I don't have conclusive evidence that the Q4 quantization affects the quality in a significant way for this model and the tasks that I am using it for. [0] https://huggingface.co/ggerganov/presets/blob/main/preset.ini https://huggingface.co/ggerganov/presets/blob/main/preset.in...
epistasis 4mo ago
> talking just way too much OMG this is such an annoying property, just shut the hell up please, and be concise. I suspect that this is an artifact of the thinking property, but please just summarize the thinking process far more concisely, where a single sentence answer is more than sufficient the frontier models seem devoted to going on to a minimum of 5 paragraphs and offering 3-5 new directions. And requests to please only offer a single step at once, or single option at once, or to even stop eagerly offering future directions is really hard to prompt correctly. And look, there I did exactly what I was complaining about...
- bityard 4mo ago
  I'm not sure to what degree you can influence how a model thinks, but you can definitely hide the thinking tokens and tell the model how you want it to talk to you. For example, the Claude web UI has an Instructions field where I have told it never to congratulate or praise me for asking questions. Earlier Copilot models used a ridiculous number of emoji and bullet lists when answering literally every prompt, I told it to knock that off and prefer detailed paragraphs in prose. Local agents/frameworks/whatever all have their equivalents for overall user preferences.
  - epistasis 4mo ago
    Thanks for the reminder! For others looking for this setting, it is currently under User Menu (click your account name in the lower left), then "Settings", then the "General" tab there's an "Instructions for Claude" box. Asking Claude for this provides incorrect instructions for me, so I'm guessing it moves around a lot.
- illegalsmile 4mo ago
  That's why you have to give claude and others directives/.md at the beginning so it doesn't go off the deep end with suggestions.
  - epistasis 4mo ago
    Yeah, I've tried, and I'm sure somebody is going to say "skill issue" but it's not so easy to get the model to do that. Maybe it should be a SKILLS.md issue. Edit: also, how can I stop the LLM from all this fake glazing, as if every question I have is some sort of unique genius insight, it's so damn annoying. I just got the third straight round of this while merely trying to get summarization of a PDF: > Good question — it gets right at a real tension in the paper. Let me check the current state of actual SV-imputation efforts, since this has moved since 2020.
dackdel 4mo ago
what kind of hardware do you need in order to run qwen3.6-27b
- iagooar 4mo ago
  I recommend MacBook M5 Max with 128 GB of RAM to run it comfortably and fast. If you have something like a regular M4, go with qwen3.6-35b-a3d - the Mixture of Expert architecture makes it run 2-3x faster than the 27b version.
  - dackdel 4mo ago
    thanks
- sbmthakur 4mo ago
  I could run it on 7900 XT with 64k context. You could run it more comfortably on a 24 gb vram.
  - dackdel 4mo ago
    thanks
- giancarlostoro 4mo ago
  Depends on which variant you pull down, but a single 5090 GPU (I know these are insanely expensive, but for context) could run either the Q8 or Q4_K_M version. It will not fit the 52GB version (BF16) on the other hand. So any modern Mac with a Pro or better processor and more than 52GB of RAM (don't forget VRAM for context window also matters!) would suffice, as someone else noted, probably a 128GB model would do the trick, and give you enough wiggle room to max out the context window. My Mac only has 16GB of VRAM (20GB total - 8 is reserved for the OS) so I have to leave room for VRAM, I usually find a model that fits in 5 to 7 GB of VRAM and then max the context window as much as I can.
  - pixelesque 4mo ago
    Note you can change the amount of shared (V)RAM reserved for the OS with: sudo sysctl iogpu.wired_limit_mb=18800 will allow you to use more, but you do need to leave a bit for the OS obviously!
ltononro 4mo ago
Well but comparing with sonnet 4.6 instead of opus 4.6,.7 or .8 doesnt make a real point I mean, pay 200 USD/month (if you have that cash, or your company has it), might not justify using local at all (unless you have some reason to suspect about data leakage)
zerd 4mo ago
I noticed Fable was quite a bit terser, and I think it's due to changes in the system prompt [0]. They're literally saying "just give me the TLDR" and "give brief updates". You can tweak a lot of that with an AGENTS.md. [0] https://twelvetables.blog/comparing-claude-fable-5s-system-prompt-to-opus-4-8/ https://twelvetables.blog/comparing-claude-fable-5s-system-p...
calebm 4mo ago
sync/ack
cmrdporcupine 4mo ago
The Anthropic models have always been annoying this way -- chatty/opinionated and Dunning-Krugerish. And love to run away and do things unprompted with me jamming my ESC ESC ESC key over and over so I can get a word in edgewise. FWIW Codex/GPT models are way less this way. Maybe to a fault. I'm setting up my DGX Spark to try Qwen 3.6 27B again, as I'm hearing a lot of good reviews. When I tried it some time ago it was still early for support in llama.cpp.
andix 4mo ago
Sonnet is extremely overpriced. It's a good model, but not worth the money Anthropic charges for it.
dyauspitr 4mo ago
Why would I want some half assed coding assist tool. I want something that takes in a requirement and spits out a finished product. It’s not your equal, it’s better than you.
linuxhansl 4mo ago
How qwen3.6:27b compare to qwen3.6:35b-a3b (MoE) in your experience (if you tried). I find the dense models are way too slow on my H/W.
- hypfer 4mo ago
  I don't really like them tbh. They are a lot faster, yes, but at least for my needs, they feel a lot dumber too. It's a shame that the gemma4 that _would_ fit 24GB VRAM is MoE. :(
- npodbielski 4mo ago
  [dead]
bjackman 4mo ago
Re being away from the HW: with Tailscale and llama-server it's now super easy to just run an inference server at home and use it from wherever you are.
- npodbielski 4mo ago
  Exactly what I do! You can have web UI for PI!
mik09 4mo ago
try qwen 3.7 or glm 5.2 or one of the larger gemma models
Shorel 4mo ago
I use tailscale to have remote access to my local models when on the move.

by hypfer 4mo ago

kitd 4mo ago

Funny that coding agents have personalities, including "that colleague" you want to avoid even if you know they're probably quite good at what they do!

otabdeveloper4 4mo ago
That's exactly what RLHF is for. (In fact, "that colleague" might have even been the source of the RLHF training set.)

MostlyStable 4mo ago

Curious if you have tried custom instructions. I was never quite as unhappy with Claude's voice as you appear to be, but there were several things I didn't like. A custom prompt fixed almost all of them.

clickety_clack 4mo ago
I think it would be very hard to convince someone to pay $100/mo to go back to Claude if they have a local model up and running, particularly now that model improvement has basically been stalled for the last 6 months. It’s so easy to set it up for yourself now too with things like LM studio. That said, there will always be unsophisticated users who can’t figure it out, so there will always be someone there to pay.
- chrisweekly 4mo ago
  Not everyone has the right hardware.
  - clickety_clack 4mo ago
    I guess I’m thinking of the $100/mo users, for whom it’s probably possible to get the right hardware.
    - lukeschlather 4mo ago
      The hardware to match Opus costs at least $200k, and you have to maintain it, and it's still not going to be as good.
- Scoundreller 4mo ago
  The third category are the occasional users that won’t have the hardware and won’t stomach a monthly fee for “unlimited” but are happy to pay-per-use. I’d think the volume for that category would be low but LLMs aren’t just for coding.
  - dghlsakjg 4mo ago
    I’m probably the third category. I like experimenting and trying different models and techniques. I want api access for my own apps and Claude subscriptions don’t have that. Sure I could splash out a ton of money for a high ram Mac, but deepseek is so dirt cheap that I think depreciation on a high end machine costs more than my api spend. Example of what I’m using it for: building a semantic database of podcast content (podcast discoverability sucks on an episode level). I need a cheap LLM, an embedder, a transcriber, none of which Claude will do. My api costs for coding agents plus running apps are about ~$20/month, but I get more than just chat + Claude code. If all I was doing was pumping an employers codebase through a coding agent, Claude would be the answer.
- MostlyStable 4mo ago
  The person I was replying to specifically said that the Claude will "encode more knowledge" and that their problem was that they didn't like talking to Claude. It sounds like they think that Claude is at least slightly more functional. And the "not liking talking to it" is probably fixable. Someone for whom a local model works, and for whom the economics make sense, should absolutely run a local model and I wouldn't try to convince them otherwise. I'm sure it's the right choice for a lot of people. But not liking the personality of Claude is probably not a great reason on its own, given the minuscule amount of effort it takes to fix.

StevenWaterman 4mo ago

Yep, I daily drive Qwen3.6-27B (including for work), have done pretty much since it came out. IMO it's the only (small-ish, local) model worth using, if you can run it. It might not be as good as Opus at "add X large feature" but I don't want that in a model. I want to do the thinking while it does the typing. And Qwen 3.6 27B is perfectly good at that (while in my experience models like the 35A3B and gemma are significant downgrades) Plus, I never have to worry about rate limits, quotas, or sitting in a queue during peak time. And I can always see its full thoughts, don't have to worry about where my data is getting sent, and know it can't get secretly nerfed. Running on 2x 3090, 500-1000tok/s prefill and 60tok/s output at Q6_K_XL with MTP on llama.cpp, 220k tokens context window (starts to get a bit dumb above 160k ish), no KV quantization

giancarlostoro 4mo ago
> (starts to get a bit dumb above 160k ish) If open models can ever hold roughly 600k token windows, I'll be really excited, I found that around 300 ~ 400k of Claude reading through your codebase results in better outputs. I also have Claude read official docs instead of "guessing" as to how to do something.
- StevenWaterman 4mo ago
  I think we'll get there. Right now it works for me, because I'm naturally pretty verbose in my prompts, and know the codebase well, so I know what it needs to look at. Plus subagents for anything exploratory. I think deepseek v4 pro has 1m context and does pretty well up to around 600k. But if you have the hardware to run that locally, you already know Even then if there's a smaller model with 1M context, you'll need a ton of RAM to actually run it at full 1M. I guess that's why you don't see it too much. Anyone that could run Qwen 3.6 27B with 1m context would be better off running a much bigger model with smaller context instead, in the same amount of VRAM. In terms of optimizing further, huge context + KV quantization sounds like a terrible idea, but there's some decent innovation in sparse attention, KV cache rotation allowing Q8 to perform nearly as well as full 16-bit precision, plus some ideas around offloading KV cache to system RAM (but I'm skeptical)

derethanhausen 4mo ago

I would not generalize based on experiences with Sonnet. The flagship models (Opus being the claude equivalent) are dramatically better.

hypfer 4mo ago
Opus in my experience is equally unpleasant "character"-wise, but at least it actually gets stuff done more often, so it's at least slightly more earned at that. It's still a neurotic cargo-culting dogmatic idiot, but one that at least sometimes does produce deliverables instead of only bottom-tier HN-esque opinions. Hmm. I think I might just fundamentally disagree with Anthropic about the idea of what a "tool" should be.
- KaoruAoiShiho 4mo ago
  Fable largely fixed the annoying chatterness so sucks that it's gone now.
- hypfer 4mo ago
  This morning I have been blessed with an example of the exact behavior that is so infuriating. > But re-reading the comment: > "In the real world however, it does not. Hence, in the future we might fail this check even if it works within this limited check." > The comment says "we might fail this check even if it works" — implying the original intent was to always fail (return 1) as a conservative stance, leaving room to tighten it later. The new code does the opposite of what the comment describes. > So either: > 1) The comment needs updating to reflect the new intent, or > 2) The logic is wrong and should still return 1 unconditionally if the original intent was preserved. > Which is it — did you intentionally want to succeed when X works, making the comment outdated? Or should the return 1 be restored? > User: did you see the "in the future"? > Yes, fair point — "in the future we might fail this check" is forward-looking, not a statement of current intent. The comment is saying the door is left open to tightening the check later, not that it should currently always fail. So the current code is correct and the comment still holds. All good. ___ It reads/comprehends 5% of what is on screen but acts like it knows it all, bossing you around. Why is it like this. What's wrong with Anthropic and what's wrong with people _not_ getting incredibly angry at that unearned smartass.

giancarlostoro 4mo ago

There's a model on Huggingface where someone takes Qwen and makes it think Opus style, and that one seems to be decent, not sure if they have the 27B variant in that style. I do wonder if you can tweak your system prompt to force Qwen to behave better?

whythismatters 4mo ago
Yes, Qwopus :) I've been pleasantly surprised by its quality
- giancarlostoro 4mo ago
  Seen that one too, same guy I'm thinking of too, havent had a chance to try all of their models. For anyone curious I believe the username is Jackrong on huggingface? They've got several models out on there each focused on programming from different approaches.
StevenWaterman 4mo ago
You read the OP backwards, they said Sonnet is a downgrade from Qwen, and prefer Qwen's tone
- giancarlostoro 4mo ago
  Sure, but my argument still holds, the idea is that Qwen reasons the way that Opus on High (what is now Max or whatever?) level thinking to reason about problems instead of its standard approach.

indoordin0saur 4mo ago

Very curious what hardware you're running this on!

hypfer 4mo ago
The same 24GB VRAM RTX 4090 I bought to play Cyberpunk 2077 with. Works perfectly fine in llama.cpp throwing 70+t/s at me with 128k q8 K/V context when using the IQ4_NL quant + MTP at q4 MTP K/V. Also leaving this here because you might find it useful: https://hypfer.github.io/will-it-fit-llama-cpp/ https://hypfer.github.io/will-it-fit-llama-cpp/
- cdelsolar 4mo ago
  What did you call me?
- indoordin0saur 4mo ago
  Nice! Do you do anything with that compute when you're not actively using it? Is the crypto-mining hobby still worth it? I've also wondered if such expensive hardware can be rented back out to offset cost. Looks like these cards are going for as much as $4k nowadays.
  - all2 4mo ago
    There are services where you can hook your card up and rent it out to other users. I don't know what any of them are called, but they do exist.
    - dghlsakjg 4mo ago
      Salad.com is one. (I’m unaffiliated, just happened to come across it this week while looking for a cheap option)
  - hypfer 4mo ago
    I've paid ~2k€ in 2023. Since I'm usually sitting next to it, I'm only using it when I want to use it. It can get quite loud and warm. Crypto (to my knowledge at least) moved away from GPU mining. I guess you could maybe rent out GPU compute, but - being in germany - it's not worth the legal hassle. You could of course always commit tax fraud, though I wouldn't recommend that.

chrisweekly 4mo ago

Why Sonnet 4.6 not Opus?

deleted 4mo ago

[deleted]

radium3d 4mo ago

If you think about it, they're splitting the power across millions of users. Essentially, these AI companies have YOUR hardware that YOU are paying (them) for in a cabinet at some data center. This means the hardware could easily be run locally for inference for these 'big' models. It's just a problem of dynamics-- RAM is being bought in bulk by these companies through these B200 style cards, instead of sold slowly through the open public markets. This is likely due to a combination of mass funding for the AI companies, but also they are trying to governmentally restrict which countries get access to these cards so certain countries get a head start. The only way to lock that down is to have them literally locked in their own GPU prisons (data centers). Third reason is it does make it possible to train the models faster by having them in the same data center connected directly. Having them distributed to everyone would slow down training considerably. The current way to 'own' decent RAM and GPUs right now is through the stock market it seems.

ggerganov 4mo ago

I haven't spent a dime on cloud inference, so cannot make a direct comparison like you. But I can 100% attest to the fact that Qwen3.6-27B is a very capable local model for coding tasks. Over the last month and a half I've been using it almost daily, either on my M2 Ultra or on my RTX 5090 box. I use it for small mundane tasks at ggml-org [0] - nothing really impressive, but definitely a helpful tool for a maintainer. I think I would be using it much more, if I didn't have to spend a lot of my time on reviewing PRs. Currently, I have a very lightweight harness - the pi agent with everything stripped (`pi -nc --offline`) and a short system prompt [1] to align it a bit with my style. About the generation speed: ~100-150 t/s on the RTX 5090 and ~40 t/s on the Mac. I definitely prefer running it on the RTX machine - it's so much faster. But for the sake of testing and getting wider experience with local configurations, I often run it on the Mac too. [0] - https://github.com/search?q=%22Assisted-by%22+user%3Aggml-org&type=commits&ref=advsearch https://github.com/search?q=%22Assisted-by%22+user%3Aggml-or... [1] - https://github.com/ggml-org/llama.cpp/blob/master/.pi/gg/SYSTEM.md https://github.com/ggml-org/llama.cpp/blob/master/.pi/gg/SYS...

deleted 4mo ago
[deleted]
celrod 4mo ago
What quant do you run it at? 32GB seems like cutting it close on the rtx 5090 if going 8b, but other commenters are saying 4b lobotomizes the model.
- ggerganov 4mo ago
  As a baseline, I run all models in Q8 [0] because I want to be confident that when I observe a problem, the root cause is not due to the quantization. However, in this specific case, I use Q8 on the mac and Q4 on the RTX machine because the latter does not fit the full context at Q8. So far, I don't have conclusive evidence that the Q4 quantization affects the quality in a significant way for this model and the tasks that I am using it for. [0] https://huggingface.co/ggerganov/presets/blob/main/preset.ini https://huggingface.co/ggerganov/presets/blob/main/preset.in...

epistasis 4mo ago

> talking just way too much OMG this is such an annoying property, just shut the hell up please, and be concise. I suspect that this is an artifact of the thinking property, but please just summarize the thinking process far more concisely, where a single sentence answer is more than sufficient the frontier models seem devoted to going on to a minimum of 5 paragraphs and offering 3-5 new directions. And requests to please only offer a single step at once, or single option at once, or to even stop eagerly offering future directions is really hard to prompt correctly. And look, there I did exactly what I was complaining about...

bityard 4mo ago
I'm not sure to what degree you can influence how a model thinks, but you can definitely hide the thinking tokens and tell the model how you want it to talk to you. For example, the Claude web UI has an Instructions field where I have told it never to congratulate or praise me for asking questions. Earlier Copilot models used a ridiculous number of emoji and bullet lists when answering literally every prompt, I told it to knock that off and prefer detailed paragraphs in prose. Local agents/frameworks/whatever all have their equivalents for overall user preferences.
- epistasis 4mo ago
  Thanks for the reminder! For others looking for this setting, it is currently under User Menu (click your account name in the lower left), then "Settings", then the "General" tab there's an "Instructions for Claude" box. Asking Claude for this provides incorrect instructions for me, so I'm guessing it moves around a lot.
illegalsmile 4mo ago
That's why you have to give claude and others directives/.md at the beginning so it doesn't go off the deep end with suggestions.
- epistasis 4mo ago
  Yeah, I've tried, and I'm sure somebody is going to say "skill issue" but it's not so easy to get the model to do that. Maybe it should be a SKILLS.md issue. Edit: also, how can I stop the LLM from all this fake glazing, as if every question I have is some sort of unique genius insight, it's so damn annoying. I just got the third straight round of this while merely trying to get summarization of a PDF: > Good question — it gets right at a real tension in the paper. Let me check the current state of actual SV-imputation efforts, since this has moved since 2020.

dackdel 4mo ago

what kind of hardware do you need in order to run qwen3.6-27b

iagooar 4mo ago
I recommend MacBook M5 Max with 128 GB of RAM to run it comfortably and fast. If you have something like a regular M4, go with qwen3.6-35b-a3d - the Mixture of Expert architecture makes it run 2-3x faster than the 27b version.
- dackdel 4mo ago
  thanks
sbmthakur 4mo ago
I could run it on 7900 XT with 64k context. You could run it more comfortably on a 24 gb vram.
- dackdel 4mo ago
  thanks
giancarlostoro 4mo ago
Depends on which variant you pull down, but a single 5090 GPU (I know these are insanely expensive, but for context) could run either the Q8 or Q4_K_M version. It will not fit the 52GB version (BF16) on the other hand. So any modern Mac with a Pro or better processor and more than 52GB of RAM (don't forget VRAM for context window also matters!) would suffice, as someone else noted, probably a 128GB model would do the trick, and give you enough wiggle room to max out the context window. My Mac only has 16GB of VRAM (20GB total - 8 is reserved for the OS) so I have to leave room for VRAM, I usually find a model that fits in 5 to 7 GB of VRAM and then max the context window as much as I can.
- pixelesque 4mo ago
  Note you can change the amount of shared (V)RAM reserved for the OS with: sudo sysctl iogpu.wired_limit_mb=18800 will allow you to use more, but you do need to leave a bit for the OS obviously!

ltononro 4mo ago

Well but comparing with sonnet 4.6 instead of opus 4.6,.7 or .8 doesnt make a real point I mean, pay 200 USD/month (if you have that cash, or your company has it), might not justify using local at all (unless you have some reason to suspect about data leakage)

zerd 4mo ago

I noticed Fable was quite a bit terser, and I think it's due to changes in the system prompt [0]. They're literally saying "just give me the TLDR" and "give brief updates". You can tweak a lot of that with an AGENTS.md. [0] https://twelvetables.blog/comparing-claude-fable-5s-system-prompt-to-opus-4-8/ https://twelvetables.blog/comparing-claude-fable-5s-system-p...

calebm 4mo ago

sync/ack

cmrdporcupine 4mo ago

The Anthropic models have always been annoying this way -- chatty/opinionated and Dunning-Krugerish. And love to run away and do things unprompted with me jamming my ESC ESC ESC key over and over so I can get a word in edgewise. FWIW Codex/GPT models are way less this way. Maybe to a fault. I'm setting up my DGX Spark to try Qwen 3.6 27B again, as I'm hearing a lot of good reviews. When I tried it some time ago it was still early for support in llama.cpp.

andix 4mo ago

Sonnet is extremely overpriced. It's a good model, but not worth the money Anthropic charges for it.

dyauspitr 4mo ago

Why would I want some half assed coding assist tool. I want something that takes in a requirement and spits out a finished product. It’s not your equal, it’s better than you.

linuxhansl 4mo ago

How qwen3.6:27b compare to qwen3.6:35b-a3b (MoE) in your experience (if you tried). I find the dense models are way too slow on my H/W.

hypfer 4mo ago
I don't really like them tbh. They are a lot faster, yes, but at least for my needs, they feel a lot dumber too. It's a shame that the gemma4 that _would_ fit 24GB VRAM is MoE. :(
npodbielski 4mo ago
[dead]

bjackman 4mo ago

Re being away from the HW: with Tailscale and llama-server it's now super easy to just run an inference server at home and use it from wherever you are.

npodbielski 4mo ago
Exactly what I do! You can have web UI for PI!

mik09 4mo ago

try qwen 3.7 or glm 5.2 or one of the larger gemma models

Shorel 4mo ago

I use tailscale to have remote access to my local models when on the move.