10 ms·

Qwen3.6-27B: Flagship-Level Coding in a 27B Dense Model

• denniszelada 6mo ago
  [dead]
• amunozo 6mo ago
  A bit skeptical about a 27B model comparable to opus...
  • wesammikhail 6mo ago
    you'd be surprised how good small models have gotten. Size of the model isnt all that matters.
    • verdverm 6mo ago
      My experience with qwen-3.6:35B-A3B reinforces this, gonna give this a spin when unsloth has quants available Gemini flash was just as good as pro for most tasks with good prompts, tools, and context. Gemma 4 was nearly as good as flash and Qwen 3.6 appears to be even better.
      • cassianoleal 6mo ago
        > when unsloth has quants available https://huggingface.co/unsloth/Qwen3.6-27B-GGUF https://huggingface.co/unsloth/Qwen3.6-27B-GGUF
        • verdverm 6mo ago
          That was quick (compared to the 1T Kimi-2.6, not surprising)
          • danielhanchen 6mo ago
            Haha :) We had some issues with Kimi-2.6 since it was int4 and we were investigating how to handle it :)
            • verdverm 6mo ago
              Appreciate what y'all do! We were slacking about how many HGX-B300 it would take to run Kimi and it looks like we could actually fit 2-3 Kimis on a single HGX.
              • danielhanchen 6mo ago
                Sorry on the delay - oh haha that would be cool :) We did release 2bit dynamic ones, but unsure if they'll be helpful
    • freedomben 6mo ago
      Plus you can control thinking time a lot more, so when Anthropic lobotomizes Opus on you...
    • dudefeliciano 6mo ago
      > Size of the model isnt all that matters. What matters is the motion in the tokens
  • esafak 6mo ago
    Some of these benchmarks are supposedly easy to game. Which ones should we pay attention to?
    • underlines 6mo ago
      well, your own, unleaked ones, representing your real workloads. if you can't afford to do that, look at a lot of them, eg. on artificialanalysis.com they merge multiple benchmarks across weighted categories and build an Intelligence Score, Coding Score and Agentic score.
    • WarmWash 6mo ago
      ARC-AGI 2 GLM 5 scores 5% on the semi-private set, compared to SOTA models which hover around 80%.
    • cbg0 6mo ago
      None. Try them out with your own typical tasks to see the performance.
    • NitpickLawyer 6mo ago
      SWE-REbench should not be gameable. They collect new issues from live repos, and if you check 1-2 months after a model was released, you can get an idea. But even that would be "benchmaxxxable", which is an overloaded term that can mean many things, but the most vanilla interpretation is that with RL you can get a model to follow a certain task pretty well, but it'll get "stuck" on that task type, or "stubborn" when asked similar but sufficiently different tasks. So for swe-rebench that would be "it fixes bugs in these types of repos, under this harness, but ask it to do soemthing else in a repo and you might not get the same results". In a nutshell.
  • originalvichy 6mo ago
    For at least a year now, it has been clear that data quality and fine-tuning are the main sources of improvement for mediym-level models. Size != quality for specialized, narrow use cases such as coding. It’s not a surprise that models are leapfrogging each other when the engineers are able to incorporate better code examples and reasoning traces, which in turn bring higher quality outputs.
    • cbg0 6mo ago
      If all you're looking at is benchmarks that might be true, but those are way too easy to game. Try using this model alongside Opus for some work in Rust/C++ and it'll be night and day. You really can't compare a model that's got trillions of parameters to a 27B one.
      • otabdeveloper4 6mo ago
        > ...and it'll be night and day. That's just, like, your opinion, man. > You really can't compare a model that's got trillions of parameters to a 27B one. Parameter count doesn't matter much when coding. You don't need in-depth general knowledge or multilingual support in a coding model.
        • cbg0 6mo ago
          I often do need in-depth general knowledge in my coding model so that I don't have to explain domain specific logic to it every time and so that it can have some sense of good UX.
  • Aurornis 6mo ago
    You should be skeptical. Benchmark racing is the current meta game in open weight LLMs. Every release is accompanied by claims of being as good as Sonnet or Opus, but when I try them (even hosted full weights) they’re far from it. Impressive for the size, though!
  • jjcm 6mo ago
    Opus 4.5 mind you, but I’m not too surprised given how good 3.5 was and how good the qwopus fine tune was. The model was shown to benefit heavily from further RL.
  • cmrdporcupine 6mo ago
    A small model can be made to be "comparable to Opus" in some narrow domains, and that's what they've done here. But when actually employed to write code they will fall over when they leave that specific domain. Basically they might have skill but lack wisdom. Certainly at this size they will lack anywhere close to the same contextual knowledge. Still these things could be useful in the context of more specialized tooling, or in a harness that heavily prompts in the right direction, or as a subagent for a "wiser" larger model that directs all the planning and reviews results.
  • rubiquity 6mo ago
    You should try it out. I'm incredibly impressed with Qwen 3.5 27B for systems programming work. I use Opus and Sonnet at work and Qwen 3.x at home for fun and barely notice a difference given that systems programming work needs careful guidance for any model currently. I don't try to one shot landing pages or whatever.
    • bityard 6mo ago
      Are you using the same agent/harness/whatever for both Claude and Qwen, or something different for each one?
      • rubiquity 6mo ago
        I use Pi at home and Claude Code at work (no choice). I use bone stock Pi. No extensions.
    • amunozo 6mo ago
      Is it available for API use? I don't have a laptop capable of running it.
  • kgeist 6mo ago
    From what I understand, ~30b is enough "intelligence" to make coding/reasoning etc. work, in general. Above ~30b, it's less about intelligence, and more about memorization. Larger models fail less and one-shot more often because they can memorize more APIs (documentation, examples, etc). Also from my experience, if a task is ambiguous, Sonnet has a better "intuition" of what my intent is. Probably also because of memorization, it has "access" to more repositories in its compressed knowledge to infer my intent more accurately.
• pama 6mo ago
  Has anyone tested it at home yet and wants to share early impressions?
  • lreeves 6mo ago
    I have been kicking the tires for about 40 minutes since it downloaded and it seems excellent at general tasks, image comprehension and coding/tool-calling (using VLLM to serve it). I think it squeaks past Gemma4 but it's hard to tell yet.
    • alfonsodev 6mo ago
      good to hear! Do you mind sharing your setup and tokens / seconds performance ?
      • lreeves 6mo ago
        I'm running the unquantized base model on 2xA6000s (Ampere gen, 48GB each). Runs at about 25 tokens/second.
        • NitpickLawyer 6mo ago
          FYI they also released FP8 quants, and those should be faster on your setup (we have the same). As long as you keep kv at 16bit, FP8 should be close-to-lossless compared to 16bit, but with more context available and faster inference speed.
  • mswphd 6mo ago
    An "obvious" point to make is that it is not particularly usable on a unified memory machine. Only getting 9 tok/s (for Q6 quants) using a Macbook M4 Pro 48GB memory (though with GGUFs, not mlx). The quality seems fine, but the 9 tok/s mean I only tried it out briefly.
• anonzzzies 6mo ago
  I wish that all announcements of models would show what (consumer) hardware you can run this on today, costs and tok/s.
  • underlines 6mo ago
    depends on format, compute type, quantization and kv cache size.
    • mottosso 6mo ago
      Specs for whatever they used to achieve the benchmarks would be a good start.
      • Aurornis 6mo ago
        The benchmarks are from the unquantized model they release. This will only run on server hardware, some workstation GPUs, or some 128GB unified memory systems. It’s a situation where if you have to ask, you can’t run the exact model they released. You have to wait for quantizations to smaller sizes, which come in a lot of varieties and have quality tradeoffs.
        • deleted 6mo ago
          [deleted]
        • bityard 6mo ago
          This would likely run fine in just 96 GB of VRAM, by my estimation. Well within the ability of an enthusiastic hobbyist with a few thousand dollars of disposable income. Quantizations are already out: https://huggingface.co/unsloth/Qwen3.6-27B-GGUF https://huggingface.co/unsloth/Qwen3.6-27B-GGUF
      • bityard 6mo ago
        The benchmarks in the model card are purported to be measurements of model quality (ability to perform tasks with few errors), not speed. They almost certainly run these benchmarks on their own cloud infrastructure (Alibaba afaik), which is typically not hardware that even the most enthusiastic homelab hobbyist can afford.
• vladgur 6mo ago
  This is getting very close to fit a single 3090 with 24gb VRAM :)
  • originalvichy 6mo ago
    Yup! Smaller quants will fit within 24GB but they might sacrifice context length. I’m excited to try out the MLX version to see if 32GB of memory from a Pro M-series Mac can get some acceptable tok/s with longer context. HuggingFace has uploaded some MLX versions already.
    • ycui1986 6mo ago
      32GB RAM on mac also need to host OS, software, and other stuff. There may not even be 24GB VRAM left for the model.
    • donmcronald 6mo ago
      I have an Mini M4 Pro with 64GB of 273GB/s memory bandwidth and it's borderline with 3.5-27B. I assume this one is the same. I don't know a ton, but I think it's the memory bandwidth that limits it. It's similar on a DGX Spark I have access to (almost the same memory bandwidth). It's been a while since I tried it, but I think I was getting around 12-15 tokens per second an that feels slow when you're used to the big commercial models. Whenever I actually want to do stuff with the open source models, I always find myself falling back to OpenRouter. I tried Intel/Qwen3.6-35B-A3B-int4-AutoRound on a DGX Spark a couple days ago and that felt usable speed wise. I don't know about quality, but that's like running a 3B parameter model. 27B is a lot slower. I'm not sure if I "get" the local AI stuff everyone is selling. I love the idea of it, but what's the point of 128GB of shared memory on a DGX Spark if I can only run a 20-30GB model before the slow speed makes it unusable?
      • verdverm 6mo ago
        There are a number of DGX benchmarks for these recent gemma-4 / qwen-3.6 models on the nvidia forum, ex: https://forums.developer.nvidia.com/t/qwen-qwen3-6-35b-a3b-and-fp8-has-landed/366822/5 https://forums.developer.nvidia.com/t/qwen-qwen3-6-35b-a3b-a...
• originalvichy 6mo ago
  Good news! Friendly reminder: wait a couple weeks to judge the ”final” quality of these free models. Many of them suffer from hidden bugs when connected to an inference backend or bad configs that slow them down. The dev community usually takes a week or two to find the most glaring issues. Some of them may require patches to tools like llama.cpp, and some require users to avoid specific default options. Gemma 4 had some issues that were ironed out within a week or two. This model is likely no different. Take initial impressions with a grain of salt.
  • Aurornis 6mo ago
    Good advice for all new LLM experimenters. The bugs come from the downstream implementations and quantizations (which inherit bugs in the tools). Expect to update your tools and redownload the quants multiple times over 2-4 weeks. There is a mad rush to be first to release quants and first to submit PRs to the popular tools, but the output is often not tested much before uploading. If you experiment with these on launch week, you are the tester. :)
  • jjcm 6mo ago
    This is probably less likely with this model, as it’s almost certainly a further RL training continuation of 3.5 27b. The bugs with this architecture were worked out when that dropped.
    • originalvichy 6mo ago
      Valuable note!
• spwa4 6mo ago
  Unsloth quants available: https://unsloth.ai/docs/models/qwen3.6 https://unsloth.ai/docs/models/qwen3.6
  • ljosifov 6mo ago
    ~/llama.cpp$ build-.../bin/llama-batched-bench -m models/....gguf -npp 512,1024,2048,4096,8192,16384,32768 -ntg 128 -npl 1 -c 36000 On amd 7900xtx Qwen3.6-27B-Q4_K_M | PP | TG | B | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s | T s | S t/s | |-------|--------|------|--------|----------|----------|----------|----------|----------|----------| | 512 | 128 | 1 | 640 | 0.743 | 689.35 | 4.605 | 27.80 | 5.348 | 119.68 | | 1024 | 128 | 1 | 1152 | 1.188 | 862.17 | 4.573 | 27.99 | 5.761 | 199.96 | | 2048 | 128 | 1 | 2176 | 2.566 | 798.09 | 4.602 | 27.81 | 7.168 | 303.57 | | 4096 | 128 | 1 | 4224 | 5.936 | 690.00 | 4.639 | 27.59 | 10.575 | 399.43 | | 8192 | 128 | 1 | 8320 | 15.034 | 544.90 | 4.729 | 27.06 | 19.763 | 420.98 | | 16384 | 128 | 1 | 16512 | 42.807 | 382.74 | 4.886 | 26.20 | 47.694 | 346.21 | | 32768 | 128 | 1 | 32896 | 137.377 | 238.53 | 5.188 | 24.67 | 142.566 | 230.74 | Qwen3.6-27B-IQ4_NL | PP | TG | B | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s | T s | S t/s | |-------|--------|------|--------|----------|----------|----------|----------|----------|----------| | 512 | 128 | 1 | 640 | 0.535 | 957.45 | 3.715 | 34.45 | 4.250 | 150.59 | | 1024 | 128 | 1 | 1152 | 1.124 | 911.16 | 3.677 | 34.81 | 4.801 | 239.97 | | 2048 | 128 | 1 | 2176 | 2.447 | 836.89 | 3.698 | 34.62 | 6.145 | 354.13 | | 4096 | 128 | 1 | 4224 | 5.711 | 717.17 | 3.729 | 34.32 | 9.441 | 447.43 | | 8192 | 128 | 1 | 8320 | 14.615 | 560.52 | 3.821 | 33.50 | 18.436 | 451.30 | | 16384 | 128 | 1 | 16512 | 41.966 | 390.41 | 3.967 | 32.26 | 45.933 | 359.48 | | 32768 | 128 | 1 | 32896 | 135.789 | 241.32 | 4.253 | 30.09 | 140.042 | 234.90 | On mbp M2 Max Qwen3.6-27B-UD-Q8_K_XL | PP | TG | B | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s | T s | S t/s | |-------|--------|------|--------|----------|----------|----------|----------|----------|----------| | 512 | 128 | 1 | 640 | 2.583 | 198.18 | 22.049 | 5.81 | 24.633 | 25.98 | | 1024 | 128 | 1 | 1152 | 8.321 | 123.06 | 22.364 | 5.72 | 30.685 | 37.54 | | 2048 | 128 | 1 | 2176 | 17.873 | 114.59 | 23.290 | 5.50 | 41.164 | 52.86 | | 4096 | 128 | 1 | 4224 | 41.967 | 97.60 | 23.624 | 5.42 | 65.591 | 64.40 | | 8192 | 128 | 1 | 8320 | 68.722 | 119.20 | 21.077 | 6.07 | 89.799 | 92.65 | | 16384 | 128 | 1 | 16512 | 142.184 | 115.23 | 22.026 | 5.81 | 164.210 | 100.55 | | 32768 | 128 | 1 | 32896 | 339.778 | 96.44 | 24.465 | 5.23 | 364.243 | 90.31 | Compared to similar prior models On amd 7900xtx Qwen3.6-35B-A3B-UD-Q4_K_S | PP | TG | B | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s | T s | S t/s | |-------|--------|------|--------|----------|----------|----------|----------|----------|----------| | 512 | 128 | 1 | 640 | 0.203 | 2517.60 | 1.482 | 86.35 | 1.686 | 379.67 | | 1024 | 128 | 1 | 1152 | 0.427 | 2399.22 | 1.471 | 87.04 | 1.897 | 607.15 | | 2048 | 128 | 1 | 2176 | 0.946 | 2165.23 | 1.478 | 86.59 | 2.424 | 897.67 | | 4096 | 128 | 1 | 4224 | 2.253 | 1818.33 | 1.502 | 85.22 | 3.755 | 1125.01 | | 8192 | 128 | 1 | 8320 | 5.849 | 1400.51 | 1.525 | 83.91 | 7.375 | 1128.17 | | 16384 | 128 | 1 | 16512 | 17.115 | 957.27 | 1.589 | 80.55 | 18.705 | 882.78 | | 32768 | 128 | 1 | 32896 | 56.008 | 585.06 | 1.704 | 75.10 | 57.712 | 570.00 | Qwen3.6-35B-A3B-UD-IQ4_XS | PP | TG | B | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s | T s | S t/s | |-------|--------|------|--------|----------|----------|----------|----------|----------|----------| | 512 | 128 | 1 | 640 | 0.204 | 2508.94 | 1.313 | 97.46 | 1.517 | 421.78 | | 1024 | 128 | 1 | 1152 | 0.423 | 2418.64 | 1.296 | 98.80 | 1.719 | 670.18 | | 2048 | 128 | 1 | 2176 | 0.946 | 2164.61 | 1.323 | 96.78 | 2.269 | 959.13 | | 4096 | 128 | 1 | 4224 | 2.235 | 1832.54 | 1.326 | 96.52 | 3.561 | 1186.06 | | 8192 | 128 | 1 | 8320 | 5.845 | 1401.44 | 1.352 | 94.70 | 7.197 | 1156.03 | | 16384 | 128 | 1 | 16512 | 17.096 | 958.38 | 1.417 | 90.33 | 18.513 | 891.94 | | 32768 | 128 | 1 | 32896 | 56.013 | 585.00 | 1.530 | 83.66 | 57.543 | 571.67 | Carnice-Qwen3.6-MoE-35B-A3B-Q4_K_S | PP | TG | B | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s | T s | S t/s | |-------|--------|------|--------|----------|----------|----------|----------|----------|----------| | 512 | 128 | 1 | 640 | 0.205 | 2499.78 | 1.483 | 86.31 | 1.688 | 379.16 | | 1024 | 128 | 1 | 1152 | 0.434 | 2361.36 | 1.448 | 88.40 | 1.882 | 612.25 | | 2048 | 128 | 1 | 2176 | 0.947 | 2161.87 | 1.478 | 86.62 | 2.425 | 897.27 | | 4096 | 128 | 1 | 4224 | 2.259 | 1813.00 | 1.472 | 86.94 | 3.732 | 1131.98 | | 8192 | 128 | 1 | 8320 | 5.892 | 1390.42 | 1.505 | 85.06 | 7.397 | 1124.85 | | 16384 | 128 | 1 | 16512 | 17.397 | 941.77 | 1.568 | 81.61 | 18.965 | 870.63 | | 32768 | 128 | 1 | 32896 | 56.296 | 582.07 | 1.690 | 75.74 | 57.986 | 567.31 | Nemotron-Cascade-2-30B-A3B-IQ4_XS | PP | TG | B | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s | T s | S t/s | |-------|--------|------|--------|----------|----------|----------|----------|----------|----------| | 512 | 128 | 1 | 640 | 0.195 | 2622.33 | 0.972 | 131.69 | 1.167 | 548.30 | | 1024 | 128 | 1 | 1152 | 0.407 | 2514.76 | 0.934 | 137.10 | 1.341 | 859.16 | | 2048 | 128 | 1 | 2176 | 0.854 | 2396.99 | 0.942 | 135.90 | 1.796 | 1211.42 | | 4096 | 128 | 1 | 4224 | 1.895 | 2161.89 | 0.953 | 134.36 | 2.847 | 1483.50 | | 8192 | 128 | 1 | 8320 | 4.593 | 1783.70 | 0.967 | 132.43 | 5.559 | 1496.60 | | 16384 | 128 | 1 | 16512 | 12.213 | 1341.53 | 0.996 | 128.56 | 13.209 | 1250.10 | | 32768 | 128 | 1 | 32896 | 36.998 | 885.66 | 1.059 | 120.89 | 38.057 | 864.39 | On mbp M2 Max Qwen3.6-35B-A3B-UD-Q6_K_XL | PP | TG | B | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s | T s | S t/s | |-------|--------|------|--------|----------|----------|----------|----------|----------|----------| | 512 | 128 | 1 | 640 | 0.540 | 947.31 | 2.489 | 51.42 | 3.030 | 211.22 | | 1024 | 128 | 1 | 1152 | 0.951 | 1077.21 | 3.237 | 39.54 | 4.188 | 275.10 | | 2048 | 128 | 1 | 2176 | 2.994 | 684.10 | 3.139 | 40.77 | 6.133 | 354.80 | | 4096 | 128 | 1 | 4224 | 6.245 | 655.85 | 3.210 | 39.88 | 9.455 | 446.75 | | 8192 | 128 | 1 | 8320 | 12.411 | 660.08 | 3.284 | 38.98 | 15.694 | 530.13 | | 16384 | 128 | 1 | 16512 | 28.321 | 578.51 | 3.584 | 35.71 | 31.905 | 517.53 | | 32768 | 128 | 1 | 32896 | 65.725 | 498.56 | 4.029 | 31.77 | 69.754 | 471.60 | Nemotron-Cascade-2-30B-A3B-Q8_0 | PP | TG | B | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s | T s | S t/s | |-------|--------|------|--------|----------|----------|----------|----------|----------|----------| | 512 | 128 | 1 | 640 | 0.528 | 969.13 | 2.036 | 62.87 | 2.564 | 249.59 | | 1024 | 128 | 1 | 1152 | 1.079 | 948.84 | 3.201 | 39.99 | 4.280 | 269.15 | | 2048 | 128 | 1 | 2176 | 3.390 | 604.10 | 2.952 | 43.36 | 6.342 | 343.11 | | 4096 | 128 | 1 | 4224 | 6.756 | 606.28 | 2.991 | 42.79 | 9.747 | 433.35 | | 8192 | 128 | 1 | 8320 | 13.647 | 600.30 | 3.061 | 41.81 | 16.708 | 497.97 | | 16384 | 128 | 1 | 16512 | 29.491 | 555.56 | 3.414 | 37.50 | 32.905 | 501.81 | | 32768 | 128 | 1 | 32896 | 65.867 | 497.49 | 3.663 | 34.95 | 69.530 | 473.12 | Dang I saw some lowish numbers there for Spaks (and Strix). As I was eyeing a spark to get some CUDA exposure... :-O
• UncleOxidant 6mo ago
  I've been waiting for this one. I've been using 3.5-27b with pretty good success for coding in C,C++ and Verilog. It's definitely helped in the light of less Claude availability on the Pro plan now. If their benchmarks are right then the improvement over 3.5 should mean I'm going to be using Claude even less.
• sietsietnoac 6mo ago
  Generate an SVG of a pelican riding a bicycle: https://codepen.io/chdskndyq11546/pen/yyaWGJx https://codepen.io/chdskndyq11546/pen/yyaWGJx Generate an SVG of a dragon eating a hotdog while driving a car: https://codepen.io/chdskndyq11546/pen/xbENmgK https://codepen.io/chdskndyq11546/pen/xbENmgK Far from perfect, but it really shows how powerful these models can get
  • tln 6mo ago
    The dragon image has issues like one eye, weird tail etc, but the pelican is imo perfect -- the best I've seen!
    • vunderba 6mo ago
      Yeah the dragon one is just a complete mess. The car is sideways but the WHEEL is oriented in a first-person perspective. Seems like a case of overfitting with regard to the thousands of pelican bike SVG samples on the internet already.
  • yrds96 6mo ago
    I wonder if this became a so well known "benchmark" that models already got trained for it.
    • Marciplan 6mo ago
      every model release Simon comes with his Pelican and then this comment follows. Can we stop both? its so boring
      • refulgentis 6mo ago
        I really appreciate you speaking up. Happened yesterday on GPT Image 2, bit my tongue b/c people would see it as fun policing, and same thing today. And it happens on every. single. LLM. release. thread. It's disruptive to the commons, doesn't add anything to knowledge of a model at this point, and it's way out of hand when people are not only engaging with the original and creating screenfuls to wade through before on-topic content, but now people are creating the thread before it exists to pattern-match on the engagement they see for the real thing. So now we have 2x.
• sowbug 6mo ago
  [dead]
• Mr_Eri_Atlov 6mo ago
  Excited to try this, the Qwen 3.6 MoE they just released a week or so back had a noticeable performance bump from 3.5 in a rather short period of time. For anyone invested in running LLMs at home or on a much more modest budget rig for corporate purposes, Gemma 4 and Qwen 3.6 are some of the most promising models available.
• vibe42 6mo ago
  Q4-Q5 quants of this model runs well on gaming laptops with 24GB VRAM and 64GB RAM. Can get one of those for around $3,500. Interesting pros/cons vs the new Macbook Pros depending on your prefs. And Linux runs better than ever on such machines.
  • doix 6mo ago
    What laptop has that much VRAM and RAM for $3500 with good/okay-ish Linux support? I was looking to upgrade my asus zephyrus g14 from 2021 and things were looking very expensive. Decided to just keep it chugging along for another year. Then again, I was looking in the UK, maybe prices are extra inflated there.
    • green7ea 6mo ago
      I got a HP g1a for about 3k€ with 64gb of ram when it came out
  • kroaton 6mo ago
    A3B-35B is better suited for laptops with enough VRAM/RAM. This dense model however will be bandwidth limited on most cards. The 5090RTX mobile sits at 896GB/s, as opposed to the 1.8TB/s of the 5090 desktop and most mobile chips have way smaller bandwith than that, so speeds won't be incredible across the board like with Desktop computers.
    • jadbox 6mo ago
      I find A3B-35B as an ideal model for small local projects- definitely the best for me so far
• techpulselab 6mo ago
  [dead]
• jameson 6mo ago
  What competitive advantage does OpenAI/Anthropic has when companies like Qwen/Minimax/etc are open sourcing models that shows similar (yet below than OpenAI/Anthropic) benchmark results? Also, the token prices of these open source models are at a fraction of Anthropic's Opus 4.6[1] [1]: https://artificialanalysis.ai/models/#pricing https://artificialanalysis.ai/models/#pricing
  • Aurornis 6mo ago
    I use Opus and the Qwen models. The gap between them is much larger than the benchmark charts show. If you want to compare to a hosted model, look toward the GLM hosted model. It’s closest to the big players right now. They were selling it at very low prices but have started raising the price recently.
    • syntaxing 6mo ago
      Yes and no. Are you using open router or local? Are the models are good as Opus? No. But 99% of the time, local models are terrible because of user errors. Especially true for MoE, even though the perplexity only drops minimal for Q4 and q4_0 for the KV cache, the models get noticeably worse.
      • acidtechno303 6mo ago
        Sounds like you're accusing a professional of holding their tool incorrectly. Not impossible, but not likely either.
        • syntaxing 6mo ago
          Inferencing is straight up hard. I’m not accusing them of anything. There’s a crap ton of variables that can go into running a local model. No one runs them at native FP8/FP16 because we cannot afford to. Sometimes llama cpp implementation has a bug (happens all the time). Sometimes the template is wrong. Sometimes the user forgot to expand the context length to above the 4096 default. Sometimes they use quantization that nerfs the model. You get the point. The biggest downside of local LLMs is that it’s hard to get right. It’s such a big problem, Kimi just rolled out a new tool so vendors can be qualified. Even on openrouter, one vendor can be half the “performance” of the other.
• syntaxing 6mo ago
  Been using Qwen 3.6 35B and Gemma 4 26B on my M4 MBP, and while it’s no Opus, it does 95% of what I need which is already crazy since everything runs fully local.
  • throwaw12 6mo ago
    can you expand more on what you mean by 95%? There are 2 aspects I am interested in: 1. accuracy - is it 95% accuracy of Opus in terms of output quality (4.5 or 4.6)? 2. capability-wise - 95% accuracy when calling your tools and perform agentic work compared to Opus - e.g. trip planning?
    • syntaxing 6mo ago
      1. What do you mean by accuracy? Like the facts and information? If so, I use a Wikipedia/kiwx MCP server. Or do you mean tool call accuracy? 2. 3.6 is noticeably better than 3.5 for agentic uses (I have yet to use the dense model). The downside is that there’s so little personality, you’ll find more entertainment talking to a wall. Anything for creative use like writing or talking, I use Gemma 4. I also use Gemma 4 as a “chat” bot only, no agents. One amazing thing about the Gemma models is the vision capabilities. I was able to pipe in some handwritten notes and it converted into markdown flawlessly. But my handwriting is much better than the typical engineer’s chicken scratch.
      • throwaw12 6mo ago
        by accuracy I meant how close is the output to your expectations, for example if you ask 8B model to write C compiler in C, it outputs theory of how to write compiler and writes pseudocode in Python. Which is off by 2 measures: (1) I haven't asked for theory (2) I haven't asked to write it in Python. Or if you want to put it differently, if your prompt is super clear about the actions you want it to do, is it following it exactly as you said or going off the rails occasionally
        • syntaxing 6mo ago
• simonw 6mo ago
  The pelican is excellent for a 16.8GB quantized local model: https://simonwillison.net/2026/Apr/22/qwen36-27b/ https://simonwillison.net/2026/Apr/22/qwen36-27b/ I ran it on an M5 Pro with 128GB of RAM, but it only needs ~20GB of that. I expect it will run OK on a 32GB machine. Performance numbers: Reading: 20 tokens, 0.4s, 54.32 tokens/s Generation: 4,444 tokens, 2min 53s, 25.57 tokens/s I like it better than the pelican I got from Opus 4.7 the other day: https://simonwillison.net/2026/Apr/16/qwen-beats-opus/ https://simonwillison.net/2026/Apr/16/qwen-beats-opus/
  • throwaw12 6mo ago
    I feel like this time it is indeed in the training set, because it is too good to be true. Can you run your other tests and see the difference?
    • simonw 6mo ago
      It went pretty wild with "Generate an SVG of a NORTH VIRGINIA OPOSSUM ON AN E-SCOOTER": https://gist.github.com/simonw/95735fe5e76e6fdf1753e6dcce360699?permalink_comment_id=6113828#gistcomment-6113828 https://gist.github.com/simonw/95735fe5e76e6fdf1753e6dcce360...
      • throwaw12 6mo ago
        compared to your test with GLM 5.1, this indeed looks off https://xcancel.com/simonw/status/2041646779553476801 https://xcancel.com/simonw/status/2041646779553476801
        • refulgentis 6mo ago
          Hoping this doesn't turn into a pelican-SVG back-and-forth: yesterday's GPT Image 2 thread ended up being three screenfuls of "I tried the prompt too" replies, and nothing on the model until you scroll past it. I appreciate the testing, and I know this sounds like fun police, but there's a pattern where well-known commenter + one-off vibe test + 1:1 sub-threads eats the whole discussion. It being fun makes it hard to push back on without looking picky.
• butz 6mo ago
  Are there any "optimized" models, that have lesser hardware requirements and are specialised in single programming language, e.g. C# ?
  • zargon 6mo ago
    LLMs need diverse and extensive training data to be good at a specific thing. We don't (yet?) know how to train a small model that is really good at one programming language. Just big models that are good at a variety of languages (plus lots of other things).
  • Abby_101 6mo ago
    Sort of - there's Qwen3-Coder and the Codestral family, but those are still multi-language, just code-focused. For truly single-language specialization, the practical path is fine-tuning an existing base model on a narrow distribution rather than training from scratch. The issue with C# specifically is dataset availability. Open source C# code on GitHub is a fraction of Python/JS, and Microsoft hasn't released a public corpus the way Meta has for their code models. You'd probably get further fine-tuning Qwen3-Coder (or a similar base) on your specific codebase with LoRA than waiting for a dedicated C#-only model to appear.
    • manquer 6mo ago
      Issues with C# not withstanding. It is not inherently bad idea for small models to trained on only specific languages like a JS/PY only model with declarative languages like HTML, CSS YAML, JSON, graph etc thrown in, probably could be more efficient for local use. Fine-tuning / LoRA on basis the org code base would be make it more useful.
• mark_l_watson 6mo ago
  I have been running the slightly larger 31B model for local coding: ollama launch claude --model qwen3.6:35b-a3b-nvfp4 This has been optimized for Apple Silicon and runs well on a 32G ram system. Local models are getting better!
  • yougotwill 6mo ago
    Can I ask how much RAM of the 32GB does it use? For example can I run a browser and VS Code at the same time?
    • mswphd 6mo ago
      1. the 35B model is a "Mixture of Experts" model. So the earlier commenter's point that it is "larger" does not mean it is more capable. Those types of models only have certain parts of themselves active (for 35b-A3b, it's only 3 billion parameters at a time, vs 27 billion for the model this post is about) at a time to speed up inference. So if you're interested in these things for the first time, Qwen3.6-35B-A3B is a good choice, but it is likely not as capable as the model this thread is about. 2. its hard to cite precise numbers because it depends heavily on configuration choices. For example 2a. on a macbook with 32GB unified memory you'll be fine. I can load a 4 bit quant of Qwen3.6-35B-A3B supporting max context length using ~20GB RAM. 2b. that 20GB ram would not fit on many consumer graphics cards. There are still things you can do ("expert offloading"). On my 3080, I can run that same model, at the same quant, and essentially the same context length. This is despite the 3080 only having ~10GB VRAM, by splitting some of the work with the CPU (roughly). Layer offloading will cause things to slow down compared to keeping layers fully resident in memory. It can still be fast though. Iirc I've measured my 3080 as having ~55 tok/s, while my M4 pro 48GB has maybe ~70 tok/s? So a slowdown but still usable. If you want to get your feet wet with this, I'd suggest trying out * Lmstudio, and * the zed.dev editor they're both pretty straightforward to setup/pretty respectable. zed.dev gives you very easy configuration to get something akin to claude code (e.g. an agent with tool calling support) in relatively little time. There are many more fancy things you can do, but that pair is along the lines of "setup in ~5 minutes", at least after downloading the applications + model weights (which are likely larger than the applications). This is assuming you're on mac. The same stack still works with nvidia, but requires more finnicky setup to tune the amount of expert offloading to the particular system. It's plausible you could do something similar with LMstudio + vscode, I'm just less familiar with that.
• LowLevelKernel 6mo ago
  How much VRAM is needed?
  • awestroke 6mo ago
    27 multiplied by quant, add context
• jedisct1 6mo ago
  I really like local models for code reviews / security audits. Even if they don't run super fast, I can let them work overnight and get comprehensive reports in the morning. I used Qwen3.6-27B on an M5 (oq8, using omlx) and Swival (https://swival.dev https://swival.dev) /audit command on small code bases I use for benchmarking models for security audits. It found 8 out of 10, which is excellent for a local model, produced valid patches, and didn't report any false positives. which is even better.
• 2001zhaozhao 6mo ago
  I'm kind of interested in a setup where one buys local hardware specifically to run a crap ton of small-to-medium LLM locally 24/7 at high throughput. These models might now be smart enough to make all kinds of autonomous agent workflows viable at a cheap price, with a good queue prioritization system for queries to fully utilize the hardware.
  • kroaton 6mo ago
    Buy any Strix Halo box and have fun with your 128GB of VRAM.
    • 2001zhaozhao 6mo ago
      I wonder whether it is much more cost-effective in terms of token throughput / hardware+power cost to get actual GPUs instead, given that the model size is only 27B.
  • OliverGuy 6mo ago
    I would love to have a shit load of small (27B dense. 35B MoE) agents running locally and looking at and ingesting every bit of data about me, my life and what I get up to see what sort of correlations it finds. Give a coding agent access to a data lake of events and let it build up its own analytics tooling to extract and draw out information from that data, and present it to me as daily/weekly/monthly summaries.
    • Havoc 6mo ago
      That's definitely doable. Planning similar except more webscraping / newsfeed / monitoring like. I've got 3x SBCs that can run the Gemma 4 26B MoE on NPU. Around 4W extra power, 3 tokens a second...so that can hammer away at tasks 24/7 without moving the needle on electricity bill
      • 2001zhaozhao 6mo ago
        I wonder if some investment firms are already doing this internally at a large scale. (Probably.)
• xrd 6mo ago
  I'm experimenting with this on my RTX 3090 and opencode. It is pretty impressive so far.
• blurbleblurble 6mo ago
  It's a rap on claude
• docheinestages 6mo ago
  Has anyone tried using this with a Claude Code or Qwen Code? They both require very large context windows (32k and 16k respectively), which on a Mac M4 48GB serving the model via LM Studio is painfully slow.
  • domh 6mo ago
    I had the best success yet earlier today running https://pi.dev https://pi.dev with a local gemma4 model on ollama on my m4 Mac with 48GB ram. I think pi is a lot lighter than Claude code.
    • qudat 6mo ago
      I didn’t think pi supported local models?
      • segmondy 6mo ago
        pi does, it can talk to any OpenAI API
        • qudat 6mo ago
          how do you provide the base url?
      • domh 6mo ago
        It does! Ollama provides a helper to launch it with the local model too: https://docs.ollama.com/integrations/pi https://docs.ollama.com/integrations/pi So you can do: ollama launch pi --model gemma4:26b And it launches and points to the local model in one command. pi seems to do some setting caching too, because after doing the above once I can just do `pi` and it's already setup to the local model.
  • swalsh 6mo ago
    Try running with Open Code. It works quite well.
• objektif 6mo ago
  Does anyone know good provider for low latency llm api provider? We tried to look at Cerebras and Groq but they have 0 capacity right now. GPT models are too slow for us at the moment. Gemini are better but not really at same level as GPT.
  • spmurrayzzz 6mo ago
    This depends a bit on your cost sensitivity and what model families you want support for, but Baseten and Fireworks have been my goto. Currently Baseten has ~610ms TTFT and ~82 tk/s for Kimi K2.6, which is roughly 2x the throughput of GPT-5.4 (per their openrouter stats). GLM 5 is slightly slower on both metrics, but still strong.
• lgessler 6mo ago
  I'll be really interested to hear qualitative reports of how this model works out in practice. I just can't believe that a model this small is actually as good as Opus, which is rumored to be about two orders of magnitude larger.
• htrp 6mo ago
  Any comparisons against Qwen3.6-35B-A3B?
• zkmon 6mo ago
  On llama server, the Q4_K_M is giving about 91k context on 24GB, which calculates to about 70MB per 1K context (KV-Cache). I could have gone for Q5 which probably leaves about 30K token space. I think this is pretty impressive.
  • sleepyeldrazi 6mo ago
    I have been getting good results with IQ4_NL and TurboQuant at 4bits on 24gb (3090). It easily fits 256k with that setup, but it starts slowing down quite a bit after 80-100k. Quality in my testing is also still good: - Coding task test: https://github.com/sleepyeldrazi/llm_programming_tests/ https://github.com/sleepyeldrazi/llm_programming_tests/ - Design task test: https://github.com/sleepyeldrazi/llm-design-showcase https://github.com/sleepyeldrazi/llm-design-showcase Coding was against minimax-m2.7 and glm-5, and the design against other small models
• navbaker 6mo ago
  TIL that our corporate network site blocker classifies qwen.ai as a sex site…
  • mistrial9 6mo ago
    .. that is what they tell YOU
• richstokes 6mo ago
  Are there benchmarks of this / what’s the best way to compare it against paid models? With all the rate limiting in Claude/Copilot/etc, running locally is more and more appealing.
• vocoda 6mo ago
  I wonder why they did not compare it to Qwen Coder Next?
  • segmondy 6mo ago
    It should crush Qwen3CoderNext especially since it beats 397B
• thot_experiment 6mo ago
  no FIM though :(, imo most slept on usecase for local models
  • harkh 6mo ago
    I agree, what would you say are currently the best local options?
    • zargon 6mo ago
      For FIM, there's Qwen3 Coder Next. Although Mistral's model card seems to indicate that Devstral 2 doesn't support FIM, it seems very odd that it wouldn't. I have been meaning to test it.
      • thot_experiment 6mo ago
        Qwen Coder 30B A3B is far better than Qwen Coder Next imo. I may have inference issues or it's just a problem with running Coder Next at IQ4 XS, vs Q8 for the earlier/smaller model but I don't find the 80B to be much better at coding, even in instruct mode, and the insane speed and low latency of the smaller model is way more useful. Good one-line completions often happen in 300ms.
  • deleted 6mo ago
    [deleted]
• finnjohnsen2 6mo ago
  Since Gemma 4 came this easter the gap from self hosting models to Claude has decreased sigificantly I think. The gap is still huge it just that local models were extremely non-competitive before easter. So now it seems Qwen 3.6 is another bump up from Gemma 4 which is exciting if it is so. I keep an Opus close ofcourse, because these local models still wander off in the wrong direction and fails. Something Opus almost never does for me anymore. But every time a local model gets me by - I feel closer to where I should be; writing code should still be free. Both free as in free beer, and free as in freedom. My setup is a seperate dedicated Ubuntu machine with RTX 5090. Qwen 3.6:27b uses 29/32gb of vram when its working right this minute. I use Ollama in a non root podman instance. And I use OpenCode as ACP Service for my editor, which I highly recommend. ACP (Agent Client Protocol) is how the world should be in case you were asking, which you didnt :) Exciting times and thank you Qwen team for making the world a better place in a world of Sam Altmans.
  • datadrivenangel 6mo ago
    Gemma4 feels the most "claude-like" of all the models I've run locally on my M5 mbp.
    • chr15m 6mo ago
      I found on coding tasks that Qwen 3.5 can actually do the thing whereas Gemma 4 went off the rails frequently. Will try this new 3.6 release today.
      • verdverm 6mo ago
        There was a qwen-3.6 MoE six days ago that I thought was better than Gemma 4. Today's is a dense model. (gemma release both a 26B MoE and a 31B dense at the same time) I have intention to evaluate all four on some evals I have, as long as I don't get squirrelled again.
      • da-x 6mo ago
        I use Qwen 3.5 122B on an RTX PRO 6000 with open code, and very pleased. I don't feel a need for using a closed model any more. The result after answering questions in Plan mode is almost always what I want, with very few occasional bugs. It does a lot of effort to see how the code I am working on is written now while extending it in the same style. If they release a Qwen 3.6 that also makes good use of the card, may move to it.
• mft_ 6mo ago
  Huh, running the Q4_K_M quant with LM Studio, and asked it "How can I set up Qwen 3.6 27b to use tools and access the local file system?". Part of its reply was: Quick clarification: As of early 2025, "Qwen 3.6" hasn't been released yet. You are likely looking for Qwen2.5, specifically the Qwen2.5-32B-Instruct model, which is the 30B-class model closest to your 27B reference. The instructions below will use this model. Weird.
  • Jowsey 6mo ago
    This is pretty standard in every model. Ask Opus or Gemini about 2026 (without a big system prompt to steer them) and they'll swear blind it's 2024/25 too.
  • mudkipdev 6mo ago
    If you are talking with Claude about AI, it will sometimes passively bring up "frontier models like GPT-4o"
    • user_7832 6mo ago
      Slightly tangential, how good/bad is 4o compared to the modern (5.3 I think?) one? TBH I personally find non-thinking replies quite poor for the type of questions I ask so I haven't touched chatgpt for months (ever since Gemini 2.5 Pro I think.) (And even Gemini 3.1 Pro tends to still be too literal at times instead of understand the implied meaning lol. We've got more place to improve.)
  • petu 6mo ago
    Models are math functions that predict next word, not conscious beings. If it was trained on dataset including data up to Q1 2025, then that's more or less expected answer -- even Qwen 3 didn't exist. If you see model that can reliably answer questions about itself (version, family, capabilities, etc), then it's most likely part of system prompt. In absence of system prompt even Claude could say it's a model created by DeepSeek: https://x.com/stevibe/status/2026227392076018101 https://x.com/stevibe/status/2026227392076018101
• storus 6mo ago
  If this runs at Opus 4.5 level for agentic coding then I don't really need any cloud models anymore.
  • senko 6mo ago
    I tested it with my standard test - HTML/JS minesweeper. From the user POV, the result is great and subjectively equivalent to Opus 4.5. I also asked Claude Code (Opus 4.7) and Codex (GPT-5.4) to review both qwen's output and that of opus 4.5, and both agents concluded qwen's was better. Minesweeper is simple but nontrivial - 600-800 lines of code that need to be internally consistent. At that complexity level, this model is definitely a viable alternative. (haven't tested with planning, debugging and more complex problems yet)
  • qudat 6mo ago
    I just want sonnet 4.5 but I also want it to be fast which is trickier
• datadrivenangel 6mo ago
  So far I'm unimpressed for local inference. got 11 tokens per second on omlx on an M5 Pro with 128gb of ram, so it took an hour to write a few hundred lines of code that didn't work. Opus and Sonnet in CC the same task successfully in a matter of minutes. The 3.6:35b model seemed okay on ollama yesterday. Need to check out other harnesses for this besides claude code, but the local models are just painfully slow.
  • noman-land 6mo ago
    OpenCode seems to be a lot better than Claude at using local models.
  • someguydave 6mo ago
    I got about 7 tokens/sec generation on an M2 max macbook running 8-bit quant on an MLX version.
  • mswphd 6mo ago
    this is a dense model, so that's expected. On a mac you'd want to try out the Mixture of Experts Qwen3.6 release, namely Qwen3.6-35B-A3B. On an M4 Pro I get ~70 tok/s with it. If your numbers are slower than this, it might be because you're accidentally using a "GGUF" formatted model, vs "MLX" (an apple-specific format that is often more performant for macs).
  • hresvelgr 6mo ago
    You have better specs than I do and I'm running the same model almost twice as fast through GGUF on llama cpp. I'd try some different harnesses.
  • fshen 6mo ago
    I use the same computer as you do. m5 can run faster: pip install mlx_lm python -m mlx_vlm.convert --hf-path Qwen/Qwen3.6-27B --mlx-path ~/.mlx/models/Qwen3.6-27B-mxfp4 --quantize --q-mode mxfp4 --trust-remote-code mlx_lm.generate --model ~/.mlx/models/Qwen3.6-27B-mxfp4 -p 'how cpu works' --max-tokens 300 Prompt: 13 tokens, 51.448 tokens-per-sec Generation: 300 tokens, 35.469 tokens-per-sec Peak memory: 14.531 GB
  • 6mo ago
• n8henrie 6mo ago
  I'm still fairly new to local LLMs, spent some time setting up and testing a few Qwen3.6-35B-A3B models yesterday (mlx 4b and 8b, gguf Q4_K_M and Q4_K_XL I think). Was impressed at how they ran on my 64G M4. It looks like this new model is slightly "smarter" (based on the tables in TFA) but requires more VRAM. Is that it? The "dense" part being the big deal? As 27B < 35B, should we expect some quantized models soon that will bring the VRAM requirement down?
  • mswphd 6mo ago
    that's not it. 35B-A3B is a "Mixture of Experts" model. Roughly, only ~3B parameters are active at a time. So, the actual computational requirements scale with this ~3B, rather than with the 35B (though you need high-bandwidth access to the full 35B layers though). This model is a "dense" model. It will be much slower on macs. Concretely, on a M4 Pro, at Q6 gguf, it was ~9tok/s for me. 35-A3B (at Q4, with mlx, so not a fair comparison) was ~70 tok/s by comparison. In general dedicated GPUs tend to do better with these kinds of "dense" models, though this becomes harder to judge when the GPU does not have enough VRAM to keep the model fully resident. For this model, I would expect if you have >=24GB VRAM you'd be fine, e.g. an NVIDIA {3,4,5}090-type thing.
• reddit_clone 6mo ago
  What can I run on a M4 Pro with 48 GB or RAM?
  • bigyabai 6mo ago
    A sparser model like Qwen3.6 35B A3B is probably your best choice: https://qwen.ai/blog?id=qwen3.6-35b-a3b https://qwen.ai/blog?id=qwen3.6-35b-a3b
    • hnfong 6mo ago
      The 35B MOE will run faster, but 48GB RAM is more than enough to run the 27B dense model as well. It's just that token/s will be on the lower side.
• SleepyQuant 6mo ago
  [dead]
• RandyOrion 6mo ago
  Thank you Qwen team. Small DENSE LLMs shapes the future of local LLM users. When Qwen 3.5 27b released, I didn't really understand why linear attention is used instead of full attention because of the performance degradation and problems introduced with extra (linear) operators. After doing some tests, I found that with llama.cpp and IQ4_XS quant, the model and BF16 cache of the whole 262k context just fit on 32GB vram, which is impossible with full attention. In contrast, with gemma 4 31b IQ4_XS quant I have to use Q8_0 cache to fit 262k context on the vram, which is a little annoying (no offenses, thank you gemma team, too). From benchmarks, 3.5->3.6 upgrade is about agent things. I hope future upgrades fix some problems I found, e.g., output repetitiveness in long conversations and knowledge broadness.
• deleted 6mo ago
  [deleted]
• pja 6mo ago
  I know this is kind of old hat by now, but it kind of blows my mind that I can upload a hand drawn decision tree & get a transcribed dot file back on consumer hardware using a pile of linear algebra that wasn’t even particularly specialised for this purpose, it’s just a capability that it picked up along with everything else during training.
  • amunozo 6mo ago
    I totally agree. The feeling you get by running these things locally is different, as if you could feel the magic closer.
  • alansaber 6mo ago
    Well this is the magic of LLMs, they learn things incidentally well, and specialised models are pretty average.
  • wongarsu 6mo ago
    If you had shown this to someone in 2018 they wouldn't have hesitated to call it an AGI. We truly reached the state where we have one model that performs at usable levels across a huge range of tasks. You don't need to assemble a training set of hand drawn diagrams and corresponding dot files and train some kind of CNN on that, you just throw the task at a preexisting LLM and get a usable result. We always talk about the negatives (in most tasks it's worse than a human domain expert, the results are soulless, the societal implications are scary), but this kind of generality really is a monumental achievement
• yamajun93 6mo ago
  The benchmarks looks great for a 27B model, but Im curious how it will perform locally. I have tried a bunch of open-source models, and still I feel we are far from getting a similar output to claude code running in my M3 in my macbook pro.
• Avlin67 6mo ago
  130 token per seconds on dual RTX 5090 for FP8 version
• synergy20 6mo ago
  wish I had a rtx5090 instead of the rtx4090 so I can run this, need 32GB VRAM I assume, no I use Linux and I do not enjoy MacOS, I wish x86 family has something like M5 Pro though.
  • fwipsy 6mo ago
    I was just wishing I had 24gb rather than 16gb so I could run q4 comfortably. Unsloth q4 variants are 15.4gb-17.6gb so it seems at least well worth trying on a 4090.
• j_gonzalez 6mo ago
  [dead]
• zhouquanxi 6mo ago
  [dead]
• fwystup 6mo ago
  [dead]
• kk_mors 6mo ago
  [flagged]