5 ms·
Supposedly this is a Kimi k3 rival. Bit of a chonker, especially since they only released bf16 and fp8. So at launch this will be harder to serve than k3. No QA
by NitpickLawyer 2mo ago
Supposedly this is a Kimi k3 rival. Bit of a chonker, especially since they only released bf16 and fp8. So at launch this will be harder to serve than k3. No QAT on q4 means that someone with deep pockets (nvda?) will have to quant it, with plenty of calibration data. Should bring it ~1.3TB, so around k3 size.
License pretty similar to k3 with some caveats. Free to use for internal or <50M$ revenue / year. Limitations above that threshold for serving the model or services targeting coding / productivity agents.
Benchmarks are looking good, trading blows w/ opus4.8 and sol, generally 10-20p under fable. But that's neither here nor there w/ qwen, their benchmark to real world usage correlation has been iffy in the past.
The local model 3.8-27B announced for Friday, same time so ~48 hours from now. That'll be a bit more exciting for a lot more people, since 3.6 was quite good for local inference, and their 3.7-max -> 3.8-max shows a lot of improvement.
- MrDrMcCoy 2mo agoLlama.cpp can quantize without special training, but I'm not sure if any special model architecture support is needed to read it in the first place. If it can be converted to gguf at all and you know what tensors to target, it can get the full ternary bonsai treatment today.
- binary132 2mo agoQAT is an optimizing quantization algorithm, not naive quant.
- MrDrMcCoy 2mo agoRight, but the way they phrased it suggested that without QAT it could not be quanted at all.
- dragonwriter 2mo agoIsn’t QAT a training approach (roughly, simulating quantization in the forward pass during training so that quantization of the level targeted in training has close-to-optimal behavior), not a quantization algorithm? Hence, the name?
- binary132 2mo agosounds like a repeatable method of optimizing quantization to me, friend
- NitpickLawyer 2mo agoSure, but that's for "personal" serving. I meant for 3rd party providers. Usually we get a good indication on what it costs to host this, as the prices settle on open router. That's why I said it's tougher to serve than kimi k3 on launch. As a provider you'd do fp8 if the model creator didn't do QAT on q4, or until someone does a good calibrated nvfp4. And that's usually nvda :)
- MrDrMcCoy 2mo agoThat makes sense, but your specific phrasing precluded the possibility of non-QAT quantization.
- NitpickLawyer 2mo agoShould have worded that better, my bad.
- esafak 2mo agoNow that they have reached the frontier in raw performance, I would like to see Chinese models improve their reasoning efficiency.
- jingpostmedia 2mo ago[flagged]
- verdverm 2mo agoFor all the talk about over reasoning, K3 on low thinking has been rather nice
- esafak 2mo agoIt looks like a work horse! Is it yours?
- JSR_FDED 2mo agoI bet they could do it faster if they weren’t blocked from buying GPUs
- ZeroCool2u 2mo agoUnsloth already has a guide for their quants: https://unsloth.ai/docs/models/qwen3.8 https://unsloth.ai/docs/models/qwen3.8
- codedokode 2mo agoI wonder who is unsloth and where they got time, hardware and knowledge to quantize them?
- NitpickLawyer 2mo agounshloth started as a finetuning library with lots of optimisations so you could finetune on lower end hardware. Kind of OGs of the local community. Started by two brothers Michael and Daniel(?) a math wiz and a community builder/communicator. They've since gotten some VC backing, are active in quantising lots of models on release day (work w/ labs to prepare things), known for their optimised quants (use different bits for different layers). Recently I saw they launched some sort of a desktop app, like lmstudio if you're familiar with it. They're really cool people and known in the local model places.
- ZeroCool2u 2mo agoIt's our guy Daniel: https://www.linkedin.com/in/danielhanchen https://www.linkedin.com/in/danielhanchen
- danielhanchen 2mo agoHey :)
- Eisenstein 2mo agoThey started with offering training methods for quantized models to save memory and added new things over time. They are very active in the local model community and have extensive documentation and tooling to help with running and training models locally.
- 2mo ago
- verdverm 2mo agoquanting is actually cheap and you can compress a model that does not fit on a GPU. You can process layer by layer, this is what the sequential processor in llm-compressor does.
- alanwreath 2mo ago> 3.8-27B announced for Friday Maybe I’m misreading this or some other post, I thought QWEN was stepping away from releasing these models for local consumption
- trollbridge 2mo agoThey reversed course and now are saying they'll be releasing their Max style models in open weights.
- arthurcolle 2mo agothank you china!
- FooBarWidget 2mo agoRather, thank you competition in China. This is Chinese "overcapacity" (of talent pool) at work.
- embedding-shape 2mo agoQuite literally the government in China came out and said "It'd be better for us if we did more open models and collaborated with other countries also doing open models" and then Qwen changed their tune. It's literally thanks to China in this case, not competitors/peers in China. Their government is horrible for a lot of stuff, but in this case they do deserve praise for forcing the "right" (according to me) direction.
- suprjami 2mo agoOur desire for better local models just happens to coincide with China's desire to destroy the western AI company business model by releasing local models. I doubt there's any philanthropy involved.
- 2mo ago
- zepearl 2mo agoThe parameter "reasoning_effort" is something new, or am I wrong? Qwen3.8 comes with official support for reasoning_effort, which can be used to adjust reasoning depth and control cost: - xhigh (default): for complex tasks demanding thorough analysis - medium: balancing accuracy and speed - low: efficient reasoning optimizing for speed and cost In addition, preserve_thinking is enabled by default for all workloads for the best out-of-the-box experience. Asking because in my case (OCR of scanned historical "National Geographic" magazines) the LLM trying to merge text split into separate columns was running in circles from time to time and needed a lot of prompt tuning when using Qwen 3.0/3.5/3.6 (still needs from time to time).
- philipkglass 2mo agoI'm using Qwen 3.5 for OCR, and reasoning_effort is supported there too. I found that it can be loop-prone (though somewhat less so) even if you set reasoning_effort to low.
- badcafe23423435 2mo agoShow me any open source model quantisite, distile or reduce size from nvidia ;)
- tinco 2mo agoFrom my evaluation[1] neither Kimi K3 nor Qwen 3.8 are as good as GLM 5.2 at coding. I wonder if there's a marketing gap that's got people underestimate it. It uses twice as much tokens to achieve the same but the results are significantly better and because it's so much cheaper it's the most economical choice too. [1] https://blog.bosun.ai/software-maintenance-with-open-weight-models-part-1/ https://blog.bosun.ai/software-maintenance-with-open-weight-...