6 ms·
Already quantized/converted into a sane format by Unsloth: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF https://huggingface.co/unsloth/Qwen3.6-35B-A3B-G
by homebrewer 6mo ago
Already quantized/converted into a sane format by Unsloth:
https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF
- txtsd 6mo agoSo I can use this in claude code with `ollama run claude`?
- pj_mukh 6mo agohave you found a model that does this with usable speeds on an M2/M3?
- postalcoder 6mo agoOn a M4 MBP ollama's qwen3.5:35b-a3b-coding-nvfp4 runs incredibly fast when in the claude/codex harness. M2/M3 should be similar. It's incomparably faster than any other model (i.e. it's actually usable without cope). Caching makes a huge difference.
- Ladioss 6mo agoMore like `ollama launch claude --model qwen3.6:latest` Also you need to check your context size, Ollama default to 4K if <24 Gb of VRAM and you need 64K minimum if you want claude to be able to at least lift a finger.
- txtsd 6mo agoI only have 16GB VRAM, and my system uses ~4GB from that. What are my options? I got this one: `Qwen3.6-35B-A3B-UD-IQ2_XXS.gguf`
- Ladioss 6mo agoMy system has 16 Gb VRAM / 32 Gb RAM, and ollama runs qwen3.6:latest at decent speed just fine. The 35b model is a moe, so I guess the whole model is offloaded.
- Patrick_Devine 6mo agoIf you're on a Mac, use the MLX backend versions which are considerably faster than the GGML based versions (including llama.cpp) and you don't need to fiddle with the context size. The models are `qwen3.6:35b-a3b-nvfp4`, `qwen3.6:35b-a3b-mxfp8`, and `qwen3.6:35b-a3b-mlx-bf16`.
- egorfine 6mo agoI was comparing various models at M5 Pro 48GB RAM MLX vs GGUF and found that MLX models have a higher time to first token (sometimes by an order of magnitude) while tokens/sec and memory usage is same as GGUF. Gemma 3 27B q4: * MLX: 16.7 t/s, 1220ms ttft * GGUF: 16.4 t/s, 760ms ttft Gemma 4 31B q8: * MLX: 8.3 t/s, 25000ms ttft * GGUF: 8.4 t/s, 1140ms ttft Gemma 4 A4B q8: * MLX: 52 t/s, 1790ms ttft * GGUF: 51 t/s, 380ms ttft All comparisons done in LM Studio, all versions of everything are the latest.
- nunodonato 6mo agohttps://sleepingrobots.com/dreams/stop-using-ollama/ https://sleepingrobots.com/dreams/stop-using-ollama/
- txtsd 6mo agoThank you, I had no idea ollama was so shady! I will start using llama.cpp directly.
- terataiijo 6mo agolmao they are so fast yooo
- ttul 6mo agoYes. How do they do it? Literally they must have PagerDuty set up to alert the team the second one of the labs releases anything.
- beernet 6mo agoThey obviously collaborate with some of the labs prior to the official release date.
- sigbottle 6mo agoThat... is a more plausible explanation I didn't think of.
- danielhanchen 6mo agoYes we collab with them!
- qskousen 6mo agoSorry this is a bit of a tangent, but I noticed you also released UD quants of ERNIE-Image the same day it released, which as I understand requires generating a bunch of images. I've been working to do something similar with my CLI program ggufy, and was curious of you had any info you could share on the kind of compute you put into that, and if you generate full images or look at latents?
- danielhanchen 6mo agoYes we have started doing diffusion GGUFs but it's in it's infancy :) But yes we do generate images to test quants out!
- palmotea 6mo agoHow much VRAM does it need? I haven't run a local model yet, but I did recently pick up a 16GB GPU, before they were discontinued.
- trvz 6mo agoIf you have to ask then your GPU is too small. With 16 GB you'll be only able to run a very compressed variant with noticable quality loss.
- FusionX 6mo agoAren't 4bits model decent? Since, this is an MOE model, I'm assuming it should have respectable tk/s, similar to previous MOE models.
- palmotea 6mo ago> If you have to ask then your GPU is too small. What's the minimum memory you need to run a decent model? Is it pretty much only doable by people running Macs with unified memory?
- bfivyvysj 6mo agoA bit like asking how long is a piece of string.
- utilize1808 6mo agoObviously going to depend on your definition of "decent". My impression so far is that you will need between 90GB to 100GB of memory to run medium sized (31B dense or ~110B MoE) models with some quantization enabled.
- Aurornis 6mo agoUnsloth is great for uploading quants quickly to experiment with, but everyone should know that they almost always revise their quants after testing. If you download the release day quants with a tool that doesn’t automatically check HF for new versions you should check back again in a week to look for updated versions. Some times the launch day quantizations have major problems which leads to early adopters dismissing useful models. You have to wait for everyone to test and fix bugs before giving a model a real evaluation.
- embedding-shape 6mo agoNot to mention that almost every model release has some (at least) minor issue in the prompt template and/or the runtime itself, so even if they (not talking unsloth specifically, in general) claim "Day 0 support", do pay extra attention to actual quality as it takes a week or two before issues been hammered out.
- danielhanchen 6mo agoYes this is fair - we try our best to communicate issues - I think we're mostly the only ones doing the communication that model A or B has been fixed etc. We try our best as model distributors to fix them on day 0 or 1, but 95% of issues aren't our issues - as you mentioned it's the chat template or runtime etc
- alfiedotwtf 6mo agoI have to ask - what do you run locally on your laptop (model, backend, and agentic cli)? Feature request: A leader board with filtering so you can enter your machine specs and it will sort all models along with all the various quantisation and then rank them all - because so far model ranking site either don’t include all available quants, don’t compare apples to apples (ie was one model tested with Claude code while another benchmark done with opencode) etc Oh - and as bonus, scoring also ranked by which agentic CLI :)
- danielhanchen 6mo ago
- sander1095 6mo agoI sense that I don't really understand enough of your comment to know why this is important. I hope you can explain some things to me: - Why is Qwen's default "quantization" setup "bad" - Who is Unsloth? - Why is his format better? What gains does a better format give? What are the downsides of a bad format? - What is quantization? Granted, I can look up this myself, but I thought I'd ask for the full picture for other readers.
- danielhanchen 6mo agoOh hey - we're actually the 4th largest distributor of OSS AI models in GB downloads - see https://huggingface.co/unsloth https://huggingface.co/unsloth https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs https://unsloth.ai/docs/basics/unsloth-dynamic-2.0-ggufs is what might be helpful. You might have heard 1bit dynamic DeepSeek quants (we did that) - not all layers can be 1bit - important ones are in 8bit or 16bit, and we show it still works well.
- dist-epoch 6mo agoThe default Qwen "quantization" is not "bad", it's "large". Unsloth releases lower-quality versions of the model (Qwen in this case). Think about taking a 95% quality JPEG and converting it to a 40% quality JPEG. Models are quantized to lower quality/size so they can run on cheaper/consumer GPUs.
- danielhanchen 6mo agoLove the JPEG analogy :)
- est 6mo agohey you can do a bit research yourself and tell your results to us!
- torginus 6mo agoWhy doesn't Qwen itself release the quantized model? My impression is that quantization is a highly nontrivial process that can degrade the model in non-obvious ways, thus its best handled by people who actually built the model, otherwise the results might be disappointing. Users of the quantized model might be even made to think that the model sucks because the quantized version does.
- bityard 6mo agoModel developers release open-weight models for all sorts of reasons, but the most common reason is to share their work with the greater AI research community. Sure, they might allow or even encourage personal and commercial use of the model, but they don't necessarily want to be responsible for end-user support. An imperfect analogy might be the Linux kernel. Linus publishes official releases as a tagged source tree but most people who use Linux run a kernel that has been tweaked, built, and packaged by someone else. That said, models often DO come from the factory in multiple quants. Here's the FP8 quant for Qwen3.6 for example: https://huggingface.co/Qwen/Qwen3.6-35B-A3B-FP8 https://huggingface.co/Qwen/Qwen3.6-35B-A3B-FP8 Unsloth and other organizations produce a wider variety of quants than upstream to fit a wider variety of hardware, and so end users can make their own size/quality trade-offs as needed.
- halJordan 6mo agoQuantization is an extraordinarily trivial process. Especially if you're doing it with llama.cpp (which unsloth obviously does). Qwen did release an fp8 version, which is a quantized version.
- halJordan 6mo agoThere's absolutely nothing wrong it insane with a safetensors file. It might be less convenient than a single file gguf. But that's just laziness not insanity
- littlestymaar 6mo ago> converted into a sane format Having implemented a GGUF parser, I'd beg to differ on the “sane format” qualifier.