3 ms·

lmao they are so fast yooo

ttul 6mo ago
Yes. How do they do it? Literally they must have PagerDuty set up to alert the team the second one of the labs releases anything.
- beernet 6mo ago
  They obviously collaborate with some of the labs prior to the official release date.
  - sigbottle 6mo ago
    That... is a more plausible explanation I didn't think of.
    - danielhanchen 6mo ago
      Yes we collab with them!
      - qskousen 6mo ago
        Sorry this is a bit of a tangent, but I noticed you also released UD quants of ERNIE-Image the same day it released, which as I understand requires generating a bunch of images. I've been working to do something similar with my CLI program ggufy, and was curious of you had any info you could share on the kind of compute you put into that, and if you generate full images or look at latents?
        danielhanchen 6mo ago
        Yes we have started doing diffusion GGUFs but it's in it's infancy :) But yes we do generate images to test quants out!
- sigbottle 6mo ago
  Is quantization a mostly solved pipeline at this point? I thought that architectures were varied and weird enough where you can't just click a button, say "go optimize these weights", and go. I mean new models have new code that they want to operate on, right, so you'd have to analyze the code and insert the quantization at the right places, automatically, then make sure that doesn't degrade perf? Maybe I just don't understand how quantization works, but I thought quantization was a very nasty problem involving a lot of plumbing
  - Readerium 6mo ago
    that is true. gguf does not support any Architecture. for the most recent example, as of April 16, 2026 (today) Turboquant isnt still added to GGUF
ekianjo 6mo ago
yeah and often their quants are broken. They had to update their Gemma4 quants like 4 times in the past 2 weeks.
- danielhanchen 6mo ago
  No it's not our fault - re our 4 uploads - the first 3 are due to llama.cpp fixing bugs - this was out of our control (we're llama.cpp contributors, but not the main devs) - we could have waited, but it's best to update when multiple (10-20) bugs are fixed. The 4th is Google themselves improving the chat template for tool calling for Gemma. https://github.com/ggml-org/llama.cpp/issues/21255 https://github.com/ggml-org/llama.cpp/issues/21255 was another issue CUDA 13.2 was broken - this was NVIDIA's CUDA compiler itself breaking - fully out of our hands - but we provided a solution for it.
bildung 6mo ago
Bad QA :/ They had a bunch of broken quantizations in the last releases
- danielhanchen 6mo ago
  1. Gemma-4 we re-uploaded 4 times - 3 times were 10-20 llama.cpp bug fixes - we had to notify people to upload the correct ones. The 4th is an official Gemma chat template improvement from Google themselves. 2. Qwen3.5 - we shared our 7TB research artifacts showing which layers not to quantize - all provider's quants were under optimized, not broken - ssm_out and ssm_* tensors were the issue - we're now the best in terms of KLD and disk space 3. MiniMax 2.7 - we swiftly fixed it due to NaN PPL - we found the issue in all quants regardless of provider - so it affected everyone not just us. We wrote a post on it, and fixed it - others have taken our fix and fixed their quants, whilst some haven't updated. Note we also fixed bugs in many OSS models like Gemma 1, Gemma 3, Llama chat template fixes, Mistral, and many more. Unfortunately sometimes quants break, but we fix them quickly, and 95% of times these are out of our hand. We swiftly and quickly fix them, and write up blogs on what happened. Other providers simply just take our blogs and fixes and re-apply, re-use our fixes.
  - bildung 6mo ago
    Fair enough, appreciate the detailed response! Can you elaborate why other quantizations weren't affected (e.g. bartowski)? Simply because they were straight Q4 etc. for every layer?
    - danielhanchen 6mo ago
      No Bartowski's are more affected - (38% NaN) than ours (22%) - for MiniMax 2.7 see https://www.reddit.com/r/LocalLLaMA/comments/1slk4di/minimax_m27_gguf_investigation_fixes_benchmarks/ https://www.reddit.com/r/LocalLLaMA/comments/1slk4di/minimax... We already fixed ours. Bart hasn't yet but is still working on it following our findings. blk.61.ffn_down_exps in Q4_K or Q5_K failed - it must be in Q6_K otherwise it overflows. For the others, yes layers in some precision don't work. For eg Qwen3.5 ssm_out must be minimum Q4-Q6_K. ssm_alpha and ssm_beta must be Q8_0 or higher. Again Bart and others apply our findings - see https://www.reddit.com/r/LocalLLaMA/comments/1rgel19/new_qwen3535ba3b_unsloth_dynamic_ggufs_benchmarks/ https://www.reddit.com/r/LocalLLaMA/comments/1rgel19/new_qwe...

ttul 6mo ago

Yes. How do they do it? Literally they must have PagerDuty set up to alert the team the second one of the labs releases anything.

beernet 6mo ago
They obviously collaborate with some of the labs prior to the official release date.
- sigbottle 6mo ago
  That... is a more plausible explanation I didn't think of.
  - danielhanchen 6mo ago
    Yes we collab with them!
    - qskousen 6mo ago
      Sorry this is a bit of a tangent, but I noticed you also released UD quants of ERNIE-Image the same day it released, which as I understand requires generating a bunch of images. I've been working to do something similar with my CLI program ggufy, and was curious of you had any info you could share on the kind of compute you put into that, and if you generate full images or look at latents?
      - danielhanchen 6mo ago
        Yes we have started doing diffusion GGUFs but it's in it's infancy :) But yes we do generate images to test quants out!
sigbottle 6mo ago
Is quantization a mostly solved pipeline at this point? I thought that architectures were varied and weird enough where you can't just click a button, say "go optimize these weights", and go. I mean new models have new code that they want to operate on, right, so you'd have to analyze the code and insert the quantization at the right places, automatically, then make sure that doesn't degrade perf? Maybe I just don't understand how quantization works, but I thought quantization was a very nasty problem involving a lot of plumbing
- Readerium 6mo ago
  that is true. gguf does not support any Architecture. for the most recent example, as of April 16, 2026 (today) Turboquant isnt still added to GGUF

ekianjo 6mo ago

yeah and often their quants are broken. They had to update their Gemma4 quants like 4 times in the past 2 weeks.

danielhanchen 6mo ago
No it's not our fault - re our 4 uploads - the first 3 are due to llama.cpp fixing bugs - this was out of our control (we're llama.cpp contributors, but not the main devs) - we could have waited, but it's best to update when multiple (10-20) bugs are fixed. The 4th is Google themselves improving the chat template for tool calling for Gemma. https://github.com/ggml-org/llama.cpp/issues/21255 https://github.com/ggml-org/llama.cpp/issues/21255 was another issue CUDA 13.2 was broken - this was NVIDIA's CUDA compiler itself breaking - fully out of our hands - but we provided a solution for it.

bildung 6mo ago

Bad QA :/ They had a bunch of broken quantizations in the last releases

danielhanchen 6mo ago
1. Gemma-4 we re-uploaded 4 times - 3 times were 10-20 llama.cpp bug fixes - we had to notify people to upload the correct ones. The 4th is an official Gemma chat template improvement from Google themselves. 2. Qwen3.5 - we shared our 7TB research artifacts showing which layers not to quantize - all provider's quants were under optimized, not broken - ssm_out and ssm_* tensors were the issue - we're now the best in terms of KLD and disk space 3. MiniMax 2.7 - we swiftly fixed it due to NaN PPL - we found the issue in all quants regardless of provider - so it affected everyone not just us. We wrote a post on it, and fixed it - others have taken our fix and fixed their quants, whilst some haven't updated. Note we also fixed bugs in many OSS models like Gemma 1, Gemma 3, Llama chat template fixes, Mistral, and many more. Unfortunately sometimes quants break, but we fix them quickly, and 95% of times these are out of our hand. We swiftly and quickly fix them, and write up blogs on what happened. Other providers simply just take our blogs and fixes and re-apply, re-use our fixes.
- bildung 6mo ago
  Fair enough, appreciate the detailed response! Can you elaborate why other quantizations weren't affected (e.g. bartowski)? Simply because they were straight Q4 etc. for every layer?
  - danielhanchen 6mo ago
    No Bartowski's are more affected - (38% NaN) than ours (22%) - for MiniMax 2.7 see https://www.reddit.com/r/LocalLLaMA/comments/1slk4di/minimax_m27_gguf_investigation_fixes_benchmarks/ https://www.reddit.com/r/LocalLLaMA/comments/1slk4di/minimax... We already fixed ours. Bart hasn't yet but is still working on it following our findings. blk.61.ffn_down_exps in Q4_K or Q5_K failed - it must be in Q6_K otherwise it overflows. For the others, yes layers in some precision don't work. For eg Qwen3.5 ssm_out must be minimum Q4-Q6_K. ssm_alpha and ssm_beta must be Q8_0 or higher. Again Bart and others apply our findings - see https://www.reddit.com/r/LocalLLaMA/comments/1rgel19/new_qwen3535ba3b_unsloth_dynamic_ggufs_benchmarks/ https://www.reddit.com/r/LocalLLaMA/comments/1rgel19/new_qwe...