3 ms·

Nice! Does anyone know how this compares to the Unsloth quantizations of this model? https://unsloth.ai/docs/models/qwen3.8#run-qwen3.8-guide https://unsloth.ai

Nice! Does anyone know how this compares to the Unsloth quantizations of this model? https://unsloth.ai/docs/models/qwen3.8#run-qwen3.8-guide https://unsloth.ai/docs/models/qwen3.8#run-qwen3.8-guide

0xbadcafebee 14d ago
Came to ask the same. From my really rough understanding, it seems like Unsloth's method allows a slightly higher precision at a higher file size, while PrismML's uses a different approach to achieve a smaller size (and presumably less precision).
nulld3v 14d ago
There's a table on the HF page that compares it against Unsloth's UD-Q4_K_XL and IQ2_XXS: https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#full-per-benchmark-results https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#fu...
- kadoban 14d ago
  Oh, wow, they think it's just a smidge below the q4? That's crazy good if true.
  - anana_ 14d ago
    The benchmarks they chose are rather cherry picked to not include long context or difficult ones that involve long horizon work or many agent turns, as I suspect this is where the model shows more differences compared to the full fat one
    - SillyUsername 14d ago
      Yep more hops from the lower Q is likely going to skew the vectors further over time. I wonder if there's a way to mitigate this by running it through an original Q8 draft model, attuned somehow for the PTQ1 quant, but giving it a higher threshold for the acceptance linear with the context length itself? The longer the context, the higher the multiplier on the threshold, and more likely the draft result is used. Not ideal but it may extend the usable max context. This model might, even without this, be amazing for short lived agents that work via generations / have changing tasks.
deleted 14d ago
[deleted]
WithinReason 14d ago
Unsloth has been dethroned by ISTA: https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF The 3-bit quant is lossless based on benchmarks.
- raylad 13d ago
  I just checked that model (ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF:IQ3_S) and it does much much worse on the "Please recite Jabberwocky" test than the original bf16 does. The bf16 only misses "snicker-snack" and this quantization becomes confused after the first stanza.
  - sorenjan 13d ago
    I think it should be expected that a smaller model is worse at reciting memorized data than a big one. I also don't think it's a good use case of small local models. Can it find and recite Jabberwocky if given access to a web search tool?
    - 0xbadcafebee 13d ago
      Forgetting things isn't lossless though is it? Makes the benchmark and the finding quite suspect
      - sorenjan 12d ago
        It depends on what you mean by lossless. If both models can perform the same tasks it can be considered lossless for those tasks. That task might be more related to language understanding rather than memorizing, they have several benchmarks in the article.
  - WithinReason 13d ago
    [dead]

Nice! Does anyone know how this compares to the Unsloth quantizations of this model? https://unsloth.ai/docs/models/qwen3.8#run-qwen3.8-guide https://unsloth.ai/docs/models/qwen3.8#run-qwen3.8-guide

0xbadcafebee 14d ago

Came to ask the same. From my really rough understanding, it seems like Unsloth's method allows a slightly higher precision at a higher file size, while PrismML's uses a different approach to achieve a smaller size (and presumably less precision).

nulld3v 14d ago

There's a table on the HF page that compares it against Unsloth's UD-Q4_K_XL and IQ2_XXS: https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#full-per-benchmark-results https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf#fu...

kadoban 14d ago
Oh, wow, they think it's just a smidge below the q4? That's crazy good if true.
- anana_ 14d ago
  The benchmarks they chose are rather cherry picked to not include long context or difficult ones that involve long horizon work or many agent turns, as I suspect this is where the model shows more differences compared to the full fat one
  - SillyUsername 14d ago
    Yep more hops from the lower Q is likely going to skew the vectors further over time. I wonder if there's a way to mitigate this by running it through an original Q8 draft model, attuned somehow for the PTQ1 quant, but giving it a higher threshold for the acceptance linear with the context length itself? The longer the context, the higher the multiplier on the threshold, and more likely the draft result is used. Not ideal but it may extend the usable max context. This model might, even without this, be amazing for short lived agents that work via generations / have changing tasks.

deleted 14d ago

[deleted]

WithinReason 14d ago

Unsloth has been dethroned by ISTA: https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF The 3-bit quant is lossless based on benchmarks.

raylad 13d ago
I just checked that model (ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF:IQ3_S) and it does much much worse on the "Please recite Jabberwocky" test than the original bf16 does. The bf16 only misses "snicker-snack" and this quantization becomes confused after the first stanza.
- sorenjan 13d ago
  I think it should be expected that a smaller model is worse at reciting memorized data than a big one. I also don't think it's a good use case of small local models. Can it find and recite Jabberwocky if given access to a web search tool?
  - 0xbadcafebee 13d ago
    Forgetting things isn't lossless though is it? Makes the benchmark and the finding quite suspect
    - sorenjan 12d ago
      It depends on what you mean by lossless. If both models can perform the same tasks it can be considered lossless for those tasks. That task might be more related to language understanding rather than memorizing, they have several benchmarks in the article.
- WithinReason 13d ago
  [dead]