3 ms·
The problem, of course, is if you run the UD_Q2 variant (Unsloth) which does only post-training, the number is pretty close to 1-bit model here and the 5% drop
by liuliu 3mo ago
The problem, of course, is if you run the UD_Q2 variant (Unsloth) which does only post-training, the number is pretty close to 1-bit model here and the 5% drop in tool-call is significant than it suggests in real-life use cases.
- liuliu 3mo agoYou also need to pay close attention to BFCLv3 multi-turn result, that helps you to get a sense how frequently these quants will be in a doom loop.
- h14h 3mo agoI'm curious what kind of results one could get from combining the clever quantization PrismML is doing here with something like LiquidAI's antidoom: https://github.com/Liquid4All/antidoom https://github.com/Liquid4All/antidoom