3 ms·
Are you running it locally with llama.cpp? If so, is it working without any tweaking of the chat template? The tool calls fail for me when using the default cha
by abhikul0 7mo ago
Are you running it locally with llama.cpp? If so, is it working without any tweaking of the chat template? The tool calls fail for me when using the default chat template, however it seems to work a whole lot better with this: https://huggingface.co/Qwen/Qwen3.5-35B-A3B/discussions/9#699f8e8d16fcfc9bb99aa77f https://huggingface.co/Qwen/Qwen3.5-35B-A3B/discussions/9#69...
- arcanemachiner 7mo agoHave you tried the '--jinja' flag in llama-server?
- abhikul0 7mo agoYes, it fails too. I’m using the unsloth q4_km quant. Similarly fails with devstral2 small too, fixed that by using a similar template i found for it. Maybe it’s the quants that are broken, need to redownload I guess.
- sosodev 7mo agoI’ve been running it via llama-server with no issues. Running the latest Bartowski 6-bit quant
- brightball 7mo agoBartowski? Like Chuck Bartowski from the TV show?
- BoredomIsFun 7mo agoDifferent one. Bartowski is a minor celebrity in the local LLM world, together with Unsloth.
- Balinares 7mo agoWhat's the selling point of these quants vs the Unsloth ones?
- BoredomIsFun 7mo agoSometimes unsloth has broken ones for a particular model, sometimes no quants at all, and there is subtle difference in behavior.
- abhikul0 7mo agoThanks, i'll check his quants.