5 ms·

I just tested this on a bug fixing benchmark I'm working on. It did not perform as well as I expected. Qwen2.5-Coder-3B (2 years old) outperformed it by a wid

I just tested this on a bug fixing benchmark I'm working on. It did not perform as well as I expected. Qwen2.5-Coder-3B (2 years old) outperformed it by a wide range -> fixing ~50% of bugs whereas this model only fixed ~12%. Granted, it's not a coder specific model, but given its benchmark performance to Gemma models, and that it's two years newer, and that it's an MoE with 8B total params, I expected it to be more competitive.

HanClinto 4mo ago
Some of the coding-specific fine-tunes were really impressive boosts. Qwen2.5-3B-Instruct is also available [0] -- if it's not too much to ask, I'd be curious how more general models stack up in your benchmark? [0] - https://huggingface.co/Qwen/Qwen2.5-3B-Instruct https://huggingface.co/Qwen/Qwen2.5-3B-Instruct
debazel 4mo ago
I tried it with OpenCode and it is borderline incapable of using tool calls, so that might be why it is doing so bad on your test.
- peder 4mo ago
  I just did the same. Absolutely awful. I assume OpenCode's heavy context is a problem, and it's probably better to use Liquid's own OpenCode alternative for this.
  - solarkraft 4mo ago
    Where can I find that agent harness? A look at their Docs and asking Gemini yielded no results. Edit: Is it this? https://github.com/Liquid4All/cookbook/tree/main/examples/localcowork https://github.com/Liquid4All/cookbook/tree/main/examples/lo... FYI: Opencode is very well tuned for Qwen models, but I haven’t found it that rare for niche models to perform badly in it.
XCSme 4mo ago
I will test it when it's accessible via OpenRouter, but the previous LFM2 model (lfm-2-24b-a2b) didn't do well on my tests, it got only 1/20 questions/tasks right, way below Gemma 31B or Qwen 35b-a3b (those get like 10/20 right)
- BoorishBears 4mo ago
  I tested it against Gemma 4 31B and it's expectedly not favorable for world knowledge. But even against E4B it's shaky, which is surprising given how many tokens they trained on. I guess it was on a lot of synthetic data.
walrus01 4mo ago
I personally find any model smaller than something like Qwen 3.6 35B-A3B (8-bit quantization, about 49GB memory usage when loaded into llama.cpp) to be too "stupid" for reliable use. I would much rather not run the model on my local laptop hardware and offload that to some system sitting under my desk in my home office, accessible via VPN, than take the risk of using an unreliable and flaky tool for the convenience of having it on the same hardware on my lap. I pay very little attention to 8 billion or whatever (or even much smaller) models these days and I don't feel like I'm missing much.
- satvikpendem 4mo ago
  Qwen 3.6 27B dense is much better than the 35B MoE model for coding, not sure if you've tried that yet.
  - walrus01 4mo ago
    yes, I have, I use both. 27B slower in tok/s due to density, obviously, 35B-A3B for speed on simpler tasks.
    - intothemild 4mo ago
      You should enable MTP now that its available. LLamaCPP has had some massive updates in the last week or so.
      - npodbielski 4mo ago
        Yes, Qwen 3.6 MoE is hitting like 80-90tk/s on Strix halo. On R9700 I had like 170t/s. It was not possible to keep up. But MoE is circling very often. I switch then to dense model and have 20-30t/s but it is able to solve quite a lot of tasks.
        intothemild 4mo ago
        I get 50-60t/s tg on my r9700 with the dense, unsloth MTP quant UD-Q5_K_XL, K@8/V@4 256k context. Using Vulkan backend. ``` llama-server -fa on -t 7 -ngl 999 --mlock --fit off --kv-offload --no-webui --metrics --chat-template-kwargs {"preserve_thinking": true} -b 2048 -ub 1024 -m /mnt/models/unsloth/Qwen3.6-27B-MTP-GGUF/Qwen3.6-27B-UD-Q5_K_XL.gguf --mmproj /mnt/models/unsloth/Qwen3.6-27B-MTP-GGUF/mmproj-F16.gguf -c 262144 --kv-unified -ctk q8_0 -ctv q4_0 --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-ngl 99 --alias unsloth/Qwen3.6-27B-MTP-GGUF --temp 0.60 --top-k 20 --top-p 0.95 --min-p 0.00 --presence-penalty 0.00 --repeat-penalty 1.00 ```
deleted 4mo ago
[deleted]
h14h 4mo ago
That's not all that surprising, IMO. From what I understand, LiquidAI is focusing pretty narrowly on building models that operate as the "agentic core" of a larger system. If I were going to use this model, I'd be looking to use it more as is the primary chat interface of a larger system, and having it orchestrate & delegate tasks to other places via tool calls. It's not quite as exciting on the surface as a local "do it all" model, but it does enable some pretty neat use-cases, IMO. I'm imagining a local agent that is super low latency, works entirely offline, and capable of queuing up complex tasks for larger/smarter cloud agents which execute them asynchronously.
- onlyrealcuzzo 4mo ago
  Interesting... Two of the other responses speak about it being abysmal at tool calling. Overall, I'm pretty impressed a model this small can find/fix ~12% of bugs with crappy context - even if they're about as easy as possible to fix. I just assumed it would perform better, given all the advancements in the space. It's possible 1B active parameters is just not enough - even if it has 8B params of knowledge to reason through bugs. Playing around with the context I fed it, it was able to fix up to ~34% of bugs vs ~46% for Qwen2.5-Coder-3B and ~54% for Qwen2.5-Coder-7B.
mike_hearn 4mo ago
It's not intended to be a coding model, however.

HanClinto 4mo ago

Some of the coding-specific fine-tunes were really impressive boosts. Qwen2.5-3B-Instruct is also available [0] -- if it's not too much to ask, I'd be curious how more general models stack up in your benchmark? [0] - https://huggingface.co/Qwen/Qwen2.5-3B-Instruct https://huggingface.co/Qwen/Qwen2.5-3B-Instruct

debazel 4mo ago

I tried it with OpenCode and it is borderline incapable of using tool calls, so that might be why it is doing so bad on your test.

peder 4mo ago
I just did the same. Absolutely awful. I assume OpenCode's heavy context is a problem, and it's probably better to use Liquid's own OpenCode alternative for this.
- solarkraft 4mo ago
  Where can I find that agent harness? A look at their Docs and asking Gemini yielded no results. Edit: Is it this? https://github.com/Liquid4All/cookbook/tree/main/examples/localcowork https://github.com/Liquid4All/cookbook/tree/main/examples/lo... FYI: Opencode is very well tuned for Qwen models, but I haven’t found it that rare for niche models to perform badly in it.

XCSme 4mo ago

I will test it when it's accessible via OpenRouter, but the previous LFM2 model (lfm-2-24b-a2b) didn't do well on my tests, it got only 1/20 questions/tasks right, way below Gemma 31B or Qwen 35b-a3b (those get like 10/20 right)

BoorishBears 4mo ago
I tested it against Gemma 4 31B and it's expectedly not favorable for world knowledge. But even against E4B it's shaky, which is surprising given how many tokens they trained on. I guess it was on a lot of synthetic data.

walrus01 4mo ago

I personally find any model smaller than something like Qwen 3.6 35B-A3B (8-bit quantization, about 49GB memory usage when loaded into llama.cpp) to be too "stupid" for reliable use. I would much rather not run the model on my local laptop hardware and offload that to some system sitting under my desk in my home office, accessible via VPN, than take the risk of using an unreliable and flaky tool for the convenience of having it on the same hardware on my lap. I pay very little attention to 8 billion or whatever (or even much smaller) models these days and I don't feel like I'm missing much.

satvikpendem 4mo ago
Qwen 3.6 27B dense is much better than the 35B MoE model for coding, not sure if you've tried that yet.
- walrus01 4mo ago
  yes, I have, I use both. 27B slower in tok/s due to density, obviously, 35B-A3B for speed on simpler tasks.
  - intothemild 4mo ago
    You should enable MTP now that its available. LLamaCPP has had some massive updates in the last week or so.
    - npodbielski 4mo ago
      Yes, Qwen 3.6 MoE is hitting like 80-90tk/s on Strix halo. On R9700 I had like 170t/s. It was not possible to keep up. But MoE is circling very often. I switch then to dense model and have 20-30t/s but it is able to solve quite a lot of tasks.
      - intothemild 4mo ago
        I get 50-60t/s tg on my r9700 with the dense, unsloth MTP quant UD-Q5_K_XL, K@8/V@4 256k context. Using Vulkan backend. ``` llama-server -fa on -t 7 -ngl 999 --mlock --fit off --kv-offload --no-webui --metrics --chat-template-kwargs {"preserve_thinking": true} -b 2048 -ub 1024 -m /mnt/models/unsloth/Qwen3.6-27B-MTP-GGUF/Qwen3.6-27B-UD-Q5_K_XL.gguf --mmproj /mnt/models/unsloth/Qwen3.6-27B-MTP-GGUF/mmproj-F16.gguf -c 262144 --kv-unified -ctk q8_0 -ctv q4_0 --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-ngl 99 --alias unsloth/Qwen3.6-27B-MTP-GGUF --temp 0.60 --top-k 20 --top-p 0.95 --min-p 0.00 --presence-penalty 0.00 --repeat-penalty 1.00 ```

deleted 4mo ago

[deleted]

h14h 4mo ago

That's not all that surprising, IMO. From what I understand, LiquidAI is focusing pretty narrowly on building models that operate as the "agentic core" of a larger system. If I were going to use this model, I'd be looking to use it more as is the primary chat interface of a larger system, and having it orchestrate & delegate tasks to other places via tool calls. It's not quite as exciting on the surface as a local "do it all" model, but it does enable some pretty neat use-cases, IMO. I'm imagining a local agent that is super low latency, works entirely offline, and capable of queuing up complex tasks for larger/smarter cloud agents which execute them asynchronously.

onlyrealcuzzo 4mo ago
Interesting... Two of the other responses speak about it being abysmal at tool calling. Overall, I'm pretty impressed a model this small can find/fix ~12% of bugs with crappy context - even if they're about as easy as possible to fix. I just assumed it would perform better, given all the advancements in the space. It's possible 1B active parameters is just not enough - even if it has 8B params of knowledge to reason through bugs. Playing around with the context I fed it, it was able to fix up to ~34% of bugs vs ~46% for Qwen2.5-Coder-3B and ~54% for Qwen2.5-Coder-7B.

mike_hearn 4mo ago

It's not intended to be a coding model, however.