4 ms·

Wow. I tried to get Qwen3.8 4B to parse song lyrics and analyse them. Getting ollama running was a minute or two. However coming up with a prompt that didn't t

Wow. I tried to get Qwen3.8 4B to parse song lyrics and analyse them. Getting ollama running was a minute or two. However coming up with a prompt that didn't turn out total garbage was impossible. After wasting over an hour and I ended up getting Qwen side by side with Llama 3.2 3B, just to see if I was being stupid. Nope, it just looks like Llama is orders of magnitude better at this specific task for some reason). If you think I'm doing it wrong, you're probably right, I don't know a ton about local LLMs. But I hand selected 50 songs, set up ollama with both LLMs, and for each iteration on the prompt text, ran both LLMs 10x times per song. Side-by-side comparisons showed that Qwen 3 4B was so bad that I actually downloaded Qwen again, thinking there must have been some mistake and I accidentally grabbed an old 1B model.

petu 1mo ago
There's no official Qwen 3.8 4B (only 27B and 2.4T.. at least for now), so if not a typo you've downloaded some third party model/finetune. Also if you have less than 24GB VRAM, then ollama defaults to 4K context. If that "Qwen 3.8" uses thinking, it might be running out of context and forgetting what it was even answering mid-generation. If that's the case, then try increasing context size: https://docs.ollama.com/context-length https://docs.ollama.com/context-length , but also: https://sleepingrobots.com/dreams/stop-using-ollama/ https://sleepingrobots.com/dreams/stop-using-ollama/
- selcuka 1mo ago
  > There's no official Qwen 3.8 4B (only 27B and 2.4T.. at least for now), so if not a typo you've downloaded some third party model/finetune. Probably this: https://huggingface.co/empero-ai/Qwen3.8-4B-Distill https://huggingface.co/empero-ai/Qwen3.8-4B-Distill > Qwen3.8-4B is a full-parameter distillation of Qwen3.8 2.4T A95B into the Qwen3.5-4B architecture.
  - formerly_proven 1mo ago
    That's not distillation, that's model homeopathy.
    - picofarad 1mo ago
      Too strong for homeopathy but your point is taken. 0.3% vs 0.000005%
      - formerly_proven 1mo ago
        4B / 2.4T is 1/600 or 0.16%, almost a 3X dilution. A bit on the weak side for homeopathy, but still readily available for many ingredients :)
- raffraffraff 1mo ago
  Right, well I'm walking the hounds right now but I think I got something like this https://huggingface.co/Qwen/Qwen3-4B/blob/main/README.md https://huggingface.co/Qwen/Qwen3-4B/blob/main/README.md SO, if that isn't official it explains the results I got. They were dreadful.
  - ElectricalUnion 1mo ago
    Qwen3 is not Qwen3.8? > commited on May 21, 2025, over 1 year ago Fairly old update to the README.md of (instead of Qwen3.8), should have raised some flags?
  - petu 1mo ago
    What you've linked is very much official -- that's original Qwen 3.0 release, so pretty old, but official.
  - selcuka 1mo ago
    There is a more recent Qwen with 4B parameters, but there is no 3.8 (or 3.6): https://huggingface.co/Qwen/Qwen3.5-4B https://huggingface.co/Qwen/Qwen3.5-4B
6wf9j 1mo ago
I found Gemma 4 models quite good at translating/explaining song lyrics. I actually prefer using 26B-A4B for this task instead of GPT
embedding-shape 1mo ago
> Nope, it just looks like Llama is orders of magnitude better at this specific task for some reason This is almost every ML model, if the task isn't part directly or indirectly of the datasets they use for training it, then the model is gonna be pretty trash at it. What the big AI labs have over the smaller labs, is a huge amount of data and diverse set of tasks, hence they generalize better, but still not great. So, how do you avoid having to spend hours on figuring out if the model is just dumb, or don't know the task? Your own private benchmarks! Figure out a way, ideally without using another LLM, to score how good a model is at doing your specific task. Come up with 3-5 examples for this benchmark yourself, ask a SOTA LLM to fill out 45 more, review everything VERY closely, then use this whenever you want to figure out if $new_model actually is an improvement over what you use today, and once you have a bunch of different tasks, you'll see that all these HUGE improvements tend to be specifically for the benchmarks they mention in the press release, as many of your own benchmarks won't show that big of a difference in reality. Yes there is a higher upfront cost, but if you're building longer-term projects that rely on LLM models, particularly local ones that seem very benchmaxxed a lot of the times, you need a quick and reproducible way of scoring them somehow, where you can just add more models to compare, and you need to keep these benchmarks to yourself.
- porridgeraisin 1mo ago
  This is the way. It's also very important to automate as much of this verification as possible into the harness, rather than sit there and prod it in the chat. Of course some things are not auto verifiable, and you'll have to give human judgement and input there, but you'll save a lot more time if you spend 1 week painstakingly writing checks for as many little things as possible and integrating them into the harness.
- catlifeonmars 1mo ago
  > What the big AI labs have over the smaller labs, is a huge amount of data and diverse set of tasks, hence they generalize better, but still not great. Hehe, this kind of sounds like the opposite of generalization. As in it’s just specialization at scale.
  - 1mo ago
BoredomIsFun 1mo ago
Llama 3.2 3B was surprisingly good for general-purpose text manipulation tasks. I'd argue it might be better than many modern tiny models for that.

petu 1mo ago

There's no official Qwen 3.8 4B (only 27B and 2.4T.. at least for now), so if not a typo you've downloaded some third party model/finetune. Also if you have less than 24GB VRAM, then ollama defaults to 4K context. If that "Qwen 3.8" uses thinking, it might be running out of context and forgetting what it was even answering mid-generation. If that's the case, then try increasing context size: https://docs.ollama.com/context-length https://docs.ollama.com/context-length , but also: https://sleepingrobots.com/dreams/stop-using-ollama/ https://sleepingrobots.com/dreams/stop-using-ollama/

selcuka 1mo ago
> There's no official Qwen 3.8 4B (only 27B and 2.4T.. at least for now), so if not a typo you've downloaded some third party model/finetune. Probably this: https://huggingface.co/empero-ai/Qwen3.8-4B-Distill https://huggingface.co/empero-ai/Qwen3.8-4B-Distill > Qwen3.8-4B is a full-parameter distillation of Qwen3.8 2.4T A95B into the Qwen3.5-4B architecture.
- formerly_proven 1mo ago
  That's not distillation, that's model homeopathy.
  - picofarad 1mo ago
    Too strong for homeopathy but your point is taken. 0.3% vs 0.000005%
    - formerly_proven 1mo ago
      4B / 2.4T is 1/600 or 0.16%, almost a 3X dilution. A bit on the weak side for homeopathy, but still readily available for many ingredients :)
raffraffraff 1mo ago
Right, well I'm walking the hounds right now but I think I got something like this https://huggingface.co/Qwen/Qwen3-4B/blob/main/README.md https://huggingface.co/Qwen/Qwen3-4B/blob/main/README.md SO, if that isn't official it explains the results I got. They were dreadful.
- ElectricalUnion 1mo ago
  Qwen3 is not Qwen3.8? > commited on May 21, 2025, over 1 year ago Fairly old update to the README.md of (instead of Qwen3.8), should have raised some flags?
- petu 1mo ago
  What you've linked is very much official -- that's original Qwen 3.0 release, so pretty old, but official.
- selcuka 1mo ago
  There is a more recent Qwen with 4B parameters, but there is no 3.8 (or 3.6): https://huggingface.co/Qwen/Qwen3.5-4B https://huggingface.co/Qwen/Qwen3.5-4B

6wf9j 1mo ago

I found Gemma 4 models quite good at translating/explaining song lyrics. I actually prefer using 26B-A4B for this task instead of GPT

embedding-shape 1mo ago

> Nope, it just looks like Llama is orders of magnitude better at this specific task for some reason This is almost every ML model, if the task isn't part directly or indirectly of the datasets they use for training it, then the model is gonna be pretty trash at it. What the big AI labs have over the smaller labs, is a huge amount of data and diverse set of tasks, hence they generalize better, but still not great. So, how do you avoid having to spend hours on figuring out if the model is just dumb, or don't know the task? Your own private benchmarks! Figure out a way, ideally without using another LLM, to score how good a model is at doing your specific task. Come up with 3-5 examples for this benchmark yourself, ask a SOTA LLM to fill out 45 more, review everything VERY closely, then use this whenever you want to figure out if $new_model actually is an improvement over what you use today, and once you have a bunch of different tasks, you'll see that all these HUGE improvements tend to be specifically for the benchmarks they mention in the press release, as many of your own benchmarks won't show that big of a difference in reality. Yes there is a higher upfront cost, but if you're building longer-term projects that rely on LLM models, particularly local ones that seem very benchmaxxed a lot of the times, you need a quick and reproducible way of scoring them somehow, where you can just add more models to compare, and you need to keep these benchmarks to yourself.

porridgeraisin 1mo ago
This is the way. It's also very important to automate as much of this verification as possible into the harness, rather than sit there and prod it in the chat. Of course some things are not auto verifiable, and you'll have to give human judgement and input there, but you'll save a lot more time if you spend 1 week painstakingly writing checks for as many little things as possible and integrating them into the harness.
catlifeonmars 1mo ago
> What the big AI labs have over the smaller labs, is a huge amount of data and diverse set of tasks, hence they generalize better, but still not great. Hehe, this kind of sounds like the opposite of generalization. As in it’s just specialization at scale.
- 1mo ago

BoredomIsFun 1mo ago

Llama 3.2 3B was surprisingly good for general-purpose text manipulation tasks. I'd argue it might be better than many modern tiny models for that.