5 ms·

Can someone explain how a 27B model (quantized no less) ever be comparable to a model like Sonnet 4.0 which is likely in the mid to high hundreds of billions of

by codemog 7mo ago

Can someone explain how a 27B model (quantized no less) ever be comparable to a model like Sonnet 4.0 which is likely in the mid to high hundreds of billions of parameters? Is it really just more training data? I doubt it’s architecture improvements, or at the very least, I imagine any architecture improvements are marginal.

otabdeveloper4 7mo ago
There's diminishing returns bigly when you increase parameter count. The sweet spot isn't in the "hundreds of billions" range, it's much lower than that. Anyways your perception of a model's "quality" is determined by careful post-training.
- zozbot234 7mo ago
  More parameters improves general knowledge a lot, but you have to quantize more in order to fit in a given amount of memory, which if taken to extremes leads to erratic behavior. For casual chat use even Q2 models can be compelling, agentic use requires more regularization thus less quantized parameters and lowering the total amount to compensate.
- codemog 7mo ago
  Interesting. I see papers where researchers will finetune models in the 7 to 12b range and even beat or be competitive with frontier models. I wish I knew how this was possible, or had more intuition on such things. If anyone has paper recommendations, I’d appreciate it.
  - stavros 7mo ago
    They're using a revolutionary new method called "training on the test set".
    - drob518 7mo ago
      So, curve fitting the training data? So, we should expect out of sample accuracy to be crap?
      - stavros 7mo ago
        Yeah, that's usually what tends to happen with those tiny models that are amazing in benchmarks.
revolvingthrow 7mo ago
It doesn’t. I’m not sure it outperforms chatgpt 3
- gunalx 7mo ago
  3 not 3.5? I think I would even prefer the qwen3.5 0.8b over GPT 3.
- BoredomIsFun 7mo ago
  You are not being serrious, are you? even 1.5 years old Mistral and Meta models outperform ChatGPT 3.
spwa4 7mo ago
The short answer is that there are more things that matter than parameter count, and we are probably nowhere near the most efficient way to make these models. Also: the big AI labs have shown a few times that internally they have way more capable models
girvo 7mo ago
Considering the full fat Qwen3.5-plus is good, but barely Sonnet 4 good in my testing (but incredibly cheap!) I doubt the quantised versions are somehow as good if not better in practice.
- stavros 7mo ago
  When you say Sonnet 4, do you mean literally 4, or 4.6?
  - girvo 7mo ago
    It's not as capable as Sonnet 4.6 in my usage over the past couple days, through a few different coding harnesses (including my own for-play one[0], that's been quite fun). [0] https://github.com/girvo/girvent/ https://github.com/girvo/girvent/
    - dr_kiszonka 7mo ago
      What is the benefit of writing your own harness? I am asking because I need to get better at using AI for programming. I have used Cursor, Gemini CLI, Antigravity quite a bit and have had a lot of difficulties getting them do what I want. They just tend to "know better."
      - newswasboring 7mo ago
        I think it's the same instinct as making your own Game Engine. You start off either because you want to learn how they work or because you think your game is special and needs its own engine. Usually, it's a combination of both.
      - everforward 7mo ago
        I’m not an expert but I started with smaller tasks to get a feel for how to phrase things, what I need to include. It’s more manageable to manually fix things it screwed up than giving it full reign. You may want to look at the AGENTS.md file too so you can include your stock style things if it’s repeatedly screwing up in the same way.
alecco 7mo ago
AFAIK post-training and distillation techniques advanced a lot in the past couple of years. SOTA big models get new frontier and within 6 months it trickles down to open models with 10x less parameters. And mind the source pre-training data was not made/written for training LLMs, it's just random stuff from Internet, books, etc. So there's a LOT of completely useless an contradictory information. Better training texts are way better and you can just generate & curate from those huge frontier LLMs. This was shown in the TinyStories paper where GPT-4 generated children's stories could make models 3 orders of magnitude smaller achieve quite a lot. This is why the big US labs complain China is "stealing" their work by distilling their models. Chinese labs save many billions in training with just a bunch of accounts. (I'm just stating what they say, not giving my opinion).

by codemog 7mo ago

otabdeveloper4 7mo ago

There's diminishing returns bigly when you increase parameter count. The sweet spot isn't in the "hundreds of billions" range, it's much lower than that. Anyways your perception of a model's "quality" is determined by careful post-training.

zozbot234 7mo ago
More parameters improves general knowledge a lot, but you have to quantize more in order to fit in a given amount of memory, which if taken to extremes leads to erratic behavior. For casual chat use even Q2 models can be compelling, agentic use requires more regularization thus less quantized parameters and lowering the total amount to compensate.
codemog 7mo ago
Interesting. I see papers where researchers will finetune models in the 7 to 12b range and even beat or be competitive with frontier models. I wish I knew how this was possible, or had more intuition on such things. If anyone has paper recommendations, I’d appreciate it.
- stavros 7mo ago
  They're using a revolutionary new method called "training on the test set".
  - drob518 7mo ago
    So, curve fitting the training data? So, we should expect out of sample accuracy to be crap?
    - stavros 7mo ago
      Yeah, that's usually what tends to happen with those tiny models that are amazing in benchmarks.

revolvingthrow 7mo ago

It doesn’t. I’m not sure it outperforms chatgpt 3

gunalx 7mo ago
3 not 3.5? I think I would even prefer the qwen3.5 0.8b over GPT 3.
BoredomIsFun 7mo ago
You are not being serrious, are you? even 1.5 years old Mistral and Meta models outperform ChatGPT 3.

spwa4 7mo ago

The short answer is that there are more things that matter than parameter count, and we are probably nowhere near the most efficient way to make these models. Also: the big AI labs have shown a few times that internally they have way more capable models

girvo 7mo ago

Considering the full fat Qwen3.5-plus is good, but barely Sonnet 4 good in my testing (but incredibly cheap!) I doubt the quantised versions are somehow as good if not better in practice.

stavros 7mo ago
When you say Sonnet 4, do you mean literally 4, or 4.6?
- girvo 7mo ago
  It's not as capable as Sonnet 4.6 in my usage over the past couple days, through a few different coding harnesses (including my own for-play one[0], that's been quite fun). [0] https://github.com/girvo/girvent/ https://github.com/girvo/girvent/
  - dr_kiszonka 7mo ago
    What is the benefit of writing your own harness? I am asking because I need to get better at using AI for programming. I have used Cursor, Gemini CLI, Antigravity quite a bit and have had a lot of difficulties getting them do what I want. They just tend to "know better."
    - newswasboring 7mo ago
      I think it's the same instinct as making your own Game Engine. You start off either because you want to learn how they work or because you think your game is special and needs its own engine. Usually, it's a combination of both.
    - everforward 7mo ago
      I’m not an expert but I started with smaller tasks to get a feel for how to phrase things, what I need to include. It’s more manageable to manually fix things it screwed up than giving it full reign. You may want to look at the AGENTS.md file too so you can include your stock style things if it’s repeatedly screwing up in the same way.

alecco 7mo ago

AFAIK post-training and distillation techniques advanced a lot in the past couple of years. SOTA big models get new frontier and within 6 months it trickles down to open models with 10x less parameters. And mind the source pre-training data was not made/written for training LLMs, it's just random stuff from Internet, books, etc. So there's a LOT of completely useless an contradictory information. Better training texts are way better and you can just generate & curate from those huge frontier LLMs. This was shown in the TinyStories paper where GPT-4 generated children's stories could make models 3 orders of magnitude smaller achieve quite a lot. This is why the big US labs complain China is "stealing" their work by distilling their models. Chinese labs save many billions in training with just a bunch of accounts. (I'm just stating what they say, not giving my opinion).