5 ms·

> People are realizing that what they need isn't more general intelligence, it's more specialization. A small but well tuned coding model... It’s not quite as

> People are realizing that what they need isn't more general intelligence, it's more specialization. A small but well tuned coding model... It’s not quite as simple as that. Several studies have shown the opposite: models trained on more diverse knowledge tend to cross-pollinate across domains. So a more generalized model can actually perform better than a specialized one. That’s why you’re not seeing tons of tiny models (one for Python, one for Pascal, one for Rust, etc).

kennywinker 20d ago
This is definitely the position of the big ai companies. But it doesn't match my experience. Qwen3.8 27b is clearly smarter at coding than MANY bigger models. gpt-oss-120b for example, is almost 4x the size, and performs way worse at coding tasks. It's clear to me that you can build small models that work well at specific tasks. Python vs Rust is probably too fine grained a way to build a model. Coding in general seems like a better target. There will always be a place for large generalist models, no doubt. But I think that place is much smaller than the big ai companies are counting on.
- vunderba 20d ago
  I think we’re in agreement. I make heavy use of smaller local models on a daily basis (Qwen3-VL for auto-captioning images, Gemma3:27b for some translation work, etc.). Gemma3:27b is a good example of a very capable general purpose multimodal model and has handled almost everything I've thrown at it from sentiment analysis to documentation writing. I suppose I was drawing a distinction between specialized and general intelligence versus small and large. I don’t think those are necessarily mutually exclusive.
- ac29 19d ago
  gpt-oss-120b only has 5B active parameters, so its not surprising Qwen3.8 27B outperforms it (Qwen3.8 is also ~13 months newer, which is forever in LLMs)
  - kennywinker 19d ago
    Fair enough. I’ve barley touched oss-120b, so i didn’t know it was so few active params. For a direct comparison, qwen3.6-35b-a3b is still better at coding than oss-120b. And Qwen3.8-27b is still better at coding than opus 4.1. Yes, if you list off models 27b is better than it’s all older models. But that’s my point - newer models are better than older models at the same AND much smaller size. That’s because model size matters less than they say. Training data and model architecture matter more.
  - anon373839 19d ago
    No, it’s not the active parameters. Qwen 3.8 Flash has 6B active and it smokes both models.
- ericd 19d ago
  Gpt-oss—120b is like 1000 years old in AI years, whereas Qwen 3.8 27b is pretty young. What you’re seeing is that parameters aren’t apples to apples, and at a given parameter level, the new models are much, much better than the ones from a year or two ago. Like, to a comical degree.
  - MichaelZuo 19d ago
    Wasnt this known by everyone who cared to pay attention? It practically became a joke about how a huge amount of the training data for GPT-4 was bottom of the barrel reddit vomit and obvious bot spam. Leading to many bizarre edge cases.
  - kennywinker 19d ago
    Does that not prove my point? Bigger doesn’t automatically mean better. Quality of training data, and model structure, matters as much or more than size
    - ericd 19d ago
      Ah sorry, I should've continued, the bigger recent models are commensurately smarter. If you really want to make the point, then you'd need to show 27b being smarter than similar vintage bigger models. And in that case, there's confounding issues like efficiency, speed due to excessive thinking maybe to make up for the smaller amount of world knowledge baked in (qwen 27b's main issue iirc), etc - they're tuned for different things.
      - kennywinker 19d ago
        https://artificialanalysis.ai/?models=gpt-5-3-codex%2Cqwen3-7-max%2Cqwen3-8-27b%2Ck2-horizon-375b-a23b%2Cgpt-5-5-instant-06-26%2Cclaude-sonnet-5-medium%2Cqwen3-6-plus%2Cdeepseek-v4-flash#artificial-analysis-intelligence-index https://artificialanalysis.ai/?models=gpt-5-3-codex%2Cqwen3-... Shows qwen3.8-27b along side seven larger models of ~similar vintage. Only one scores above 27b. Many of those are closed models so idk their exact parameter count / active param count, but it hardly matters - i’m sure all of them are far above 100b params My point is not that bigger is pointless. It’s just clearly not the only road to take to make a model better, which is obvious just from seeing how models of the same size have gotten better over the past few years
lelanthran 20d ago
Problem is conflict of interest: the studies are mostly from the providers of the biggest models, or someone who received free tokens to do the research.
- phoghed 19d ago
  It would be nice to hear exactly how the conflict of interest has impacted the specific studies and how they are wrong rather than conspiracy theory level speculation and hand waving at the entire category
  - butterNaN 19d ago
    I think is more than reasonable to be suspicious of studies funded by party with conflict of interest. Think of how many studies about climate change were funded by big polluters, for example. In 2026, the default outlook should be suspicion for any big private organisations with profit motive.
    - phoghed 19d ago
      I think it’s less than reasonable to operate mainly on vibes, rumors, and hearsay. You don’t need to think about climate change studies. Instead you can read the allegedly tainted studies we’re actually talking about and profess to all of us what is wrong with them. You can’t point to exactly where they’ve fudged them.
      - zug_zug 19d ago
        In an alternate universe where all research is completely auditable and we all have infinite time, yes that's a valid approach. And you're welcome to spend your life going that route, but the rest of us are gonna trust our common sense on this.
      - kennywinker 19d ago
        They don't even need to fudge the data on the studies they have published. They just need to hold back other studies that contradict the idea. Pouring over the published data looking for flaws will never get us access to the unpublished data.
vmg12 19d ago
> more diverse knowledge tend to cross-pollinate across domains Yeah, the cross domain transfer learning from RL is overstated by a lot.
CuriouslyC 19d ago
The western labs are very AGI pilled, and their public models are distilled down from larger research-only models that are uneconomical to serve directly. They could (and probably will) start distilling models for more niche use cases eventually, but we're not there yet.
RA_Fisher 19d ago
Yes, you're referring to transfer learning. It's an empirical question that also depends on the nature of activities.

kennywinker 20d ago

This is definitely the position of the big ai companies. But it doesn't match my experience. Qwen3.8 27b is clearly smarter at coding than MANY bigger models. gpt-oss-120b for example, is almost 4x the size, and performs way worse at coding tasks. It's clear to me that you can build small models that work well at specific tasks. Python vs Rust is probably too fine grained a way to build a model. Coding in general seems like a better target. There will always be a place for large generalist models, no doubt. But I think that place is much smaller than the big ai companies are counting on.

vunderba 20d ago
I think we’re in agreement. I make heavy use of smaller local models on a daily basis (Qwen3-VL for auto-captioning images, Gemma3:27b for some translation work, etc.). Gemma3:27b is a good example of a very capable general purpose multimodal model and has handled almost everything I've thrown at it from sentiment analysis to documentation writing. I suppose I was drawing a distinction between specialized and general intelligence versus small and large. I don’t think those are necessarily mutually exclusive.
ac29 19d ago
gpt-oss-120b only has 5B active parameters, so its not surprising Qwen3.8 27B outperforms it (Qwen3.8 is also ~13 months newer, which is forever in LLMs)
- kennywinker 19d ago
  Fair enough. I’ve barley touched oss-120b, so i didn’t know it was so few active params. For a direct comparison, qwen3.6-35b-a3b is still better at coding than oss-120b. And Qwen3.8-27b is still better at coding than opus 4.1. Yes, if you list off models 27b is better than it’s all older models. But that’s my point - newer models are better than older models at the same AND much smaller size. That’s because model size matters less than they say. Training data and model architecture matter more.
- anon373839 19d ago
  No, it’s not the active parameters. Qwen 3.8 Flash has 6B active and it smokes both models.
ericd 19d ago
Gpt-oss—120b is like 1000 years old in AI years, whereas Qwen 3.8 27b is pretty young. What you’re seeing is that parameters aren’t apples to apples, and at a given parameter level, the new models are much, much better than the ones from a year or two ago. Like, to a comical degree.
- MichaelZuo 19d ago
  Wasnt this known by everyone who cared to pay attention? It practically became a joke about how a huge amount of the training data for GPT-4 was bottom of the barrel reddit vomit and obvious bot spam. Leading to many bizarre edge cases.
- kennywinker 19d ago
  Does that not prove my point? Bigger doesn’t automatically mean better. Quality of training data, and model structure, matters as much or more than size
  - ericd 19d ago
    Ah sorry, I should've continued, the bigger recent models are commensurately smarter. If you really want to make the point, then you'd need to show 27b being smarter than similar vintage bigger models. And in that case, there's confounding issues like efficiency, speed due to excessive thinking maybe to make up for the smaller amount of world knowledge baked in (qwen 27b's main issue iirc), etc - they're tuned for different things.
    - kennywinker 19d ago
      https://artificialanalysis.ai/?models=gpt-5-3-codex%2Cqwen3-7-max%2Cqwen3-8-27b%2Ck2-horizon-375b-a23b%2Cgpt-5-5-instant-06-26%2Cclaude-sonnet-5-medium%2Cqwen3-6-plus%2Cdeepseek-v4-flash#artificial-analysis-intelligence-index https://artificialanalysis.ai/?models=gpt-5-3-codex%2Cqwen3-... Shows qwen3.8-27b along side seven larger models of ~similar vintage. Only one scores above 27b. Many of those are closed models so idk their exact parameter count / active param count, but it hardly matters - i’m sure all of them are far above 100b params My point is not that bigger is pointless. It’s just clearly not the only road to take to make a model better, which is obvious just from seeing how models of the same size have gotten better over the past few years

lelanthran 20d ago

Problem is conflict of interest: the studies are mostly from the providers of the biggest models, or someone who received free tokens to do the research.

phoghed 19d ago
It would be nice to hear exactly how the conflict of interest has impacted the specific studies and how they are wrong rather than conspiracy theory level speculation and hand waving at the entire category
- butterNaN 19d ago
  I think is more than reasonable to be suspicious of studies funded by party with conflict of interest. Think of how many studies about climate change were funded by big polluters, for example. In 2026, the default outlook should be suspicion for any big private organisations with profit motive.
  - phoghed 19d ago
    I think it’s less than reasonable to operate mainly on vibes, rumors, and hearsay. You don’t need to think about climate change studies. Instead you can read the allegedly tainted studies we’re actually talking about and profess to all of us what is wrong with them. You can’t point to exactly where they’ve fudged them.
    - zug_zug 19d ago
      In an alternate universe where all research is completely auditable and we all have infinite time, yes that's a valid approach. And you're welcome to spend your life going that route, but the rest of us are gonna trust our common sense on this.
    - kennywinker 19d ago
      They don't even need to fudge the data on the studies they have published. They just need to hold back other studies that contradict the idea. Pouring over the published data looking for flaws will never get us access to the unpublished data.

vmg12 19d ago

> more diverse knowledge tend to cross-pollinate across domains Yeah, the cross domain transfer learning from RL is overstated by a lot.

CuriouslyC 19d ago

The western labs are very AGI pilled, and their public models are distilled down from larger research-only models that are uneconomical to serve directly. They could (and probably will) start distilling models for more niche use cases eventually, but we're not there yet.

RA_Fisher 19d ago

Yes, you're referring to transfer learning. It's an empirical question that also depends on the nature of activities.