3 ms·

I'm wondering how small of a model can be "generally intelligent" (as in LLM intelligent, not AGI). Like there must be a size too small to hold "all the inform

by timmg 2y ago

I'm wondering how small of a model can be "generally intelligent" (as in LLM intelligent, not AGI). Like there must be a size too small to hold "all the information" in. And I also wonder at what point we'll see specialized small models. Like if I want help coding, it's probably ok if the model doesn't know who directed "Jaws". I suspect that is the future: many small, specialized models. But maybe training compute will just get to the point where we can run a full-featured model on our desktop (or phone)?

idonotknowwhy 2y ago
> Like there must be a size too small to hold "all the information" in. We're already there. If you running a Mistral-Large-2411 and Mistral-Small-2409 locally, you'll find the larger model is able to recall more specific details about works of fiction. And Deepseek-R1 is aware of a lot more. Then you ask one of the Qwen2.5 coding models, and they won't even be aware of it, because they're: > small, specialized models. > But maybe training compute will just get to the point where we can run a full-featured model on our desktop (or phone)? Training time compute won't allow the model to do anything out of distribution. You can test this yourself if you run one of the "R1 Distill" models. Eg. If you run the Qwen R1 distill and ask it about niche fiction, no matter how long you let it <think> for, it can't tell you something the original Qwen didn't know.
- timmg 2y ago
  > We're already there. If you running a Mistral-Large-2411 and Mistral-Small-2409 locally, you'll find the larger model is able to recall more specific details about works of fiction. Oh, for sure. I guess what I'm wondering is if we know the Small model (in this case) is too small -- or if we just haven't figured out how to train well enough? Like, have we hit the limit already -- or, in (say) a year, will the Small model be able to recall everything the Big model does (say, as of today)?
- hnuser123456 2y ago
  I suppose we could eventually get to a super-MoE architecture. Models are limited to 4-16GB in size, but you could have hundreds of models on various topics. Load from storage to RAM and unload as needed. Should be able to load up any 4-16GB model in a few seconds. Maybe as well as a 4GB "Resident LLM" that is always ready to figure out which expert to load.
svachalek 2y ago
It's a sliding scale based on what you consider "generally intelligent" but they're getting smaller and smaller. This 27b model is comparable to 400b models not much over a year ago. But we'll start to see limits on how far that can go, maybe soon. You can try different sizes of gemma3 models, though. The biggest one can answer a lot of things factually, while the smallest one is a hilarious hallucination factory, and the others are different levels in between.
- JKCalhoun 2y ago
  That's interesting. Is there some quantitative way to know that a modern 27b model is equal to an older 400b model?
  - selcuka 2y ago
    https://medium.com/@elmo92/gemma-3-a-27b-multimodal-llm-better-than-really-big-models-b4fe0f4949b4 https://medium.com/@elmo92/gemma-3-a-27b-multimodal-llm-bett... > It comes in sizes from 1 billion to 27 billion parameters (to be precise: 1B, 4B, 12B, 27B), with the 27B version notably competing with much larger models, such as those with 400B or 600B parameters by LLama and DeepSeek.

by timmg 2y ago

idonotknowwhy 2y ago

> Like there must be a size too small to hold "all the information" in. We're already there. If you running a Mistral-Large-2411 and Mistral-Small-2409 locally, you'll find the larger model is able to recall more specific details about works of fiction. And Deepseek-R1 is aware of a lot more. Then you ask one of the Qwen2.5 coding models, and they won't even be aware of it, because they're: > small, specialized models. > But maybe training compute will just get to the point where we can run a full-featured model on our desktop (or phone)? Training time compute won't allow the model to do anything out of distribution. You can test this yourself if you run one of the "R1 Distill" models. Eg. If you run the Qwen R1 distill and ask it about niche fiction, no matter how long you let it <think> for, it can't tell you something the original Qwen didn't know.

timmg 2y ago
> We're already there. If you running a Mistral-Large-2411 and Mistral-Small-2409 locally, you'll find the larger model is able to recall more specific details about works of fiction. Oh, for sure. I guess what I'm wondering is if we know the Small model (in this case) is too small -- or if we just haven't figured out how to train well enough? Like, have we hit the limit already -- or, in (say) a year, will the Small model be able to recall everything the Big model does (say, as of today)?
hnuser123456 2y ago
I suppose we could eventually get to a super-MoE architecture. Models are limited to 4-16GB in size, but you could have hundreds of models on various topics. Load from storage to RAM and unload as needed. Should be able to load up any 4-16GB model in a few seconds. Maybe as well as a 4GB "Resident LLM" that is always ready to figure out which expert to load.

svachalek 2y ago

It's a sliding scale based on what you consider "generally intelligent" but they're getting smaller and smaller. This 27b model is comparable to 400b models not much over a year ago. But we'll start to see limits on how far that can go, maybe soon. You can try different sizes of gemma3 models, though. The biggest one can answer a lot of things factually, while the smallest one is a hilarious hallucination factory, and the others are different levels in between.

JKCalhoun 2y ago
That's interesting. Is there some quantitative way to know that a modern 27b model is equal to an older 400b model?
- selcuka 2y ago
  https://medium.com/@elmo92/gemma-3-a-27b-multimodal-llm-better-than-really-big-models-b4fe0f4949b4 https://medium.com/@elmo92/gemma-3-a-27b-multimodal-llm-bett... > It comes in sizes from 1 billion to 27 billion parameters (to be precise: 1B, 4B, 12B, 27B), with the 27B version notably competing with much larger models, such as those with 400B or 600B parameters by LLama and DeepSeek.