3 ms·

Can someone explain to me please why Meta doesn't create subject specific versions of their LLMs such as one that knows only about computer programming, compute

Can someone explain to me please why Meta doesn't create subject specific versions of their LLMs such as one that knows only about computer programming, computers, hardware software. I would have imagined such a thing would be smaller and thus run on smaller configurations. But since I am only a layman maybe someone can tell me why this isn't the case?

KTibow 1y ago
Other companies have done this (see Qwen Coder). It doesn't scale past a few disciplines like math and code though, and using mixtures of experts give you most of the same benefits.
- littlestymaar 1y ago
  Unlike what their names imply, MoE don't have a concept of domain expert at all. It's just a fancy name for sparse evaluation of the total network to save compute and memory bandwidth.
cube2222 1y ago
Generally, all that non-tech content still helps the model “to learn”. Also, the software you’re working on will generally in some way have a real-world domain - without knowing it the AI all likely be a less effective assistant. Design conversations with it would likely be pretty non-fun, too. Finally, the “bitter lesson” article[0] from a couple years ago is I think somewhat applicable too. [0]: http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html
bbatha 1y ago
To add on to the sibling. Specialized models, including fine tuned ones, continually have their lunch eaten by general models within 3-6 months. This time round is mixture of experts that’ll do it, next year it’ll be something else. Tuned models are expensive to produce and are benchmark kings but less do less well in the real world qualitative experience. The juice just ain’t worth the squeeze most of the time. Meta does have some specialized models though, llamaguard was released for llama 2 and 3.
- littlestymaar 1y ago
  > Tuned models are expensive to produce The expensive part is building the dataset, training itself isn't too expensive (you can even fine-tune small models on free colab instances!), and when you have your dataset, you can just fine tune the next generalist model as soon as it's released and you're good to go ago.
ntonozzi 1y ago
One of the weirdest and most interesting parts of LLMs is that they grow more effective the more languages and disciplines they are trained in. It turns out training LLMs on code instead of just prose boosted their intelligence and reasoning capabilities by huge amounts.
- gardenhedge 1y ago
  Source? Sounds interesting
  - CelticBard 1y ago
    There are a lot of papers that stumble upon this while working with LLMs and its mentioned off hand in the conclusions. If you want a paper that explicitly tested this here you go - https://aclanthology.org/2022.coling-1.437.pdf https://aclanthology.org/2022.coling-1.437.pdf

KTibow 1y ago

Other companies have done this (see Qwen Coder). It doesn't scale past a few disciplines like math and code though, and using mixtures of experts give you most of the same benefits.

littlestymaar 1y ago
Unlike what their names imply, MoE don't have a concept of domain expert at all. It's just a fancy name for sparse evaluation of the total network to save compute and memory bandwidth.

cube2222 1y ago

Generally, all that non-tech content still helps the model “to learn”. Also, the software you’re working on will generally in some way have a real-world domain - without knowing it the AI all likely be a less effective assistant. Design conversations with it would likely be pretty non-fun, too. Finally, the “bitter lesson” article[0] from a couple years ago is I think somewhat applicable too. [0]: http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html

bbatha 1y ago

To add on to the sibling. Specialized models, including fine tuned ones, continually have their lunch eaten by general models within 3-6 months. This time round is mixture of experts that’ll do it, next year it’ll be something else. Tuned models are expensive to produce and are benchmark kings but less do less well in the real world qualitative experience. The juice just ain’t worth the squeeze most of the time. Meta does have some specialized models though, llamaguard was released for llama 2 and 3.

littlestymaar 1y ago
> Tuned models are expensive to produce The expensive part is building the dataset, training itself isn't too expensive (you can even fine-tune small models on free colab instances!), and when you have your dataset, you can just fine tune the next generalist model as soon as it's released and you're good to go ago.

ntonozzi 1y ago

One of the weirdest and most interesting parts of LLMs is that they grow more effective the more languages and disciplines they are trained in. It turns out training LLMs on code instead of just prose boosted their intelligence and reasoning capabilities by huge amounts.

gardenhedge 1y ago
Source? Sounds interesting
- CelticBard 1y ago
  There are a lot of papers that stumble upon this while working with LLMs and its mentioned off hand in the conclusions. If you want a paper that explicitly tested this here you go - https://aclanthology.org/2022.coling-1.437.pdf https://aclanthology.org/2022.coling-1.437.pdf