4 ms·

Are we beginning to see "specialized SLMs"? We've already seen some pretend-agent based solutions (where the same model is given several different roles and mad

Are we beginning to see "specialized SLMs"? We've already seen some pretend-agent based solutions (where the same model is given several different roles and made to act as eg. ceo / architect / dev / sales in a startup). I wonder if the way forward is to train smaller models with different sets of "skills" or "neural affinities". One for reasoning, one for summarization, one for math, one for code, etc - then combining them into full-fledged solutions. Perhaps smaller models can be "better" at their specific domains/tasks than the giant generalist models can be at any of them.

worldsayshi 3y ago
Isn't this the whole idea with Mixture Of Experts approach that is GPT-4 is using?
- htrp 3y ago
  Isn't MoE with switch transformers massively inefficiemt compared to being able to customize which LLMs you are using? I've seen a lot of agent swarm concepts in the smaller llm space that seem to provide some feedback that this is a viable avenue of research.
- esafak 3y ago
  Is GPT-4's MOE based on combining specialized models?
hobofan 3y ago
Yes, I think that is the general trend. Have one model tuned for reasoning that decides a plan, based on which you invoke other models as tools (see e.g. the ReWOO paper[0]). If I had to guess, an approach like this is what powers the recent Custom GPT/Assistant API products (based on the lag between tool invocations I would guess that they also re-prompt for plan adjustments between every set of tool calls). Do that with a small model and hot-swap LORAs, and it should be possible to build a quite powerful local assistant on consumer hardware. [0]: https://arxiv.org/abs/2305.18323 https://arxiv.org/abs/2305.18323
trash_cat 3y ago
Yes, this is the trend. OAIs marketplace of GPTs is a confirmation of this. BabyAGI, AutoGen, AutoGPT are all multiple LLM/SLM architectures under the hood. While we don´t have access to proprietary data or the ability to run bigger models, the natural direction is to combine them with specialized tasks like you just described. The issue is then the interface - making it good and communicate seamlessly between models and what roles they + the architecture the models are operating in. The last point is up to your imagination.
imhoguy 3y ago
Specialized LLMs, and likely SLMs too, are really the future. I use them mostly to aid programming and really just stopped paying for GPT-4. Phind and others are really on par now in my coding needs.

worldsayshi 3y ago

Isn't this the whole idea with Mixture Of Experts approach that is GPT-4 is using?

htrp 3y ago
Isn't MoE with switch transformers massively inefficiemt compared to being able to customize which LLMs you are using? I've seen a lot of agent swarm concepts in the smaller llm space that seem to provide some feedback that this is a viable avenue of research.
esafak 3y ago
Is GPT-4's MOE based on combining specialized models?

hobofan 3y ago

Yes, I think that is the general trend. Have one model tuned for reasoning that decides a plan, based on which you invoke other models as tools (see e.g. the ReWOO paper[0]). If I had to guess, an approach like this is what powers the recent Custom GPT/Assistant API products (based on the lag between tool invocations I would guess that they also re-prompt for plan adjustments between every set of tool calls). Do that with a small model and hot-swap LORAs, and it should be possible to build a quite powerful local assistant on consumer hardware. [0]: https://arxiv.org/abs/2305.18323 https://arxiv.org/abs/2305.18323

trash_cat 3y ago

Yes, this is the trend. OAIs marketplace of GPTs is a confirmation of this. BabyAGI, AutoGen, AutoGPT are all multiple LLM/SLM architectures under the hood. While we don´t have access to proprietary data or the ability to run bigger models, the natural direction is to combine them with specialized tasks like you just described. The issue is then the interface - making it good and communicate seamlessly between models and what roles they + the architecture the models are operating in. The last point is up to your imagination.

imhoguy 3y ago

Specialized LLMs, and likely SLMs too, are really the future. I use them mostly to aid programming and really just stopped paying for GPT-4. Phind and others are really on par now in my coding needs.