4 ms·
related: I imagine in the future we might several "expert" LLM's and a wrapper can delegate tasks as needed as if it were a "tool". That way we can have segrega
by simianwords 2y ago
related: I imagine in the future we might several "expert" LLM's and a wrapper can delegate tasks as needed as if it were a "tool". That way we can have segregation of expertise - each individual model can excel at one single thing.
A prover model might be used as a tool in the coming future.
- koakuma-chan 2y ago> related: I imagine in the future we might several "expert" LLM's and a wrapper can delegate tasks as needed as if it were a "tool". That way we can have segregation of expertise - each individual model can excel at one single thing. In the future? I'm pretty sure people do that already.
- energy123 2y agoIt's crudely done though.
- kratom_sandwich 2y agoMistrals model is a mixture-of-experts model
- simianwords 2y agoNo I disagree. I would want ChatGPT to abstract away expert models - biochemistry model, coding model, physics model and maybe O3 would use these models as tools to come up with an answer. The point being that a separate expert model would be better at its own field than a single model that tries to be good at everything. Intuitively it makes sense, in practice I have seen anecdotes where finetuning a small model on domain data makes the model lose coherence on other topics.
- koakuma-chan 2y ago> have seen anecdotes where finetuning a small model on domain data makes the model lose coherence on other topics This is expected behaviour.
- simianwords 2y agoi know. so why don't we have domain specific models as tools in consumer llm products
- deleted 2y ago[deleted]
- Arcuru 2y agoFor a concrete example today, see https://openrouter.ai/openrouter/auto https://openrouter.ai/openrouter/auto
- simianwords 2y agothats nice but imagine first having models that are expert in specific domains. routing seems to be the easy part (just feed the available models as tools to your wrapper LLM)
- someguy101010 2y agoThe No Free Lunch Theorem implies that something like this is inevitable https://en.wikipedia.org/wiki/No_free_lunch_in_search_and_optimization https://en.wikipedia.org/wiki/No_free_lunch_in_search_and_op...
- repsilat 2y agoA system of n experts is no different to a single expert wrt the NFLT. The theorem is entirely indifferent to (ie "equally skeptical of") the idea.
- samvaran 2y agoIs that not what MoE models already do?
- oofbaroomf 2y agoNo. Each expert is not separately trained, and while they may store different concepts, they are not meant to be different experts in specific domains. However, there are certain technologies to route requests to different domain expert LLMs or even fine-tuning adapters, such as RouteLLM.
- retinaros 2y agoThat might already happen behind what they call test time compute
- oofbaroomf 2y agoMany models that use test time compute are MoEs, but test-time compute is generally meant to refer to reasoning about the prompt/problem the model is given, not about reasoning about which model to pick, and I don't think anyone has released an LLM router under that name.
- retinaros 2y agowe dont know what OAI does to find the best answer when reasoning but I am pretty sure that having variations of a same model is part of it.
- woah 2y agoWhy do you think that a hand-configured selection between "different domains" is better than the training-based approach in MoE?
- oofbaroomf 2y agoFirst off, they are basically completely different technologies, so it would be disingenuous to act like it's an apples-to-apples comparison. But a simple way to see it is that when you pick between multiple large models that have different strengths, you have a larger amount of parameters just to work with (e.g. Deepseek R1 + V3 + Qwen + LLaMA ends up being 2 trillion total parameters to pick from), whereas "picking" the experts in an MoE like has a smaller amount of total different parameters you are working with (e.g. R1 is 671 billion, Qwen is 235).