4 ms·
Couldn't someone build an adaptive system, where the llm is frequently judging the difficulty of a task and switching to a more/less powerful model?
by xyzsparetimexyz 3mo ago
Couldn't someone build an adaptive system, where the llm is frequently judging the difficulty of a task and switching to a more/less powerful model?
- bkotrys 3mo agodefinitely something that will come sooner than later. In my view, adaption of model routing / model orchestration and mixing providers is just natural next step.
- iamflimflam1 3mo agoThis only works well if you are creating sub agents with clean contexts for each task. If you constantly are switching models part way through some work then the whole session needs to be replayed each switch. You lose all the benefits of the context cache.
- xyzsparetimexyz 3mo agoWell yeah, it'd require the models to be loaded on the same system and the cache to be shared between them somehow
- inigyou 3mo agoCache sharing is not possible. The numbers in the cache are completely specific to the model.
- xyzsparetimexyz 3mo agoCurrently. I'm sure that you could make a system where the cache values are a superset C of e.g. models A and B where C is probably bigger than max(A,B) but smaller than A+B
- inigyou 3mo agoIt's equal to A+B. There is literally no sharing possible.
- xyzsparetimexyz 3mo agoEven when training both models together in a novel way?
- wonnage 3mo agoReddit post so take with a grain of salt but this does seem possible. But unclear whether this is actually a viable architecture https://www.reddit.com/r/LocalLLaMA/comments/1t8s83r/nvidia_ai_releases_star_elastic_one_checkpoint/ https://www.reddit.com/r/LocalLLaMA/comments/1t8s83r/nvidia_...
- deleted 3mo ago[deleted]
- bob1029 3mo agoThe judgment of task difficulty is a cursed problem. In most domains you really don't know what is down there until you start digging a little bit.
- xyzsparetimexyz 3mo agoWell yeah. You start on as medium model, it does the digging says says 'this task is too easy/hard/just right for me' and go from there.
- zombot 3mo agoAnd while you're at it, solve the halting problem.