6 ms·
I have always been bothered with the multinomial layer that picks the next token. It seems like you should be able to stick a classifier in there to detect if t
by mathiasgredal 3y ago
I have always been bothered with the multinomial layer that picks the next token. It seems like you should be able to stick a classifier in there to detect if the probabilities align with a satisfactory answer and whether the model is unsure of the response. If that is the case, then it should branch out to a bigger model which would compute that single token that it is unsure about, and from that the rest of the response is obvious for the smaller model.
This way you could combine e.g. a 7B parameter model with a 70B parameter model, and get the quality of the larger model while most of the time you are only running the small model.
Edit: You could also store the full probabilities for each token, and then the classifier could detect if it had gone down a bad path, and then unwind the tokens and pick a different path.
- jacquesm 3y agoThat's reminiscent of a caching scheme, you could probably layer that down to even smaller and up to even larger models for increased performance and accuracy. Better yet: have many smaller models that can when confused call upon larger models and the larger models could then pick the most appropriate smaller 'expert' model or, alternatively themselves escalate. Sort of a supervisor tree for language models.
- photoGrant 3y agoI want to say this is where the progress is being made currently. Not only having 'experts per domain' but 'experts per pairing'
- briancleland 3y agoSounds like speculative decoding. https://huggingface.co/blog/vivien/optimal-lossy-variant-of-speculative-decoding https://huggingface.co/blog/vivien/optimal-lossy-variant-of-...
- soulofmischief 3y agoContextual loading of a bigger model would incur a lot of latency. In this case it might be a good idea to farm out the request to a non-local model. Could be a good avenue for a hybrid local & remote completion system.
- pyinstallwoes 3y agoIs there anything experimenting with that?
- Roark66 3y agoI would be very surprised if (not at all)OpenAI didn't. I've noticed it with both gpt 3.5 and Gpt4 that if you ask it a difficult question the latency increases a lot, especially at the beginning of the answer. I wouldn't ve surprised if they did exactly what was described.
- pyinstallwoes 3y agoRight I’ve noticed that too. It’s like it takes a pause and reevals. Definitely feels like a cache or memoization of token paths. Interesting.