3 ms·
One big one is speculative decoding for larger models, the more “accurate”* the smaller model, the more the speed up on the bigger model * as in matches the to
by kirill5pol 3y ago
One big one is speculative decoding for larger models, the more “accurate”* the smaller model, the more the speed up on the bigger model
* as in matches the token that the larger model would output
- sroussey 3y agoIt would be better if they also had a 70B model for this. They would need the same vocabulary, etc. What else?