There is also speculative sampling - you decode a few tokens with a smaller model, then use the big model to validate them simultaneously. The big model might t

There is also speculative sampling - you decode a few tokens with a smaller model, then use the big model to validate them simultaneously. The big model might trim the prediction up to a point and add an extra token. Then cycle again with the small model -> 2-2.5x speedup