2 ms·
Speculative decoding requires you to load the smaller model into memory and run inference on it.
by Palmik 2y ago
Speculative decoding requires you to load the smaller model into memory and run inference on it.
- renonce 2y agoI think the smaller model is at least 20 times smaller. If you do speculative decoding on a 70B model an 1B model would be appropriate.