2 ms·
Waiting for the MTP version to pop up on Unsloth. Speculative decoding makes a huge difference. Been running quantized 3.6 at 110t/s on a cheap 5060Ti and quit
by gaigalas 2mo ago
Waiting for the MTP version to pop up on Unsloth. Speculative decoding makes a huge difference.
Been running quantized 3.6 at 110t/s on a cheap 5060Ti and quite happy with it. If 3.8 improves on it, it would be awesome.
- zazibar 2mo agoGood news, MTP support is already included in this release. Not sure why they haven't made this clearer.
- gaigalas 2mo agoI don't see an MTP entry on Unsloth though. Maybe it's not available in a lower quant I need for my poor GPU.
- greenicon 2mo agoIt's included in the model gguf itself.
- gaigalas 2mo agoMaybe I'm not seeing the speed improvements because it's a dense model, and I'm comparing it to the 3.6 MoE with MTP. Likely, I'm attributing the speed I see to something else. Honestly, if MTP is available, I don't know why it's so much slower here.