5 ms·
I don't see an MTP entry on Unsloth though. Maybe it's not available in a lower quant I need for my poor GPU.
by gaigalas 2mo ago
I don't see an MTP entry on Unsloth though. Maybe it's not available in a lower quant I need for my poor GPU.
- greenicon 2mo agoIt's included in the model gguf itself.
- gaigalas 2mo agoMaybe I'm not seeing the speed improvements because it's a dense model, and I'm comparing it to the 3.6 MoE with MTP. Likely, I'm attributing the speed I see to something else. Honestly, if MTP is available, I don't know why it's so much slower here.