4 ms·

Same, I have one workload where on 3.6 drafting 6 tokens is the fastest setting.

• hypfer 2mo ago
  I wonder if we could take a page out of the Solar power book and do MPPT but for draft count. Constantly sweep through possible values to find the best result for the current conditions. Though it might turn out that that doesn't offer any benefits. Has it been tried yet? ___ EDIT: LLM pointed me towards this thing I'm not going to read https://arxiv.org/abs/2512.11280 https://arxiv.org/abs/2512.11280 "AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference" Some higher level nerd please go have a look at that. I have important meetings and golf classes to attend.
  • ndriscoll 2mo ago
    Don't you just need to occasionally try n+1 for your current setting? Then record acceptance rates. The nature of the thing is that if you try n, you already have a good sampling of how well m works for all m<n, right?
    • hedgehog 2mo ago
      This is all a pretty active area of research, both adaptive drafting, different draft approaches (DFlash etc), I expect to see really big improvements land in the popular runtimes this year.
  • petu 2mo ago
    Deepseek's DSpark does dynamically adjust speculated token count per user/completion. https://arxiv.org/abs/2607.05147 https://arxiv.org/abs/2607.05147 But they do so to maximise total throughput, I don't think there's reason to do that for batch=1.