3 ms·
DFlash 2: Keep Drafting Parallel
- verdverm 1mo agovllm PR for DFlash2: https://github.com/vllm-project/vllm/pull/52816 https://github.com/vllm-project/vllm/pull/52816
- cogman10 1mo agollama.cpp PR https://github.com/ggml-org/llama.cpp/pull/27342 https://github.com/ggml-org/llama.cpp/pull/27342
- adefa 1mo agoI'm getting around 27 tokens per second decode using vLLM + Qwen 3.8 27b nvfp4 + DFlash 2 on the DGX Spark.
- pdntspa 1mo agoI am working this out with Fable right now, for getting this running on my DGX spark homelab; it mentioned that there might be issues with the 'optimized LM-head restrictions' that unsloth NVFP4 ships with. Have you had any issues here? Are you trying this with vLLM? Or a different engine? I am getting about 15 tok/s on my spark on my current setup using the 0.26 nvidia vLLM image and MTP.
- adefa 1mo agoIf you are using the Unsloth nvfp4 checkpoint, you need to patch vLLM+DFlash 2 to accept the quant's FP8 `lm_head`.
- hypfer 1mo agoAmazing tech > An agent writes in an afternoon what a chatbot writes in a month But can you just.. not. Your tech is so good, it speaks for itself. Don't ruin that.
- sarjann 1mo agoGreat news, has made low memory bandwidth model usage so much nicer.
- ilc 1mo agoWatch the video carefully. DFlash2's tool call fails on python syntax. Usually models in this class nail things like that 1 shot, which the other side did. I don't know the cause. It may be nothing. But I'd like to see the model doing something where its path is a bit more constrained, to help out rule out such oddities.
- zackangelo 1mo agoDFlash is lossless so this would be a bug in the implementation if it is indeed a regression against the target model.
- liuliu 1mo agoOnly if you do greedy sampling. With probabilisitic sampling (categorical sampling), you will end up with different trajectory just “mathematically equivalent”.
- stymaar 1mo agoCan you explain a little bit more please?
- stkdump 1mo agoNot OP, but you can influence how deterministic your LLM behaves using the temperature setting. The neural network doesn't directly output tokens, but logits which are then converted to probabilities and then a token is chosen at random, unless the temperature is 0 (i.e. greedy, we just always pick the most probable token without any randomness). All speculative decoding methods have to "commit" to a token though even when they don't know the actual logits of the full size NN yet. The question then is (and I don't know the answer): how do the common inference engines behave when the speculation landed on the most probable token, but the random choice still doesn't land on it? You can imagine that in the interest of performance as long as we stay reasonably inside the probability we just go ahead with the speculation. Not sure if thats implemented like that though. Edit: I just looked up the math, and actually the idea of speculative decoding is done in a clever way that fully preserves the probability distribution while still maximizing the acceptance rate of draft tokens. So I would have to disagree with OP and say that no, non-greedy sampling doesn't influence the trajectories.