3 ms·
Thank you, very interesting to read. I haven't considered batch sizes at all up to now. Seems I should look into it. How did you get the prediction that good?
by aktenlage 29d ago
Thank you, very interesting to read. I haven't considered batch sizes at all up to now. Seems I should look into it.
How did you get the prediction that good? The default spec-draft-max-n is 3 and if I go beyond, the output rate stays the same or gets lower any magic spell you could share?
- aktenlage 27d agoWow, setting the "ubatch" size to 2048 (default is 512), like the "batch" size roughly doubled prompt processing speed.
- tharkun__ 25d agoRegarding the MTP, if you haven't found out why yet, the `--spec-draft-p-min 0.8`. Something to play with and of course it's non-deterministic and depends on both the model itself and whatever you're actually throwing at it. This is what I read before I went to try it: https://github.com/ggml-org/llama.cpp/discussions/25198 https://github.com/ggml-org/llama.cpp/discussions/25198