3 ms·
Switching to mlx-vlm is basically harmful since it (like vllm and sglang) have such garbage support for modern samplers. To be clear, I am one of the authors on
by Der_Einzige 3mo ago
Switching to mlx-vlm is basically harmful since it (like vllm and sglang) have such garbage support for modern samplers. To be clear, I am one of the authors on the min_p paper, and if min_p is the best you have (when llamacpp supports the far superior top-n-sigma), than I have no reason to switch even if you are somehow faster.
(https://arxiv.org/abs/2411.07641 https://arxiv.org/abs/2411.07641)
And if you do care to support modern samplers, you can start with the following:
1. https://arxiv.org/abs/2509.23234 https://arxiv.org/abs/2509.23234
2. https://arxiv.org/abs/2509.02510 https://arxiv.org/abs/2509.02510
3. https://arxiv.org/abs/2604.11012 https://arxiv.org/abs/2604.11012
- woadwarrior01 3mo agoThanks for bringing this up. You're 100% right. But most people, even technical ones are oblivious to how much of a difference modern samplers and higher quality quantization algorithms make for on-device LLM inference and are stuck with good old top-p, top-k samplers and RTN quantization. TBF mlx-vlm does support min-p sampling, but none of the other modern samplers that you list. Ollama and LM Studio are even worse with only top-p and top-k samplers.