2 ms·
Here are tips for running it on macOS using MLX: https://twitter.com/awnihannun/status/1883611098081099914 https://twitter.com/awnihannun/status/188361109808109
by simonw 2y ago
Here are tips for running it on macOS using MLX: https://twitter.com/awnihannun/status/1883611098081099914 https://twitter.com/awnihannun/status/1883611098081099914 - using https://huggingface.co/mlx-community/Qwen2.5-7B-Instruct-1M-4bit https://huggingface.co/mlx-community/Qwen2.5-7B-Instruct-1M-...
- woadwarrior01 2y agoMLX does not support dual chunk attention[1] that these models use for long contexts, yet. [1]: https://arxiv.org/abs/2402.17463 https://arxiv.org/abs/2402.17463