3 ms·
I got it working on my laptop! M2 64GB and needs 38GB of disk space: uv run --with 'numpy<2.0' --with mlx-vlm python \ -m mlx_vlm.generate \ --mode
by simonw 2y ago
I got it working on my laptop! M2 64GB and needs 38GB of disk space:
uv run --with 'numpy<2.0' --with mlx-vlm python \
-m mlx_vlm.generate \
--model mlx-community/QVQ-72B-Preview-4bit \
--max-tokens 10000 \
--temp 0.0 \
--prompt "describe this" \
--image pelicans-on-bicycles-veo2.jpg
Output of that command here: https://simonwillison.net/2024/Dec/24/qvq/#with-mlx-vlm https://simonwillison.net/2024/Dec/24/qvq/#with-mlx-vlm
- benreesman 2y ago+1 for also teaching me more cool uv flags.
- tkgally 2y agoIs the locally run model censored in the same way as the model hosted on HuggingFace? If so, I wonder how that censorship is baked into the weights—particularly the errors thrown when it starts to output specific strings.
- deleted 2y ago[deleted]
- simonw 2y agoOK, this surprised me: uv run --with 'numpy<2.0' --with mlx-vlm python \ -m mlx_vlm.generate \ --model mlx-community/QVQ-72B-Preview-4bit \ --max-tokens 10000 \ --temp 0.0 \ --prompt "describe this" \ --image Tank_Man_\(Tiananmen_Square_protester\).jpg Here's the output: https://gist.github.com/simonw/e04e4fdade0c380ec5dd1e90fb5f3b96 https://gist.github.com/simonw/e04e4fdade0c380ec5dd1e90fb5f3... It included this bit: > I remember seeing this image before; it's famous for capturing a moment of civil resistance. The person standing alone against the tanks symbolizes courage and defiance in the face of overwhelming power. It's a powerful visual statement about the human spirit and the desire for freedom or protest.
- tkgally 2y agoThanks for testing that! (And, while I’m at it, thank you for your blog and for all of your contributions to HN as well. I am constantly learning from you.) I wonder if the response was not truncated this time only because the word “Tiananmen” did not happen to appear in the response. At least some of the censorship of Chinese models seems to be triggered by specific strings in the output.
- int_19h 2y agoBoth Qwen and QwQ are generally pretty easy to get talking about Tiananmen etc, so this isn't exactly new.
- deleted 2y ago[deleted]
- dostick 2y agoI wonder if it knows about Winnie the Pooh, or that character was wiped and don’t exist for it?
- mkagenius 2y agoBeing able to run STOA models on my mac has been so cool. I am using it to let molmo vision via `mlx-vlm` click on my phone[1]. Although its 4bit quantised, it works surprisingly well. 1. Controlling phone using local LLMs - https://github.com/BandarLabs/clickclickclick https://github.com/BandarLabs/clickclickclick
- talldayo 2y agoTo be fair, if you pay Apple's prices for RAM upgrades then your computer better come with every bell and whistle in the book. The fact that Apple gets away with compute shaders instead of full-fat OpenCL compliance is a bit of a testament to how little they care about GPU compute.
- brianjking 2y agoDamn, I'm struggling to run this on a M4 Pro with 48gb.
- bt1a 2y agosudo sysctl iogpu.wired_limit_mb=45000