10 ms·
Qwen2.5-VL-32B: Smarter and Lighter
- jauntywundrkind 2y agoWish I knew better how to estimate what sized video card one needs. HuggingFace link says this is bfloat16, so at least 64GB? I guess the -7B might run on my 16GB AMD card?
- wgd 2y agoYou can run 4-bit quantized version at a small (though nonzero) cost to output quality, so you would only need 16GB for that. Also it's entirely possible to run a model that doesn't fit in available GPU memory, it will just be slower.
- xiphias2 2y agoI wish they would start producing graphs with quantized version performances as well. What matters is RAM/bandwidth vs performance, not number of parameters.
- clear_view 2y agodeepseek-r1:14b/mistral-small:24b/qwen2.5-coder:14b fit 16GB VRAM with fast generation. 32b versions bleed into RAM and take a serious performance hit but still usable.
- zamadatix 2y agohttps://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator https://huggingface.co/spaces/NyxKrage/LLM-Model-VRAM-Calcul... That will help you quickly calculate the model VRAM usage as well as the VRAM usage of the context length you want to use. You can put "Qwen/Qwen2.5-VL-32B-Instruct" in the "Model (unquantized)" field. Funnily enough the calculator lacks the option to see without quantizing the model, usually because nobody worried about VRAM bothers running >8 bit quants.
- azinman2 2y agoExcept when it comes to deepseek
- zamadatix 2y agoFor others not as familiar, this is pointing out DeepSeek-v3/DeepSeek-R1 are natively FP8 so selecting "Q8_0" aligns with not selecting quantization for that model (though you'll need ~1 TB of memory to use these model unquantized at full context). Importantly, this does not apply to the "DeepSeek" distills of other models, which retain natively being the same as the base model they distill. I expect more and more worthwhile models to natively have <16 bit weights as time goes on but for the moment it's pretty much "8 bit DeepSeek and some research/testing models of various parameter width".
- azinman2 2y agoI wish deepseek distills were somehow branded differently. The amount of confusion I’ve come across from otherwise technical folk, or simply mislabeling (I’m running r1 on my MacBook!) is shocking. It’s my new pet peeve.
- simonw 2y agoBig day for open source Chinese model releases - DeepSeek-v3-0324 came out today too, an updated version of DeepSeek v3 now under an MIT license (previously it was a custom DeepSeek license). https://simonwillison.net/2025/Mar/24/deepseek/ https://simonwillison.net/2025/Mar/24/deepseek/
- chaosprint 2y agoit seems that this free version "may use your prompts and completions to train new models" https://openrouter.ai/deepseek/deepseek-chat-v3-0324:free https://openrouter.ai/deepseek/deepseek-chat-v3-0324:free do you think this needs attention?
- deleted 2y ago[deleted]
- wgd 2y agoThat's typical of the free options on OpenRouter, if you don't want your inputs used for training you use the paid one: https://openrouter.ai/deepseek/deepseek-chat-v3-0324 https://openrouter.ai/deepseek/deepseek-chat-v3-0324
- overfeed 2y agoIs OpenRouter planning on distilling models off the prompts and responses from frontier models? That's smart - a little gross - but smart.
- numlocked 2y agoCOO of OpenRouter here. We are simply stating the WE can’t vouch for the behavior of the upstream provider’s retention and training policy. We don’t save your prompt data, regardless of the model you use, unless you explicitly opt-in to logging (in exchange for a 1% inference discount).
- 2y ago
- simonw 2y ago32B is one of my favourite model sizes at this point - large enough to be extremely capable (generally equivalent to GPT-4 March 2023 level performance, which is when LLMs first got really useful) but small enough you can run them on a single GPU or a reasonably well specced Mac laptop (32GB or more).
- clear_view 2y ago32B don't fully fit 16GB of VRAM. Still fine for higher quality answers, worth the extra wait in some cases.
- abraxas 2y agoWould a 40GB A6000 fully accommodate a 32B model? I assume an fp16 quantization is still necessary?
- manmal 2y agoAt FP16 you‘d need 64GB just for the weights, and it‘d be 2x as slow as a Q8 version, likely with little improvement. You‘ll also need space for attention and context etc, so 80-100GB (or even more) VRAM would be better. Many people „just“ use 4x consumer GPUs like the 3090 (24GB each) which scales well. They’d probably buy a mining rig, EPYC CPU, Mainboard with sufficient PCIe lanes, PCIe risers, 1600W PSU (might need to limit the GPUs to 300W), and 128GB RAM. Depending what you pay for the GPUs that‘ll be 3.5-4.5k
- postalrat 2y agoI haven't found a good case/risers/etc I really like. Most the miner stuff wasn't made for PCIe 16x.
- manmal 2y agoIs that a problem? According to this, the GPUs don’t communicate that much once the weights are loaded: https://github.com/turboderp/exllama/discussions/16#discussioncomment-6245573 https://github.com/turboderp/exllama/discussions/16#discussi... > So at FP16 precision that's a grand total of 16 kB you're transmitting over the PCIe bus, once per token. If you multiply by, say, 20 tokens per second, then you're still only using like 0.1% of your PCIe bandwidth. Intra GPU memory bandwidth is very important, but I‘ve seen lots of people use just a x4 lane and they didn’t complain much.
- gatienboquet 2y agoSo today is Qwen. Tomorrow a new SOTA model from Google apparently, R2 next week. We haven't hit the wall yet.
- behnamoh 2y agoGoogle's announcements are mostly vaporware anyway. Btw, where is Gemini Ultra 1? how about Gemini Ultra 2?
- karmasimida 2y agoIt is already on the LLM arena right, codename nebula? But you are right they can fuck up their releases royally.
- aoeusnth1 2y agoI guess they don’t do ultras anymore, but where was the announcement for it? What other announcement was vaporware?
- zamadatix 2y agoQwen 3 is coming imminently as well https://github.com/huggingface/transformers/pull/36878 https://github.com/huggingface/transformers/pull/36878 and it feels like Llama 4 should be coming in the next month or so. That said none of the recent string of releases has done much yet to "smash a wall", they've just met the larger proprietary models where they already were. I'm hoping R2 or the like really changes that by showing ChatGPT 3->3.5 or 3.5->4 level generational jumps are still possible beyond the current state of the art, not just beyond current models of a given size.
- YetAnotherNick 2y ago> met the larger proprietary models where they already were This is smashing the wall. Also if you just care about breaking absolute numbers, OpenAI released 4.5 a month back which is SOTA in base model, planning to release O3 full in maybe a month, and Deepseek released new V3 which is again SOTA in many aspects.
- Arcuru 2y agoDoes anyone know how making the models multimodal impacts their text capabilities? The article is claiming this achieves good performance on pure text as well, but I'm curious if there is any analysis on how much impact it usually has. I've seen some people claim it should make the models better at text, but I find that a little difficult to believe without data.
- netdur 2y agoMy understanding is that in multimodal models, both text and image vectors align to the same semantic space, this alignment seems to be the main difference from text-only models."
- kmacdough 2y agoI am having a hard time finding controlled testing, but the premise is straightforward: different modalities encourage different skills and understandings. Text builds up more formal idea tokenization and strengthens logic/reasoning while images require it learns a more robust geometric intuition. Since these learnings are applied to the same latent space, the strengths can be cross-applied. The same applies to humans. Imagine a human who's only life involved reading books in a dark room, vs one who could see images vs one who can actually interact.
- ItDoBeWimdyTho 2y agoThat comparison actually makes human reasoning abilities more impressive. Helen Keller still learned robust generalizations.
- lysace 2y agoTo clarify: Qwen is made by Alibaba Cloud. (It's not mentioned anywhere in the blog post.)
- nodesocket 2y agoWhat is the recommended model to process a RAG of PDF text documents? I've seen some recommendations for Mistral:7b. Looking to run on a consumer pedestrian home PC (ollama) with a Nvidia 4060ti and Ryzen 5700x.
- unquietwiki 2y agoApparently there are two versions of the 4060Ti, with 8GB and 16GB of VRAM respectively. I've got an 8GB 3060 that runs gemma2:9b nicely, and that will parse PDF files; gemma3:4b also seems to analyze PDFs decently.
- nodesocket 2y agoI also have the 8GB 4060ti variant. Want to upgrade to a 4070 super, but prices on them are still ridiculous. Could be had for $599 a handful of months ago, now on ebay going for $750 plus. Thanks for the recommendations. I'll give gemma3:12b a try and if needed go down to gemma:4b.
- manmal 2y agoMay I ask why you don’t get a used 3090 with 24GB VRAM?
- nodesocket 2y agoThe rig also is my gaming PC, so want the latest generation chips.
- hmottestad 2y agoJust don’t ask it about the tiananmen square massacre or you’ll get a security warning. Even if you rephrase it. It’ll happily talk about Bloody Sunday. Probably a great model, but it worries me that it has such restrictions. Sure OpenAI also has lots of restrictions, but this feels more like straight up censorship since it’ll happily go on about bad things the governments of the west have done.
- generalizations 2y agoNah, it's great for things that Western models are censored on. The True Hacker will keep an Eastern and Western model available, depending on what they need information on.
- hmottestad 2y agoI tried to ask it about Java exploits that would allow me to gain RCE, but it refused just as most western models do. That was the only thing I could think to ask really. Do you have a better example maybe?
- OkGoDoIt 2y agoAdult content and things like making biological/chemical/nuclear weapons are the other main topics that usually get censored. I don’t think the Chinese models tend to be less censored than western models in these dimensions. You can sometimes find “uncensored“ models on HuggingFace where people basically finetune sensitive topics back in. There is a finetuned version of R1 called 1776 that will correctly answer Chinese-censored questions, for example.
- theturtletalks 2y agoWouldn’t they just run R1 locally and not have any censorship at all? The model isn’t censored at its core, it’s censored through the system prompt. Perplexity and Huggingface have their own versions of R1 that is not censored.
- 2y ago
- 101008 2y agoSilly question: how can OpenAI, Claude and all, have a valuation so large considering all the open source models? Not saying they will disappear or be tiny (closed models), but why so so so valuable?
- barbarr 2y agoIt's user base and brand. Just like with Pepsi and Coca Cola. There's a reason OpenAI ran a Super Bowl ad.
- 101008 2y agoYeah but cheaper alternatives (and open source and local ones) it would be super easy for most of the customers to migrate to a different provider. I am not saying they don't provide any value, but it's like paid software vs open source alternative. Open source alternative ends up imposing, especially among tech people.
- TechDebtDevin 2y agoMost "normies" I know only recognize ChatGPT with AI, so for sure, brand recognition is the only thing that matters.
- seydor 2y agoPeople cannot normally invest in their competitors. It's not unlikely that chinese products may be banned / tarriff'd
- FreakyT 2y agoThere are non-Chinese open LLMs (Mistral, LLama, etc), so I don't think that explains it.
- neither_color 2y agoChatGPT's o1 is still really good and the free options are not compelling enough to switch if you've been using it for a while. They've positioned themselves to be a good mainstream default.
- admiralrohan 2y agoAny update on the privacy concerns?
- ggregoire 2y agoWe were using Llama vision 3.2 a few months back and were very frustrated with it (both in term of speed and results quality). Some day we were looking for alternatives on Hugging Face and eventually stumbled upon Qwen. The difference in accuracy and speed absolutely blew our mind. We ask it to find something in an image and we get a response in like half a second with a 4090 and it's most of the time correct. What's even more mind blowing is that when we ask it to extract any entity name from the image, and the entity name is truncated, it gives us the complete name without even having to ask for it (e.g. "Coca-C" is barely visible in the background, it will return "Coca-Cola" on its own). And it does it with entities not as well known as Coca-Cola, and with entities only known in some very specific regions too. Haven't looked back to Llama or any other vision models since we tried Qwen.
- Alifatisk 2y agoEver since I switched to Qwen as my go to, it's been a bliss. They have a model for many (if not all) cases. No more daily quota! And you get to use their massive context window (1M tokens).
- Hugsun 2y agoHow are you using them? Who is enforcing the daily quota?
- Alifatisk 2y agoI use them through chat.qwenlm.ai, what's nice is that you can run your prompt through 3 different modes in parallel to see which suits the best for that case. The daily quota I spoke about is chatgpt and claude, those are very limited on the usage (for free users at least, understandable), while on Qwen, I have felt likeI am abusing it with how much I use it. It's very versatile in the sense that it has capabilities like image generation, video generation, massive context window, both visual and textual reasoning all in one place. Alibaba is really doing something amazing here.
- exe34 2y ago
- tomcam 2y agoI wish I were smarter and lighter tbh I’d settle for just lighter
- wg0 2y agoFor now, motivation is to win the AI race. Not sure how or who would keep these models updated in the future. I mean all of them.
- i_love_retros 2y agoAny security risks running these Chinese LLMs on my local computer?
- TrueDuality 2y agoAlways a possibility with custom runtimes, but the weights alone do not pose any form of malicious code risk. The asterisk there is allowing them to run arbitrary commands on your computer but that is ALWAYS a massive risk with these things. That risk is not from who trained the model. I could have missed a paper but it seems very unlikely even closed door research has gotten to the stage of maliciously tuning models to surreptitiously backdoor someone's machine in a way that wouldn't be very easy to catch. Your threat model may vary.
- londons_explore 2y agoThe model itself poses no risks (beyond potentially saying things you would prefer not to see). The code that comes with the model should be treated like any other untrusted code.
- nickthegreek 2y agomodel file will be safe
- samuel 2y agoIt's an interesting question! In my opinion, if you don't use tools it's very unlikely it can do any harm. I doubt the model files can be engineered to overflow llama.cpp or ollama, or cause any other damage, directly. But if you use tools, for example for extending its knowledge through web searches, it could be used to exfiltrate information. It could do it by visiting some specially crafted url's to leak parts of your prompts (this includes the contents of documents added to them with RAG). If given an interpreter, even if sandboxed, could try to do some kind of sabotage or "call home" with locally gathered information, obviously disguised as safe "regular" code. It's unlikely that a current model that is runnable in "domestic" hardware could have those capabilities, but in the future these concerns will be more relevant.
- nenaoki 2y ago
- simonw 2y agoThis model is available for MLX now, in various different sizes. I ran https://huggingface.co/mlx-community/Qwen2.5-VL-32B-Instruct-4bit https://huggingface.co/mlx-community/Qwen2.5-VL-32B-Instruct... using uv (so no need to install libraries first) and https://github.com/Blaizzy/mlx-vlm https://github.com/Blaizzy/mlx-vlm like this: uv run --with 'numpy<2' --with mlx-vlm \ python -m mlx_vlm.generate \ --model mlx-community/Qwen2.5-VL-32B-Instruct-4bit \ --max-tokens 1000 \ --temperature 0.0 \ --prompt "Describe this image." \ --image Mpaboundrycdfw-1.png That downloaded an ~18GB model and gave me a VERY impressive result, shown at the bottom here: https://simonwillison.net/2025/Mar/24/qwen25-vl-32b/ https://simonwillison.net/2025/Mar/24/qwen25-vl-32b/
- john_alan 2y agoDoes quantised MLX support vision though? Is UV the best way to run it?
- dphnx 2y agouv is just a Python package manager. No idea why they thought it was relevant to mention that
- stavros 2y agoBecause that one-liner will result in the model instantly running on your machine, which is much more useful than trying to figure out all the dependencies, invariably failing, and deciding that technology is horrible and that all you ever wanted was to be a carpenter.
- simonw 2y agoRight: I could give you a recipe that tells you to first create a Python virtual environment, then install mlx-vlm, then make sure to downgrade to numpy 1.0 because some of the underlying libraries don't work with numpy 2.0 yet... ... or I can give you a one-liner that does all of that with uv.
- michaelt 2y agoHas anyone successfully run a quantized version of any of the Qwen2.5-VL series of models? I've run the smallest model in non-quantized format, but when I've tried to run a AWQ version of one of the bigger models I've struggled to find a combination of libraries that works right - even though it should fit on my GPU.
- slics 2y agoWarning- this is a lengthy comment. I deployed that model locally via Ollama and the reasoning model is not accurate. This is lengthy but I thought I share its reasoning- at some point it just stopped working - then it started responding in Chinese. Query input: “1+5 = 7, and that is the correct answer” Reasoning response: The equation "1 + 5 = 7" isn’t about regular arithmetic addition. Instead, it refers to the number of segments used to display the digits 1 and 5 on a seven-segment display (like on digital clocks or calculators). Here's the breakdown: • Digit 1 uses 2 segments (the vertical lines on the right). • Digit 5 uses 5 segments (top, upper-left, middle, lower-left, and bottom segments). Adding these segments: 2 (for 1) + 5 (for 5) = 7 segments total. This is a clever visual riddle, not a math problem! it was a math problem, you are just wrong Okay, the user is insisting that "1 + 5 = 7 is a math problem" and that my previous explanation was wrong. Hmm, I need to address this carefully. First, in standard arithmetic, 1 + 5 is indeed 6. So if it's a straightforward math problem, the correct answer should be 6. The user might be mistaken, or perhaps they have a different context in mind. Wait, in my previous response, I considered the seven-segment display explanation, which is a common riddle. The user is now rejecting that and stating it's a math problem, implying that their answer of 7 is correct in some mathematical context. Maybe they're referring to a different base? Let me revisit the base idea. If the equation is in a different base, maybe base 9? Let's see: In base b, the equation "1 + 5 = 11" would be 1 + 5 = 6 in decimal, which isn't helpful. Wait, maybe the result is written in a different base? For example, if we compute 1 + 5 in decimal (which is 6) and then represent that in base 7? 6 in base 7 is still "6", not 7. Hmm, not helpful. Alternatively, maybe modulo arithmetic? Like (1 + 5) mod something equals 7? Let's see: 1 +5 = 6. To get 7, maybe modulo something where 6 ≡7 mod (negative number?), which doesn't make sense. Alternatively, maybe the user is adding in a cyclic group? Like mod 6? 6 mod 6 is 0, not 7. Alternatively, could it be a different operation? Like 1 plus 5 in some operation other than addition? Maybe concatenation? 1 concatenated with 5 is 15, but that's not 7. Wait, the user said "another one 1+5 =7". Maybe they're referring to a previous problem where the answer was different, and this is a follow-up? But I don't have context on previous interactions. Alternatively, perhaps the user is joking or trolling, but I should take their statement at face value. Since they insist it's a math problem and that 7 is correct, I need to explore if there's any mathematical framework where this holds. Wait, let's consider if it's a sum in a different base where the result is written in that base. Suppose the equation is in base 8. Let's see: In base 8, "7" is still 7 in decimal, so 1 (base8) +5(base8) = 6 (base8), which is 6 in decimal. Not 7. Alternatively, if the answer is written in base 9: 1 +5 =6 in decimal, which is still 6 in base9. Still not 7. Wait a different angle: Maybe the '+' is not addition, but some other operation. Like bitwise XOR? 1 in binary is 01, 5 is 101. XOR gives 110 (6). Not 7. Or maybe adding digits of the sum? Like 1+5=6, then 6 is a single digit, so adding digits 6, but that's not helpful. Wait, the user might have made a mistake, but they're insistent. Since they called it a math problem, perhaps they're using a different definition of "+"? Like 1 +5 in some algebraic structure where the result is 7? That's possible, but without more context, it's hard to define. Alternatively, maybe the numbers are not in base 10. Suppose the numbers are in base 12. Wait, 1+5 is still 6. Hmm. Wait, maybe the user is considering the numbers as Roman numerals? I+V=VI which is 6. Still not 7. Alternatively, maybe it's a sequence where 1+5 is the 7th term? Unlikely. Alternatively, it could be a trick with letters. Like 1 is the first letter (A), 5 is the fifth letter (E), so AE... but that doesn't lead to 7. Alternatively, adding their positions: 1 +5 =6, which is F. Still not 7. Alternatively, maybe counting letters in spelled-out words? "One" has 3 letters, "five" has 4, total 7? That could be a different riddle. Let's see: 1 (one) +5 (five) = 3 letters +4 letters =7. That's a possible interpretation! If the problem is "1+5" where each number is spelled out in English and you add the number of letters, then 3+4=7. So maybe that's the case here? Alternatively, if it's counting the strokes to write the numbers. For example, writing '1' takes one stroke, '5' takes two strokes (depending on how you write it)? Not sure. Alternatively, the user might have meant 1+5 in a non-standard math problem, like in a clock arithmetic? For example, mod 12. 1+5=6 mod12 is still 6. Hmm, I need to consider that maybe the user intended a different approach that I haven't thought of. Since they are insistent that 7 is correct, perhaps they meant a different operation or context. Alternatively, maybe the problem is written vertically: 1 + 5 ---- 7 But that still doesn't make sense unless there's a carryover. But 1+5 is 6, so even with carryover, you can't get 7. Unless adding digits with some twist. Wait, maybe they're adding the numbers' positions in a different sequence? Like 1st prime plus 5th prime? The first prime is 2, fifth prime is 11. 2+11=13, nope. Alternatively, if you consider 1 and 5 as bits in a binary number, like 1 and 5 (binary 101) concatenated? Not helpful. Alternatively, the problem might be in a different numeral system, like in base 1? But that's not standard. Alternatively, maybe it's a simple mistake, but the user is confident. Since I need to respond, perhaps I should outline the possible interpretations where 7 could be the answer, like the spelled-out letters interpretation. Alternatively, the user could have a typo and meant 2+5=7, but wrote 1+5=7. Alternatively, in some contexts like in programming, maybe 1+5=7 due to a specific coding (like 1 is assigned to a=1, so a+f=6? Not sure. Alternatively, it's a lateral thinking puzzle where the answer is 7 because of some visual or language trick. Wait, another angle: If you add 1 and 5 on a calculator upside down, but that might not help. Alternatively, in temperature scales? For example, converting from Fahrenheit to Celsius or vice versa? 1°F to Celsius is negative, 5°C is 41°F. Not adding to 7. Hmm. Given the user's assertion that 1+5=7 is correct in a math problem, I might need to explain that typically 1+5=6, but there are alternative contexts where it might be considered 7, like counting letters in their names (1 is "one" (3 letters), 5 is "five" (4 letters), 3+4=7). That's a possible interpretation. Alternatively, in some base where the addition is represented differently. Alternatively, a trick question where the numbers are not in base 10 but in another base where the representation is different. Alternatively, the user might have seen a problem where adding 1 and 5 gives 7 through some function, like a + b = a + b +1, but that's arbitrary.
- LeoPanthera 2y agoOpen weight models are coming out so quickly it's difficult to keep track. Is anyone maintaining a list of what is "current" from each model?
- ftufek 2y agohttps://huggingface.co/models https://huggingface.co/models is usually good place to look, you can sort by trending and filter by the task you care about (ex: Image-Text-to-Text). The first page will usually have the leading edge/newer models.
- Tewboo 2y agoImpressive to see advancements in both performance and efficiency. Excited to see how it stacks up in real-world use.
- nico 2y agoHow does it compare to gemma3?
- informal007 2y agoIt's still BF16 model. Deepseek has proved that fp8 is more cost-effectiveness than fp16, isn't it valid for dozens-B model?
- tucnak 2y agoI don't understand what's your point? The optimiser is not fp8 anyway, is it? It's just the weights. I think the extent of fp8 "effectiveness" is greatly exaggerated. Yes, DeepSeek did use fp8, and they did implement it nicely, but it doesn't mean everybody is now got to be using fp8 all of the sudden.
- submeta 2y agoHeavily censors certain political topics (in the Middle East). Grok is doing none of this. How is this possible, coming from China? What stakes got China in the ME conflict?
- voidUpdate 2y agoDoes Grok still deny that trans women are women?
- submeta 2y agoThat’s a red herring. The original point was about censorship on Middle East topics.
- voidUpdate 2y agoIt sounded like you were saying that grok is doing none of the censoring that anything else is doing. And I don't think it's a red herring, since it was definitely doing that in the past
- submeta 2y agoI did not argue for Grok, but was showing that Qwen--to my surprise--is censoring topics that are rather censored in the West, not in China. And Grok--on the contrary--did not censor said topic.
- buyucu 2y agosame day as deepseek v3 update as well!
- rcdwealth 2y ago[dead]
- naasking 2y agoI found Qwen never completed answering my standard coding task that I ask to check a model. Claude did great, DeepSeek R1 did well.
- cryptocrat7 2y agoi have been using it, agreed :100
- cryptocrat7 2y agothere should be a way to share these prompts + tools through visuals