5 ms·
This may be obvious to people who do this regularly, but what kind of machine is required to run this? I downloaded & tried it on my Linux machine that has a 1
by rwmj 1y ago
This may be obvious to people who do this regularly, but what kind of machine is required to run this? I downloaded & tried it on my Linux machine that has a 16GB GPU and 64GB of RAM. This machine can run SD easily. But Qwen-image ran out of space both when I tried it on the GPU and on the CPU, so that's obviously not enough. But am I off by a factor of two? An order of magnitude? Do I need some crazy hardware?
- zippothrowaway 1y agoYou're probably going to have to wait a couple of days for 4 bit quantized versions to pop up. It's 20B parameters.
- pollinations 1y ago# Configure NF4 quantization quant_config = PipelineQuantizationConfig( quant_backend="bitsandbytes_4bit", quant_kwargs={"load_in_4bit": True, "bnb_4bit_quant_type": "nf4", "bnb_4bit_compute_dtype": torch.bfloat16}, components_to_quantize=["transformer", "text_encoder"], ) # Load the pipeline with NF4 quantization pipe = DiffusionPipeline.from_pretrained( model_name, quantization_config=quant_config, torch_dtype=torch.bfloat16, use_safetensors=True, low_cpu_mem_usage=True ).to(device) seems to use 17gb of vram like this update: doesn't work well. this approach seems to be recommended: https://github.com/QwenLM/Qwen-Image/pull/6/files https://github.com/QwenLM/Qwen-Image/pull/6/files
- mortsnort 1y agoI believe it's roughly the same size as the model files. If you look in the transformers folder you can see there are around 9 5gb files, so I would expect you need ~45gb vram on your GPU. Usually quantized versions of models are eventually released/created that can run on much less vram but with some quality loss.
- foobarqux 1y agoWhy doesn't huggingface list the aggregate model size?
- bavell 1y agoI've been wondering this for literally years now...
- matcha-video 1y agoHuggingface is just a git hosting service, like github. You can add up the sizes of all the files in the directory yourself
- Auracle 1y agoThat’s what we have computers for though - to compute.
- simonw 1y agoI've been bugging them about this for a while. There are repos that contain multiple model weights in a single repo which means adding up the file sizes won't work universally, but I'd still find it useful to have a "repo size" indicator somewhere. I ended up building my own tool for that: https://tools.simonwillison.net/huggingface-storage https://tools.simonwillison.net/huggingface-storage
- deleted 1y ago[deleted]
- Gracana 1y agoHF does this for ggufs, and it’ll show you what quantizations will work on the GPU(s) you’ve selected. Hopefully that feature gets expanded to support more model types.
- halJordan 1y agoModel size = file for fp8, so if this was released at fp16 then 40-ish, if it's quantized to fp4 then 10ish
- slickytail 1y ago[dead]
- TacticalCoder 1y ago> I think the fact that, as far as I understand, it takes 40GB of VRAM to run, is probably dampening some of the enthusiasm. For PCs I take it one that has two PCIe 4.0 x16 or more recent slots? As in: quite some consumers motherboards. You then put two GPU with 24 GB of VRAM each. A friend runs this (don't know if the tried this Qwen-Image yet): it's not an "out of this world" machine.
- ticulatedspline 1y agomaybe not "out of this world" but still not cheap. probably $4,000 with 3090s. pretty big chunk of change for some ai pictures.
- Auracle 1y agoYou can’t split diffusion models like that.
- deleted 1y ago[deleted]
- icelancer 1y ago> This may be obvious to people who do this regularly This is not that obvious. Calculating VRAM usage for VLMs/LLMs is something of an arcane art. There are about 10 calculators online you can use and none of them work. Quantization, KV caching, activation, layers, etc all play a role. It's annoying. But anyway, for this model, you need 40+ GB of VRAM. System RAM isn't going to cut it unless it's unified RAM on Apple Silicon, and even then, memory bandwidth is shot, so inference is much much slower than GPU/TPU.
- cellis 1y agoAlso I think you need a 40GB "card", not just 40GB of vram. I wrote about this upthread, you're probably going to need one card, I'd be surprised if you could chain several GPUs together.
- rapfaria 1y agoNot sure what you mean or new to llms, but two RTX 3090 will work for this, and even lower-end cards will (RTX3060) once it's GGUF'd
- karolist 1y agodo you mean https://github.com/pollockjj/ComfyUI-MultiGPU https://github.com/pollockjj/ComfyUI-MultiGPU? One GPU would do the computation, but others could pool in for VRAM expansion, right? (I've not used this node)
- Auracle 1y agoNah, that won’t gain you much (if anything?) over just doing the layer swaps on RAM. You can put the text encoder on the second card but you can also just put it in your RAM without much for negatives.
- axoltl 1y agoThis isn't a transformer, it's a diffusion model. You can't split diffusion models across compute nodes.
- liuliu 1y ago16GiB RAM with 8-bit quantization. This is a slightly scaled up SD3 Large model (38 layers -> 60 layers).
- philipkiely 1y agoFor prod inference, 1xH100 is working well.
- ethan_smith 1y agoQwen-Image requires at least 24GB VRAM for the full model, but you can run the 4-bit quantized version with ~8GB VRAM using libraries like AutoGPTQ.
- cjtrowbridge 1y agotwo p40 cards together will run this for under $300