4 ms·
It's true that image models are much less of a burden on GPU VRAM than a model like BLOOM where fitting it into a few A100s is ideal, but these diffusion models
by gwern 4y ago
It's true that image models are much less of a burden on GPU VRAM than a model like BLOOM where fitting it into a few A100s is ideal, but these diffusion models are a PITA for a ordinary hobbyist in terms of total compute: the CLIP pass over the text input is almost free, but then you feed it into the diffusion model, for one sample you'll be doing 10-100 forward passes (depending on how fancy the diffusion methods are - maybe even 1000 passes if you're using older/simpler ones), and for interactive use, you really want more like 6-9 separate samples; then they have to pass through the upscalers, which are diffusion models themselves and need to do a bunch of forward passes to denoise it. If you do 1 sample in 10s on your 1 consumer GPU, which would be pretty good, 6-9 means a joykilling minute+ wait. And then the user will pick a variation or edit one or tweak the prompt, and start all over again! It's like being back on 16kb dialup waiting for the newest .com to load.
- sinenomine 4y agoAll valid points, of course. As an independent explorer I adapted my workflow to use night's worth of workstation compute to generate a crop of new images from a simple templated prompt language. It also helps a lot to have at least two presets - "exploratory" and "hq", to minimize iteration time and maximize quality of promising prompts. Still, I think optimization of diffusion models for efficient inference isn't yet pushed to the limits. At least if we look at what's available to the public - AFAIK public inference software distributions didn't even quantize their weights.
- antiterra 4y ago> 6-9 means a joykilling minute+ wait Excet we’re already waiting 90+ seconds for dall-e mini.
- meowface 4y agoDALL-E 2 tends to be 15 - 30 seconds, from my experience.
- l33tman 4y agomin-dalle with MEGA model params takes about 20 seconds to generate 9 images on a RTX3080 if you run it locally, including the param loading time (which is about 8 of those seconds)
- MrBra 4y agoHow much VRAM will minDALL-E plus the MEGA model take?