4 ms·
> I think people don't realize how huge these models really are. They really aren't that large by the contemporary scaling race standards. DALLE-2 has 3.5B par
by sinenomine 4y ago
> I think people don't realize how huge these models really are.
They really aren't that large by the contemporary scaling race standards.
DALLE-2 has 3.5B parameters, which should fit on an old GPU like Nvidia RTX2080, especially if you optimize your model for inference [1][2] which is commonly done by ML engineers to minimize costs. With optimized model, your memory footprint is ~1 byte per parameter, and some less than 1 ratio (commonly ~0.2) of all parameters to store intermediate activations.
You should be able to run it on Apple M1/M2 with 16GB RAM via CoreML pretty fine, if an order of magnitude slower than on an A100.
Training isn't unreasonably costly as well: you can train a model given O(100k)$ which is less than a yearly salary of a mid-tier developer in silicon valley.
There is no reason these models shouldn't be trained cooperatively and run locally on our own machines. If someone is interested in cooperating with me on such a project, my email is in the profile.
1. https://arxiv.org/abs/2206.01861 https://arxiv.org/abs/2206.01861
2. https://pytorch.org/blog/introduction-to-quantization-on-pytorch/ https://pytorch.org/blog/introduction-to-quantization-on-pyt...
- gwern 4y agoIt's true that image models are much less of a burden on GPU VRAM than a model like BLOOM where fitting it into a few A100s is ideal, but these diffusion models are a PITA for a ordinary hobbyist in terms of total compute: the CLIP pass over the text input is almost free, but then you feed it into the diffusion model, for one sample you'll be doing 10-100 forward passes (depending on how fancy the diffusion methods are - maybe even 1000 passes if you're using older/simpler ones), and for interactive use, you really want more like 6-9 separate samples; then they have to pass through the upscalers, which are diffusion models themselves and need to do a bunch of forward passes to denoise it. If you do 1 sample in 10s on your 1 consumer GPU, which would be pretty good, 6-9 means a joykilling minute+ wait. And then the user will pick a variation or edit one or tweak the prompt, and start all over again! It's like being back on 16kb dialup waiting for the newest .com to load.
- sinenomine 4y agoAll valid points, of course. As an independent explorer I adapted my workflow to use night's worth of workstation compute to generate a crop of new images from a simple templated prompt language. It also helps a lot to have at least two presets - "exploratory" and "hq", to minimize iteration time and maximize quality of promising prompts. Still, I think optimization of diffusion models for efficient inference isn't yet pushed to the limits. At least if we look at what's available to the public - AFAIK public inference software distributions didn't even quantize their weights.
- antiterra 4y ago> 6-9 means a joykilling minute+ wait Excet we’re already waiting 90+ seconds for dall-e mini.
- meowface 4y agoDALL-E 2 tends to be 15 - 30 seconds, from my experience.
- l33tman 4y agomin-dalle with MEGA model params takes about 20 seconds to generate 9 images on a RTX3080 if you run it locally, including the param loading time (which is about 8 of those seconds)
- MrBra 4y agoHow much VRAM will minDALL-E plus the MEGA model take?
- nprateem 4y agoI've been playing with neural text to speech APIs recently and wondered the same. Why are all the cloud providers charging around $16 per million characters? Do they really need to run on massive machines or are they just scalping/trying to recover training costs?