3 ms·
Diffusion Training from Scratch on a Micro-Budget
- deleted 2y ago[deleted]
- orbital-decay 2y agoReminds me of PixArt-α which was also trained on the similarly tiny budget ($28,000). [0] How good is their result, though? Training a toy model is one thing, making something usable (let alone competitive) is another. Edit: they do have comparisons in the paper, and PixArt-α seems to be... more coherent? [0] https://pixart-alpha.github.io/ https://pixart-alpha.github.io/
- daghamm 2y agoThey mention that as state of the art, this one is supposed to be 14-18x better. By the way, has anyone ran these locally? Is the inference time also lower?
- p1esk 2y agoInteresting - they say using FP8 didn’t provide any speed up.
- Flux159 2y agoThis kind of research is great for reducing training costs as well as enabling more people to experiment with training large models. Hopefully in 5-10 years we'll be able to train a model on par with SD 1.5 with consumer gpus since that would be great for teaching model development.
- Blackthorn 2y agoGetting parity with SD 1.5 should require a similarly comprehensive data set, which seems a lot harder to source than a computer GPU. Especially now that we've got the A I-equivalent of pre/post nuclear steel.
- roenxi 2y agoGiven how little artistic data humans need, there are probably breakthroughs coming that will reduce the size of the data set needed. Or make it so that a lot of the data required is more generic (like how a human artist needs vast amounts of audio-visual data from walking around every day, but maybe as little as a few megabytes to go from nothing to copying a new style and subject - then we can have a curated open source "highlights of the first 20 years of life" data set that everyone uses for basic training).
- whywhywhywhy 2y ago> Getting parity with SD 1.5 should require a similarly comprehensive data set, which seems a lot harder to source Wasn't SD1.5 trained on LAION? So we know what it was and you could recreate it. Although I thought LAION was why SD1.5 is kinda ugly at base settings because LAION is just random images both good and bad content and quality not aesthetic and high quality images.
- philipkglass 2y agoThe LAION datasets don't contain actual images, but URLs pointing to images. Due to link rot and deliberate scraper-blocking it may be difficult to download LAION images to retrain a model to match SD 1.5.
- Blackthorn 2y agoLow-quality, non-aesthetic (but still natural) images are still pretty important for training a high quality, diverse model. Also unless someone saved the actual data somewhere and will make it available, LAION-5B is rotted away.
- Ruby496Williams 2y ago[flagged]
- thomashop 2y agoI'm pretty sure we are looking at something like 12 months. Not 5-10 years. Pixart and this paper are good data points. Another even just 50x reduction in cost will make it possible on consumer hardware easily. This paper already claims over 100x reduction
- pk-protect-ai 2y agoI hope that somewhere around that period of time we will have AI-based "game" engines working at 30-40fps at 4K (of course with upscaling). I mean it might not be game engines per se, but universal, interactive pipelines for audio-visual content creation/consumption. Because right now I do not see any hope of such engines due to the number of models involved and the latencies this implies.
- kaiokendev 2y agoi'm really hoping for this as well! i'm a big believer that neural rendering pipelines will overtake traditional push-tris-to-gpus that we've essentially been using since Descent
- deleted 2y ago[deleted]
- worstspotgain 2y agoAsymptotic improvements are flattening the cost curves so fast that AI regulation might become practically meaningless by the end of the year. If you want unregulated output you'll have tons of offshore models to choose from. The risk is that the good guys end up being the only ones hampered by it. Hopefully it won't be so large a burden that the bad guys and especially the so-so guys (those with a real chance, e.g. Alibaba) get a massive leg up.
- FeepingCreature 2y agoUnregulated output at small scales. The really big training runs will still cost millions.
- worstspotgain 2y agoNot when we're talking asymptotically. The linked paper for instance claims 14- to 118-fold cost reductions. 1-2 GPU generations from now you'll train this model for $0.12.
- impossiblefork 2y agoSurely not $0.12. Maybe $100.
- moffkalast 2y agoPeople are casually dropping thousands on cloud GPUs making random fine tunes over at r/localllama, the threshold will be met far sooner. Plus datacenters selling away their collection of A100s and eventually H100s when they become EoL for their standards.
- impossiblefork 2y agoYes, but that doesn't mean that things are suddenly incredibly cheap. It just means that people have hardware. $0.12 I feel is extremely superlative and I feel that $100 is more reasonable.
- sorenjan 2y agoOne thing I've wondered about is fine tuning a large model from multiple LoRAs. If the model doesn't fit in your vram you can train a LoRA, apply it to the model, train another LoRA from the same data, apply it, and so on. Iterative low rank parameter updates. Would that work?