7 ms·

Is there a good explanation of how to train this from scratch with a custom dataset[0]? I've been looking around the documentation on Huggingface, but all I co

Is there a good explanation of how to train this from scratch with a custom dataset[0]? I've been looking around the documentation on Huggingface, but all I could find was either how to train unconditional U-Nets[1], or how to use the pretrained Stable Diffusion model to process image prompts (which I already know how to do). Writing a training loop for CLIP manually wound up with me banging against all sorts of strange roadblocks and missing bits of documentation, and I still don't have it working. I'm pretty sure I also need some other trainables at some point, too. [0] Specifically, Wikimedia Commons images in the PD-Art-100 category, because the images will be public domain in the US and the labels CC-BY-SA. This would rule out a lot of the complaints people have about living artists' work getting scraped into the machine; and probably satisfy Debian's ML guidelines. [1] Which actually does work

echelon 4y ago
> train this from scratch If you're talking about training from scratch and not fine tuning, that won't be cheap or easy to do. You need thousands upon thousands of dollars of GPU compute [1] and a gigantic data set. I trained something nowhere near the scale of Stable Diffusion on Lambda Labs, and my bill was $14,000. [1] Assuming you rent GPUs hourly, because buying the hardware outright will be prohibitively expensive.
- pfd1986 4y ago
  Hi, do you have a writeup of that anywhere? Would love to hear (read) more about it
- kmeisthax 4y ago
  I have... ~11TBs of free disk space and a 1080ti. Obviously nowhere close to being able to crunch all of Wikimedia Commons, but I'm also not trying to beat Stability AI at their own game. I just want to move the arguments people have about art generators beyond "this is unethical copyright laundering" and "the model is taking reference just like a real human".
  - ftufek 4y ago
    To put things in perspective, the dataset it's trained on is ~240TB and Stability has over ~4000 Nvidia A100 (which is much faster than a 1080ti). Without those ingredients, you're highly unlikely to get a model that's worth using (it'll produce mostly useless outputs). That argument also makes little sense when you consider that the model is a couple gigabytes itself, it can't memorize 240TB of data, so it "learned". But if you want to create custom versions of SD, you can always try out dreambooth: https://github.com/XavierXiao/Dreambooth-Stable-Diffusion https://github.com/XavierXiao/Dreambooth-Stable-Diffusion, that one is actually feasible without spending millions of dollars on GPUs.
    - wokwokwok 4y ago
      Quite right, but… > That argument also makes little sense when you consider that the model is a couple gigabytes itself, it can't memorize 240TB of data, so it "learned". The matter is really very nuanced and trivialising it that way is unhelpful. If I recompress 240TB as super low quality jpgs and manage to zip them up as single file that is significantly smaller than 240TB (because you can), does the fact they are not pixel perfect matches for the original images mean you’re not violating copyright? If an AI model can generate statistically significantly similar images from the training data, with a trivial guessable prompt (“a picture by xxx” or whatever) then it’s entirely arguable that the model is similarly infringing. The exact compression algorithm, be it model or jpg or zip is irrelevant to that point. It’s entirely reasonable to say, if this is so good at learning, why don’t you train it without the art station dataset. …because if it’s just learning techniques, generic public domain art should be fine right? Can’t you just engineer the prompting better so that it generates “by Greg Rutkowski“ images without being trained on actual images by Greg? If not, then it’s not just learning technique, it’s copying. So; tldr: there’s plenty of scope for trying to train a model on an ethically sourced dataset, and investigation of techniques vs copying in generative models. It is 100% not something we can just brush off.
      - visarga 4y ago
        > The matter is really very nuanced and trivialising it that way is unhelpful. Harping about copyrights in the Age of Diffusion Models is unhelpful (for artists) like protesting against a tsunami. It's time to move up the ladder. ML engineers have a similar predicament - GPT-3 like models can solve at first try, without specialised training, tasks that took a whole team a few years of work. Who dares still use LSTMs now like it's 2017? Moving up the ladder, learning to prompt and fine-tune ready made models is the only solution for ML eng. The reckoning is coming for programmers and for writers as well. Even scientific papers can be generated by LLMs now - see the Galactica scandal where some detractors said it will empower people to write fake papers. It also has the best ability to generate appropriate citations. The conclusion is that we need to give up some of the human-only tasks and hop on the new train.
- fastball 4y ago
  - adamdusty 4y ago
    "nowhere near"
    - fastball 4y ago
      Yep, missed the word "nowhere". My mistake.
- Jack000 4y ago
  Depends on the dataset. You can probably get decent results by restricting the modality of the images (faces, cars, bedrooms etc) I trained from scratch with 4x3090 and while it’s not as good as SD it’s surprisingly better with hands.
  - svnt 4y ago
    How long did the training take on 4 3090s?
    - Jack000 4y ago
      About 1 month actual training time. It’s a smaller (650m) model and probably still undertrained. Glid3 on GitHub.
  - hackater 4y ago
    Can you go into a bit more detail? What architecture did you use? Is the month training time really just training with mini batches with a constant learning rate? Or are these many failed attempts until you trained a successful model for a few days in the end? I particularly interested in the image generation part (the DDPM/SGM)
    - Jack000 4y ago
      Yeah I did have a few false starts. Total time is more like 3 months vs 1 month for the final model. For small scale training I found it’s necessary to use a long lr warmup period, followed by constant lr. There’s code on my GitHub (glid3) edit: The architecture is identical to SD except I trained on 256px images with cosine noise schedule instead of linear. Using the cosine schedule makes the unet converge faster but can overfit if overtrained. edit 2: Just tried it again and my model is also pretty bad at hands actually. It does get lucky once in a while though.
      - rrobukef 4y ago
        I keep wondering if using not only statistical noise but also deformations would help with the generation of deformable things - say human hands.
ShamelessC 4y ago
> Writing a training loop for CLIP manually wound up with me banging against all sorts of strange roadblocks and missing bits of documentation, and I still don't have it working. There is working training code for openCLIP https://github.com/mlfoundations/open_clip https://github.com/mlfoundations/open_clip But training multi-modal text-to-image models is still a _very_ new thing, in terms of the software world. Given that, my experience has been that it's never been easier to get to work on this stuff from the software POV. The hardware is the tricky bit (and preventing bandwidth issues on distributed systems). That isn't to say that there isn't code out there for training. Just that you're going to run into issues and learning how to solve those issues as you encounter them is going to be a highly valuable skill soon. edit: I'm seeing in a sibling comment that you're hoping to train your own model from scratch on a single GPU. Currently, at least, scaling laws for transformers [0] mean that the only models that perform much of anything at all need a lot of parameters. The bigger the better - as far as we can tell. Very simply - researchers start by making a model big enough to fill a single GPU. Then, they replicate the model across hundreds/thousands of GPU's, but feed each on a different set of the data. Model updates are then synchronized, hopefully taking advantage of some sort of pipelining to avoid bottlenecks. This is referred to as data-parallel. [0] https://www.lesswrong.com/tag/scaling-laws https://www.lesswrong.com/tag/scaling-laws
- jimmySixDOF 4y ago
  All this horsepower deployed to image generation is interesting but somebody wake me up when there is a stable diffusion for SQL or when on demand generative User Interfaces are spun up on the fly to suit the purpose.
  - ShamelessC 4y ago
    Will do!
__rito__ 4y ago
Well, you can learn about generative models from MOOCs like the ones taught at UMich, Universitat Tubingen, or New York University (taught by Yann LeCun), and can gain knowledge there. You can also watch the fast.ai MOOC titled Deep Learning from Scratch to Stable Diffusion [0]. You can also look at open source implementation of text2image models like Dall-E Mini or the works of lucid rain. I worked on the Dall-E Mini project, and the technical knowhow that you need isn’t closely taught at MOOCs. You need to know, on top of Deep Learning theory, many tricks, gotchas, workarounds, etc. You could follow the works of Eluther AI, follow Boris Dayma (project leader of Dall-E Mini) and Horace Ho on twitter. And any such people who have significant experience in practical AI and regularly share their tricks. The PyTorch forums is also a good place. Learn PyTorch and/or JAX/Flax really well. [0]: https://www.fast.ai/posts/part2-2022.html https://www.fast.ai/posts/part2-2022.html
TaylorAlexander 4y ago
Ah I am glad to see someone else talking about using public domain images! Honestly it baffles me that in all this discussion, I rarely see people discussing how to do this with appropriately licensed images. There are some pretty large datasets out there of public images, and doing so might even help encourage more people to contribute to open datasets. Also if the big ML companies HAD to use open images, they would be forced to figure out sample efficiency for these models. Which is good for the ML community! They would also be motivated to encourage the creation of larger openly licensed datasets, which would be great. I still think if we got twitter and other social media sites to add image license options, then people who want to contribute to open datasets could do so in an easy and socially contagious way. Maybe this would be a good project for mastodon contributors, since that is something we actually have control over. I'd be happy to license my photography with an open license! It is really a wonderful idea to try to do this with open data. Maybe it won't work very well with current techniques, but that just becomes an engineering problem worth looking at (sample efficiency).
- bowsamic 4y ago
  No one is ever going to stop using all the available images until there is a law against it. Why would they?
  - LunaSea 4y ago
    There are already laws against it but enforcement is laking, as always.
    - bonoboTP 4y ago
      If you can't process/digest copyrighted content with algorithms/machine learning then Google Search (the whole thing, not just Image Search) is dead. So no, it's not at all clear where the legal lines are drawn. There have been no court cases yet, regarding the training of ML models. People are trying to draw analogies from other types of cases, but this has not been tried in court yet. And then the answer will likely differ based on country.
j0hnM1st 4y ago
It will be worthwhile to use images from commons. I have found that my photography is used in the stable diffusion data set. What was funny is that they have taken the images from other URLs than my flickr account.
TeMPOraL 4y ago
> Specifically, Wikimedia Commons images in the PD-Art-100 category, because the images will be public domain in the US and the labels CC-BY-SA. Doesn't the "BY" part of the license mean you have to provide attribution along with your models' output[0]? I feel you'll have the equivalent of Github Copilot problem: it might be prohibitive to correctly attribute each output, and listing the entire dataset in attribution section won't fly either. And if you don't attribute, your model is no different than Stable Diffusion, Copilot and other hot models/tools: it's still a massive copyright violation and copyright laundering tool. ---- [0] - https://creativecommons.org/licenses/by-sa/4.0/ https://creativecommons.org/licenses/by-sa/4.0/
- deleted 4y ago
  [deleted]
- etienne618 4y ago
  I feel quite strongly that there is a large difference between Stable Diffusion and Copilot: with the size of the training set vs the number of parameters, it should be very difficult if not impossible for Stable Diffusion to memorize and, by extension, copy paste to produce its outputs. Copilot is trained on text and outputs text. Coding is also inherently more difficult for an AI model to do. I expect it will memorize large portions of its input and is copy pasting in many cases to produce output. I therefore believe Copilot is doing "copyright laundering" but Stable Diffusion is not. Furthermore, I do not believe, for example, that artists should be able to copyright a "style" - but I would like to see them not be negatively impacted by this. Its complicated.
  - Xelynega 4y ago
    Let me guess that you write more code than visual art? Isnt it a bit anthropomorphic to compare the two algorithms by "how a human believes they work" instead of "what they're actually doing different to the inputs to create the outputs"? These are algorithms and we can look at how they work, so it feels like a cop-out to not do that.
- amadvance 4y ago
  The SA part (ShareAlike) is even more restrictive, as it imposes a license on the derivative work. "— If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original"
sabalaba 4y ago
Here’s a tutorial on how to fine tune stable diffusion form the guy who made text-to-pokemon: https://lambdalabs.com/blog/how-to-fine-tune-stable-diffusion-how-we-made-the-text-to-pokemon-model-at-lambda https://lambdalabs.com/blog/how-to-fine-tune-stable-diffusio...
deleted 4y ago
[deleted]
Der_Einzige 4y ago
A paper was already presented at this workshop at COLING 2022 by Nvidia which already does this https://arxiv.org/abs/2209.14697 https://arxiv.org/abs/2209.14697

echelon 4y ago

> train this from scratch If you're talking about training from scratch and not fine tuning, that won't be cheap or easy to do. You need thousands upon thousands of dollars of GPU compute [1] and a gigantic data set. I trained something nowhere near the scale of Stable Diffusion on Lambda Labs, and my bill was $14,000. [1] Assuming you rent GPUs hourly, because buying the hardware outright will be prohibitively expensive.

pfd1986 4y ago
Hi, do you have a writeup of that anywhere? Would love to hear (read) more about it
kmeisthax 4y ago
I have... ~11TBs of free disk space and a 1080ti. Obviously nowhere close to being able to crunch all of Wikimedia Commons, but I'm also not trying to beat Stability AI at their own game. I just want to move the arguments people have about art generators beyond "this is unethical copyright laundering" and "the model is taking reference just like a real human".
- ftufek 4y ago
  To put things in perspective, the dataset it's trained on is ~240TB and Stability has over ~4000 Nvidia A100 (which is much faster than a 1080ti). Without those ingredients, you're highly unlikely to get a model that's worth using (it'll produce mostly useless outputs). That argument also makes little sense when you consider that the model is a couple gigabytes itself, it can't memorize 240TB of data, so it "learned". But if you want to create custom versions of SD, you can always try out dreambooth: https://github.com/XavierXiao/Dreambooth-Stable-Diffusion https://github.com/XavierXiao/Dreambooth-Stable-Diffusion, that one is actually feasible without spending millions of dollars on GPUs.
  - wokwokwok 4y ago
    Quite right, but… > That argument also makes little sense when you consider that the model is a couple gigabytes itself, it can't memorize 240TB of data, so it "learned". The matter is really very nuanced and trivialising it that way is unhelpful. If I recompress 240TB as super low quality jpgs and manage to zip them up as single file that is significantly smaller than 240TB (because you can), does the fact they are not pixel perfect matches for the original images mean you’re not violating copyright? If an AI model can generate statistically significantly similar images from the training data, with a trivial guessable prompt (“a picture by xxx” or whatever) then it’s entirely arguable that the model is similarly infringing. The exact compression algorithm, be it model or jpg or zip is irrelevant to that point. It’s entirely reasonable to say, if this is so good at learning, why don’t you train it without the art station dataset. …because if it’s just learning techniques, generic public domain art should be fine right? Can’t you just engineer the prompting better so that it generates “by Greg Rutkowski“ images without being trained on actual images by Greg? If not, then it’s not just learning technique, it’s copying. So; tldr: there’s plenty of scope for trying to train a model on an ethically sourced dataset, and investigation of techniques vs copying in generative models. It is 100% not something we can just brush off.
    - visarga 4y ago
      > The matter is really very nuanced and trivialising it that way is unhelpful. Harping about copyrights in the Age of Diffusion Models is unhelpful (for artists) like protesting against a tsunami. It's time to move up the ladder. ML engineers have a similar predicament - GPT-3 like models can solve at first try, without specialised training, tasks that took a whole team a few years of work. Who dares still use LSTMs now like it's 2017? Moving up the ladder, learning to prompt and fine-tune ready made models is the only solution for ML eng. The reckoning is coming for programmers and for writers as well. Even scientific papers can be generated by LLMs now - see the Galactica scandal where some detractors said it will empower people to write fake papers. It also has the best ability to generate appropriate citations. The conclusion is that we need to give up some of the human-only tasks and hop on the new train.
fastball 4y ago
- adamdusty 4y ago
  "nowhere near"
  - fastball 4y ago
    Yep, missed the word "nowhere". My mistake.
Jack000 4y ago
Depends on the dataset. You can probably get decent results by restricting the modality of the images (faces, cars, bedrooms etc) I trained from scratch with 4x3090 and while it’s not as good as SD it’s surprisingly better with hands.
- svnt 4y ago
  How long did the training take on 4 3090s?
  - Jack000 4y ago
    About 1 month actual training time. It’s a smaller (650m) model and probably still undertrained. Glid3 on GitHub.
- hackater 4y ago
  Can you go into a bit more detail? What architecture did you use? Is the month training time really just training with mini batches with a constant learning rate? Or are these many failed attempts until you trained a successful model for a few days in the end? I particularly interested in the image generation part (the DDPM/SGM)
  - Jack000 4y ago
    Yeah I did have a few false starts. Total time is more like 3 months vs 1 month for the final model. For small scale training I found it’s necessary to use a long lr warmup period, followed by constant lr. There’s code on my GitHub (glid3) edit: The architecture is identical to SD except I trained on 256px images with cosine noise schedule instead of linear. Using the cosine schedule makes the unet converge faster but can overfit if overtrained. edit 2: Just tried it again and my model is also pretty bad at hands actually. It does get lucky once in a while though.
    - rrobukef 4y ago
      I keep wondering if using not only statistical noise but also deformations would help with the generation of deformable things - say human hands.

ShamelessC 4y ago

> Writing a training loop for CLIP manually wound up with me banging against all sorts of strange roadblocks and missing bits of documentation, and I still don't have it working. There is working training code for openCLIP https://github.com/mlfoundations/open_clip https://github.com/mlfoundations/open_clip But training multi-modal text-to-image models is still a _very_ new thing, in terms of the software world. Given that, my experience has been that it's never been easier to get to work on this stuff from the software POV. The hardware is the tricky bit (and preventing bandwidth issues on distributed systems). That isn't to say that there isn't code out there for training. Just that you're going to run into issues and learning how to solve those issues as you encounter them is going to be a highly valuable skill soon. edit: I'm seeing in a sibling comment that you're hoping to train your own model from scratch on a single GPU. Currently, at least, scaling laws for transformers [0] mean that the only models that perform much of anything at all need a lot of parameters. The bigger the better - as far as we can tell. Very simply - researchers start by making a model big enough to fill a single GPU. Then, they replicate the model across hundreds/thousands of GPU's, but feed each on a different set of the data. Model updates are then synchronized, hopefully taking advantage of some sort of pipelining to avoid bottlenecks. This is referred to as data-parallel. [0] https://www.lesswrong.com/tag/scaling-laws https://www.lesswrong.com/tag/scaling-laws

jimmySixDOF 4y ago
All this horsepower deployed to image generation is interesting but somebody wake me up when there is a stable diffusion for SQL or when on demand generative User Interfaces are spun up on the fly to suit the purpose.
- ShamelessC 4y ago
  Will do!

__rito__ 4y ago

Well, you can learn about generative models from MOOCs like the ones taught at UMich, Universitat Tubingen, or New York University (taught by Yann LeCun), and can gain knowledge there. You can also watch the fast.ai MOOC titled Deep Learning from Scratch to Stable Diffusion [0]. You can also look at open source implementation of text2image models like Dall-E Mini or the works of lucid rain. I worked on the Dall-E Mini project, and the technical knowhow that you need isn’t closely taught at MOOCs. You need to know, on top of Deep Learning theory, many tricks, gotchas, workarounds, etc. You could follow the works of Eluther AI, follow Boris Dayma (project leader of Dall-E Mini) and Horace Ho on twitter. And any such people who have significant experience in practical AI and regularly share their tricks. The PyTorch forums is also a good place. Learn PyTorch and/or JAX/Flax really well. [0]: https://www.fast.ai/posts/part2-2022.html https://www.fast.ai/posts/part2-2022.html

TaylorAlexander 4y ago

Ah I am glad to see someone else talking about using public domain images! Honestly it baffles me that in all this discussion, I rarely see people discussing how to do this with appropriately licensed images. There are some pretty large datasets out there of public images, and doing so might even help encourage more people to contribute to open datasets. Also if the big ML companies HAD to use open images, they would be forced to figure out sample efficiency for these models. Which is good for the ML community! They would also be motivated to encourage the creation of larger openly licensed datasets, which would be great. I still think if we got twitter and other social media sites to add image license options, then people who want to contribute to open datasets could do so in an easy and socially contagious way. Maybe this would be a good project for mastodon contributors, since that is something we actually have control over. I'd be happy to license my photography with an open license! It is really a wonderful idea to try to do this with open data. Maybe it won't work very well with current techniques, but that just becomes an engineering problem worth looking at (sample efficiency).

bowsamic 4y ago
No one is ever going to stop using all the available images until there is a law against it. Why would they?
- LunaSea 4y ago
  There are already laws against it but enforcement is laking, as always.
  - bonoboTP 4y ago
    If you can't process/digest copyrighted content with algorithms/machine learning then Google Search (the whole thing, not just Image Search) is dead. So no, it's not at all clear where the legal lines are drawn. There have been no court cases yet, regarding the training of ML models. People are trying to draw analogies from other types of cases, but this has not been tried in court yet. And then the answer will likely differ based on country.

j0hnM1st 4y ago

It will be worthwhile to use images from commons. I have found that my photography is used in the stable diffusion data set. What was funny is that they have taken the images from other URLs than my flickr account.

TeMPOraL 4y ago

> Specifically, Wikimedia Commons images in the PD-Art-100 category, because the images will be public domain in the US and the labels CC-BY-SA. Doesn't the "BY" part of the license mean you have to provide attribution along with your models' output[0]? I feel you'll have the equivalent of Github Copilot problem: it might be prohibitive to correctly attribute each output, and listing the entire dataset in attribution section won't fly either. And if you don't attribute, your model is no different than Stable Diffusion, Copilot and other hot models/tools: it's still a massive copyright violation and copyright laundering tool. ---- [0] - https://creativecommons.org/licenses/by-sa/4.0/ https://creativecommons.org/licenses/by-sa/4.0/

deleted 4y ago
[deleted]
etienne618 4y ago
I feel quite strongly that there is a large difference between Stable Diffusion and Copilot: with the size of the training set vs the number of parameters, it should be very difficult if not impossible for Stable Diffusion to memorize and, by extension, copy paste to produce its outputs. Copilot is trained on text and outputs text. Coding is also inherently more difficult for an AI model to do. I expect it will memorize large portions of its input and is copy pasting in many cases to produce output. I therefore believe Copilot is doing "copyright laundering" but Stable Diffusion is not. Furthermore, I do not believe, for example, that artists should be able to copyright a "style" - but I would like to see them not be negatively impacted by this. Its complicated.
- Xelynega 4y ago
  Let me guess that you write more code than visual art? Isnt it a bit anthropomorphic to compare the two algorithms by "how a human believes they work" instead of "what they're actually doing different to the inputs to create the outputs"? These are algorithms and we can look at how they work, so it feels like a cop-out to not do that.
amadvance 4y ago
The SA part (ShareAlike) is even more restrictive, as it imposes a license on the derivative work. "— If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original"

sabalaba 4y ago

Here’s a tutorial on how to fine tune stable diffusion form the guy who made text-to-pokemon: https://lambdalabs.com/blog/how-to-fine-tune-stable-diffusion-how-we-made-the-text-to-pokemon-model-at-lambda https://lambdalabs.com/blog/how-to-fine-tune-stable-diffusio...

deleted 4y ago

[deleted]

Der_Einzige 4y ago

A paper was already presented at this workshop at COLING 2022 by Nvidia which already does this https://arxiv.org/abs/2209.14697 https://arxiv.org/abs/2209.14697