6 ms·

From Deep Learning Foundations to Stable Diffusion

258 points by wasimlorgat 4y ago

O__________O 4y ago
>> The course will be available for free online from early 2023. Anyone aware of an in-depth intro-level text-based explanation of Stable Diffusion that covers the whole pipeline, including training on an extremely limited dataset? Here’s an example, but open to suggestions too: https://huggingface.co/blog/stable_diffusion https://huggingface.co/blog/stable_diffusion
- nyoomboom 4y ago
  This year's Deep Learning Indaba had a tutorial on diffusion models in Jax: https://github.com/deep-learning-indaba/indaba-pracs-2022/tree/main/practicals https://github.com/deep-learning-indaba/indaba-pracs-2022/tr...
- mpaepper 4y ago
  I wrote some of that as a blog entry: https://www.paepper.com/blog/posts/how-and-why-stable-diffusion-works-for-text-to-image-generation/ https://www.paepper.com/blog/posts/how-and-why-stable-diffus...
hansworst 4y ago
This is exactly the kind of course I’ve wanted to do for some time now. Even before stable diffusion it felt like other media synthesis applications like StyleGAN were what I wanted to learn, but most machine learning courses focus on more traditional data science topics. Of course you can start with a more traditional course and then learn something like stable diffusion afterwards, but as a newbie it’s quite hard to figure out where to even start. A full-fledged course that takes you exactly where you want to go is a lot easier and I think it can help learners to stay motivated because they have a clear goal in mind. If I want to learn how to create cool images, I want to spend as little time as possible predicting housing prices in the Bay Area.
- mudrockbestgirl 4y ago
  > If I want to learn how to create cool images, I want to spend as little time as possible predicting housing prices in the Bay Area. I think that's somewhat of a dangerous mindset to have. If you want to create cool images you can use pre-trained models and high-level APIs without needing to understand any of the internals. But if you want to truly understand how these models work, you need to make effort to study the basics. Maybe not predicting housing prices, but learn the foundational math and primitives behind all of the components from the ground up (and the Diffusion models are a complex beast made up of many components). And getting an intuitive understanding of how models behave when you tune certain knobs takes much longer. Many researchers in the field have spent years developing their intuition of what works and what doesn't. Both of these are fine, but I think I think we should stop encouraging people to be in the middle. Have courses that that promise "Learn Deep Learning / Transformers / Diffusion models in 7 days!" but then go on and teach you how to call blackbox APIs, giving you an illusion of knowledge and understanding where there is none. I don't know if this applies to this specific course, but there are a bunch of those out there, and highly recommend staying away from those. I know it's a hard sell in this modern instant gratification age, but if you actually want to understand something you need to put in some possibly hard work.
  - jan_Inkepa 4y ago
    > I don't know if this applies to this specific course, but there are a bunch of those out there, and highly recommend staying away from those. fast.ai do stuff pretty well. FWIW, I did one of their earlier free courses and, as a maths grad, got my fill of maths learning as well as my fill of practical 'doing stuff with ML' stuff. If I didn't have my plate full I'd probably pay the 500 quid or whatever to do this course now rather than wait for the free version.
  - boredemployee 4y ago
    >> But if you want to truly understand how these models work, you need to make effort to study the basics. I was very confused by this in the beginning of my journey. I was trying to learn everything involved with ML/DL, but in the end everything is already implemented with APIs, and your boss doesnt care if you know how to implement a MLP from scratch or if you use Tensorflow. My (poor) analogy is: you don't need to know how a car works (or how to build one) in every detail to drive it. When I understood it, it was liberating.
neodypsis 4y ago
This course looks interesting. My only concern is that I don't have real experience with NLP. Anybody can recommend resources to get to speed on this pre-requisite? My NLP knowledge is very basic.
- Version467 4y ago
  The "Practical Deep learning" course from fast.ai has a section on NLP that's probably a good starting point.
  - neodypsis 4y ago
    Thanks, I'll check it out.
bilsbie 4y ago
Is there any way to motivate myself to take a class like this? I keep turning to easy distractions like twitter and Pac-Man.
- dagmx 4y ago
  When I teach people, the most effective thing I find is to give them small usable projects at the end of each little milestone. That helps them have something they can use right away, which helps with a few things: - at each milestone, they have a distinct goal that’s reachable - they can understand concrete use cases immediately - they get a little dopamine hit if satisfaction of having completed something. Whenever you’re doing a course that doesn’t structure itself that way, it’s good to try and break down the components and set yourself little tasks as you go.
  - bilsbie 4y ago
    Thanks. That’s an idea.
- thisismyswamp 4y ago
  There's no way to trick your brain into enjoying something it finds boring. You can only switch to something that it doesn't find boring, even if the final result is the same.
ralusek 4y ago
> in depth course that started right from the foundations—implementing and GPU-optimising matrix multiplications and initialisations—and covered from scratch implementations of all the key applications of the fastai library. I haven't taken the course, but that sounds like a horrible place to start a course on understanding deep learning. GPU matrix operations are literally an implementation detail. I think the proper way to teach deep learning "from scratch" would be: 1.) show simple example of regression using high level library 2.) implement same regression by writing a simple neutral network from scratch (explain going through, multiplying weights, adding biases, applying activation function, calculating loss, back propagation). 3. use NN on more complicated problem with more parameters and a larger training set, so that user sees they've hit a wall in performance, and now implementation needs to be optimized 4. at this point, say okay, our implementation of looping and multiplying can be done much faster with matrix multiplication on GPU, and even faster parallelized across GPUs on a network. If you're interested in that, here is an optional fork in the course that gets into specifics. Anything after this point will assume that implementation of NN calls will be using these techniques under the hood 5. move onto classification, q learning, GANs, transformers. 95% should have skipped step 4 and only revisited if they become interested in this for a specific reason. To start with it is crazy. It's like starting a course about flying by explaining how certain composites allowed us to transition from propellers to jets, and let's dive into how those composites are made.
- idf00 4y ago
  > I haven't taken the course, but that sounds like a horrible place to start a course on understanding deep learning. Sounds like you support the course author's decision to make this part 2 of the course series to be taken after part 1 is completed!
  - ralusek 4y ago
    Sorry for failing to clarify: > Three years ago we pioneered Deep Learning from the Foundations, an in depth course that started right from the foundations—implementing and GPU-optimising matrix multiplications and initialisations They're talking about how Part 1 starts
jph00 4y ago
Sorry folks for not contributing to this thread earlier - didn't realise this popped up on HN while I was sleeping! I'm Jeremy, and I'll be running this course. Lemme know if you've got any questions about it, or anything related to the topic of DL, stable diffusion, etc.
- frozencell 4y ago
  Thank you so much for doing this! Does the course mention sampling methods (Euler, LMS, DDIMS, etc.), what are they, how do you they work and relate?
  - jph00 4y ago
    Yes, but more from a "what do they do, and how do we code them" than "how are they derived mathematically". I think sampling methods might be the bit of this which is the hardest to fully understand given our prerequisites (just high school math) and may require a follow up course to cover more deeply.

258 points by wasimlorgat 4y ago

>> The course will be available for free online from early 2023. Anyone aware of an in-depth intro-level text-based explanation of Stable Diffusion that covers the whole pipeline, including training on an extremely limited dataset? Here’s an example, but open to suggestions too: https://huggingface.co/blog/stable_diffusion https://huggingface.co/blog/stable_diffusion

nyoomboom 4y ago
This year's Deep Learning Indaba had a tutorial on diffusion models in Jax: https://github.com/deep-learning-indaba/indaba-pracs-2022/tree/main/practicals https://github.com/deep-learning-indaba/indaba-pracs-2022/tr...
mpaepper 4y ago
I wrote some of that as a blog entry: https://www.paepper.com/blog/posts/how-and-why-stable-diffusion-works-for-text-to-image-generation/ https://www.paepper.com/blog/posts/how-and-why-stable-diffus...

hansworst 4y ago

This is exactly the kind of course I’ve wanted to do for some time now. Even before stable diffusion it felt like other media synthesis applications like StyleGAN were what I wanted to learn, but most machine learning courses focus on more traditional data science topics. Of course you can start with a more traditional course and then learn something like stable diffusion afterwards, but as a newbie it’s quite hard to figure out where to even start. A full-fledged course that takes you exactly where you want to go is a lot easier and I think it can help learners to stay motivated because they have a clear goal in mind. If I want to learn how to create cool images, I want to spend as little time as possible predicting housing prices in the Bay Area.

mudrockbestgirl 4y ago
> If I want to learn how to create cool images, I want to spend as little time as possible predicting housing prices in the Bay Area. I think that's somewhat of a dangerous mindset to have. If you want to create cool images you can use pre-trained models and high-level APIs without needing to understand any of the internals. But if you want to truly understand how these models work, you need to make effort to study the basics. Maybe not predicting housing prices, but learn the foundational math and primitives behind all of the components from the ground up (and the Diffusion models are a complex beast made up of many components). And getting an intuitive understanding of how models behave when you tune certain knobs takes much longer. Many researchers in the field have spent years developing their intuition of what works and what doesn't. Both of these are fine, but I think I think we should stop encouraging people to be in the middle. Have courses that that promise "Learn Deep Learning / Transformers / Diffusion models in 7 days!" but then go on and teach you how to call blackbox APIs, giving you an illusion of knowledge and understanding where there is none. I don't know if this applies to this specific course, but there are a bunch of those out there, and highly recommend staying away from those. I know it's a hard sell in this modern instant gratification age, but if you actually want to understand something you need to put in some possibly hard work.
- jan_Inkepa 4y ago
  > I don't know if this applies to this specific course, but there are a bunch of those out there, and highly recommend staying away from those. fast.ai do stuff pretty well. FWIW, I did one of their earlier free courses and, as a maths grad, got my fill of maths learning as well as my fill of practical 'doing stuff with ML' stuff. If I didn't have my plate full I'd probably pay the 500 quid or whatever to do this course now rather than wait for the free version.
- boredemployee 4y ago
  >> But if you want to truly understand how these models work, you need to make effort to study the basics. I was very confused by this in the beginning of my journey. I was trying to learn everything involved with ML/DL, but in the end everything is already implemented with APIs, and your boss doesnt care if you know how to implement a MLP from scratch or if you use Tensorflow. My (poor) analogy is: you don't need to know how a car works (or how to build one) in every detail to drive it. When I understood it, it was liberating.

neodypsis 4y ago

This course looks interesting. My only concern is that I don't have real experience with NLP. Anybody can recommend resources to get to speed on this pre-requisite? My NLP knowledge is very basic.

Version467 4y ago
The "Practical Deep learning" course from fast.ai has a section on NLP that's probably a good starting point.
- neodypsis 4y ago
  Thanks, I'll check it out.

bilsbie 4y ago

Is there any way to motivate myself to take a class like this? I keep turning to easy distractions like twitter and Pac-Man.

dagmx 4y ago
When I teach people, the most effective thing I find is to give them small usable projects at the end of each little milestone. That helps them have something they can use right away, which helps with a few things: - at each milestone, they have a distinct goal that’s reachable - they can understand concrete use cases immediately - they get a little dopamine hit if satisfaction of having completed something. Whenever you’re doing a course that doesn’t structure itself that way, it’s good to try and break down the components and set yourself little tasks as you go.
- bilsbie 4y ago
  Thanks. That’s an idea.
thisismyswamp 4y ago
There's no way to trick your brain into enjoying something it finds boring. You can only switch to something that it doesn't find boring, even if the final result is the same.

ralusek 4y ago

> in depth course that started right from the foundations—implementing and GPU-optimising matrix multiplications and initialisations—and covered from scratch implementations of all the key applications of the fastai library. I haven't taken the course, but that sounds like a horrible place to start a course on understanding deep learning. GPU matrix operations are literally an implementation detail. I think the proper way to teach deep learning "from scratch" would be: 1.) show simple example of regression using high level library 2.) implement same regression by writing a simple neutral network from scratch (explain going through, multiplying weights, adding biases, applying activation function, calculating loss, back propagation). 3. use NN on more complicated problem with more parameters and a larger training set, so that user sees they've hit a wall in performance, and now implementation needs to be optimized 4. at this point, say okay, our implementation of looping and multiplying can be done much faster with matrix multiplication on GPU, and even faster parallelized across GPUs on a network. If you're interested in that, here is an optional fork in the course that gets into specifics. Anything after this point will assume that implementation of NN calls will be using these techniques under the hood 5. move onto classification, q learning, GANs, transformers. 95% should have skipped step 4 and only revisited if they become interested in this for a specific reason. To start with it is crazy. It's like starting a course about flying by explaining how certain composites allowed us to transition from propellers to jets, and let's dive into how those composites are made.

idf00 4y ago
> I haven't taken the course, but that sounds like a horrible place to start a course on understanding deep learning. Sounds like you support the course author's decision to make this part 2 of the course series to be taken after part 1 is completed!
- ralusek 4y ago
  Sorry for failing to clarify: > Three years ago we pioneered Deep Learning from the Foundations, an in depth course that started right from the foundations—implementing and GPU-optimising matrix multiplications and initialisations They're talking about how Part 1 starts

jph00 4y ago

Sorry folks for not contributing to this thread earlier - didn't realise this popped up on HN while I was sleeping! I'm Jeremy, and I'll be running this course. Lemme know if you've got any questions about it, or anything related to the topic of DL, stable diffusion, etc.

frozencell 4y ago
Thank you so much for doing this! Does the course mention sampling methods (Euler, LMS, DDIMS, etc.), what are they, how do you they work and relate?
- jph00 4y ago
  Yes, but more from a "what do they do, and how do we code them" than "how are they derived mathematically". I think sampling methods might be the bit of this which is the hardest to fully understand given our prerequisites (just high school math) and may require a follow up course to cover more deeply.