11 ms·
Deep Learning Foundations to Stable Diffusion
- jay3ss 4y agoI just started part 1 and had no idea part 2 was coming out. This looks great
- tmabraham 4y agoI have worked on the course with @jph00 and it's been an incredible experience! This is one of the most cutting edge courses about deep learning and diffusion models. Highly recommend folks check it out! Link for more info: https://www.fast.ai/posts/part2-2023.html https://www.fast.ai/posts/part2-2023.html
- jph00 4y agoHi folks. Nice to see our new free (and ad-free) course here on HN! This course is for folks that are already comfortable training neural nets and understand the basic ideas of SGD, cross-entropy, embeddings, etc. It will help you both understand these foundations more deeply, since you'll be creating everything from scratch (i.e. from only Python and its standard library), and to understand modern generative modeling approaches. We study and implement a lot of papers, including many that came out during the course, which is a great way to get practice and get comfortable with reading deep learning literature. If you need to get up to speed on the foundations first, you should start with part 1 of the course, which is here: https://course.fast.ai https://course.fast.ai If you've got any questions about the course, generative modeling, or deep learning in general, feel free to ask!
- rjb7731 4y agoThanks Jeremy for putting this course out for everyone, looking forward to completing it.
- it_citizen 4y agoThank you Jeremy, it is an amazing course!
- jamestimmins 4y agoThanks Jeremy, this looks awesome. Do you think there's much value in learning much of the math in addition to the material in these courses?
- jph00 4y agoThe math is covered in this course. We look at both ordinary and stochastic differential equations, and at Langevin dynamics / score matching, as well as the needed probability foundations. We only cover the necessary bits of these topics -- just enough to understand how diffusion models work (e.g. we don't look at the physics stuff which is where Langevin dynamics came from). On the whole, we cover the needed math at the point where we use it in the course. But there's also a bonus lesson that focuses on just the mathematical side, if you're interested: https://youtu.be/mYpjmM7O-30 https://youtu.be/mYpjmM7O-30
- jamestimmins 4y agoExcellent, thanks for the thorough response!
- detrites 4y ago> since you'll be creating everything from scratch (i.e. from only Python and its standard library) Is this correct? I skimmed the notes for both parts 1 & 2 and they stated requiring PyTorch?
- jph00 4y agoYes, after we create something from scratch, we then see how to use implementations from libraries such as PyTorch. For basic operations like convolution, whilst we do create a GPU-accelerated version, it's not as fast as PyTorch/cudnn's highly-optimised version, so we use the PyTorch implementation for the rest of the course, rather than our handmade version. So that means that nothing is mysterious, since we know how it's all made, but we also see where and how to use existing libraries as appropriate. In practice, during the course we end up using PyTorch for stuff like gradients, matrix multiplication, and convolutions, and our own implementations for a lot of the stuff that's at a higher level than that (e.g. we use our own ResNet, U-net, etc.)
- detrites 4y agoAmazing, that's exactly what I've been looking for and what I'm sure many here will appreciate. Thank you!
- psychoslave 4y agoWow, you are really fulfilling the promise of a pragmatic approach. Awesome!
- matsemann 4y agoAh, I've been waiting for this, looking forward to going through it! While I appreciate your top-down approach and buy your arguments for doing it this way in previous courses, I also like this new bottom-up where one really learns what lies beneath.
- noob_eng 4y agoSomething of this sort is also done by Andrej Karpathy in his Zero to Hero Series:https://karpathy.ai/zero-to-hero.html https://karpathy.ai/zero-to-hero.html
- startup_99 4y agonot to derail the conversation... I came across your APL study session videos while exploring the other material. I used APL professionally for about a decade back in the early 80's. I am always pleasantly surprised when I see interesting work being done in APL. Interestingly enough I always thought APL would eventually evolve into a central language for AI. There were attempts at designing hardware-based APL machines way back when. Of course, much as the language, they were ahead of the technology of the times. I don't do much with APL these days. I do keep it around for quick calculations and exploration, far more so than doing any projects. In many ways, a thinking tool of sorts.
- pvitz 4y agoI have recently read "The Little Learner" and was quite amazed how much I knew already just because I know APL/J. With the right libraries, these vector languages would be perfect for neural network / deep learning tasks.
- tomcam 4y agoNot criticizing you, here, just asking a sincere question. It was obvious to me on first encounter that APL would never become widespread. Its character set was too abstruse for most programmers and at the time required a special monitor. Plus it seemed hard to maintain if you weren’t either a mathematician or a full-time APL developer. Your thoughts?
- bwanab 4y agoNot OP, but I'll say you're right, but for the wrong reasons. Having embarked on learning APL by doing a project in it, the character set becomes second nature in a few days time. No special monitor needed, at least on my Mac, just a new font and it all worked flawlessly. What caused me to hang up my glyphs were the inconsistencies and head scratching behavior of the language in so many corner cases. Working with very nice mentors I found that one could get around them, but because of the legacy of the language they have to be kept around to support existing code bases. I loved the notion of having a language that gave a first class experience with matrices, though, and after looking around the space, I finally came to Julia and have been very happy.
- wodenokoto 4y agoHow long does it take to go through the course? I know the on-demand are self paced, but are we talking “expect 1 lecture a week full time”?
- jph00 4y agoBased on the last time we did a part 2 course, time spent varied a lot. Some folks could do it in 10 hours study (plus watch time) per lesson, and some took a whole year off to do the two parts of the course (including doing significant projects).
- frabia 4y agoMaybe odd question, but: would you recommend taking this course if my goal is to build (and sell) products leveraging ML? (e.g. SaaS) As in, with the pace of improvements from other AI startups and general availability of their APIs (e.g. GPT-4), is there a specific advantage (aside from maybe cost) to learning to build my own models? Or is the course more suitable for people wanting to become ML engineers (or similar) and to find a job as such? Thanks in advance
- jph00 4y agoPart 1 would probably be best for that: https://course.fast.ai https://course.fast.ai . Lots of alums have gone on to do what you describe -- it's probably a good idea to have the level of understanding introduced there even if you mainly use external APIs. Part 2 is more for folks who want to go deeper in order to try new research ideas, optimize their models beyond standard approaches, etc.
- bravura 4y agoJust my two cents, as someone with 8000 academic citations and one AI SaaS exit, on AI entrepreneurship: The more time you spend on marketing, the better. We found that as our AI got worse, our product got better.
- frabia 4y agoThanks for your reply, though I'm not sure if I get your comment correctly. As in, I agree that marketing is super important but if I had an idea about a certain SaaS product that requires a certain ML model, I need to decide either to build it myself, or using somebody else's APIs. > We found that as our AI got worse, our product got better. Interesting. Could you elaborate on this too?
- bravura 4y agoI guess my last sentence was confusing. What I meant was that we fell into the common trap that scientists want to do as fancy science as possible when they leave academia and enter the startup world. Because there's an issue of pride and desire for uniqueness. Whereas good business is not like art, where the mandate is to be as creative and unique as possible. The analogy from sports (which I don't watch) applicable to business is that teams just copy each other's plays and focus on out-executing each other. So, specifically, our startup started with the idea that high-brow tech would be a key differentiator and give the best user experience. This is the common story trotted out by survivorship bias stories that make good tech news articles. Whereas making cuts to our R&D time and focusing on UI/UX and working with simpler science ultimately led to better product. From consulting, sales, and corporate work, one learns that the dirty secret of big-iron large tech companies is that all stuff sold as ML is just nicely packaged logistic regression. Or it was was five years ago. Nowadays I guess it would MAYBE be transformers, but the point being that off-the-shelf ML with solid data engineering work is what drives 99% of good products. Rarely is it truly innovative tech. I think Pete Skomoroch was the one that joked: "People say I'm a data scientist. I'm actually a data plumber." I guess one could take this lesson from science. What you learn from a good PhD advisor is: a) read the latest work b) note the simple baseline approach constantly trashed as scoring 2% worse than the sophisticated intricate new things proposed and c) implement the simple baseline. Achieve impact not by hillclimbing on a standard metric but define a new problem or arbitrage insights from adjacent fields, etc.
- _giorgio_ 4y agoI hope that one day you will do another part about LLM, in pytorch, that would be great :-) Thanks for this material.
- tmabraham 4y agoThere are plans for an LLM course but very early ideas, stay tuned!
- indigoabstract 4y agoThank you for putting out this course!
- sine_break 4y ago[dead]
- howon92 4y agofast.ai is hands down the most beautiful learning material for deep learning / ai
- andrewm4894 4y agoLove all his courses!
- nbzso 4y agoThis is cool, but I have a fundamental question. Why learn machine learning when all the machines will learn how to create and control themselves and powerful elites will offload the burden of governments to control us? Or we race ourselves to fulfill the last jobs on earth? Is this progress for humanity or enslavement and the end of our species? I am serious. This question is honest. Maybe my IQ is too low to understand the logic of this innovation. Maybe I have seen too much tech implementation doing bad stuff to humans, who knows? Please, enlighten me.
- jstx1 4y agoThis question isn't specific to machine learning - if you've given up already, don't bother learning anything and see how it goes.
- nbzso 4y agoI never give up. Just asked for information, to fight my own bias. Obviously there is a world outside which will have analogue and mechanical parts. But this is not the question. The question is why to cut your wrists with a blunt object and stream the event for validation is suddenly a rational behavior?
- andrewm4894 4y agoAre you a LLM or something? :)
- nbzso 4y agoYes. Like you.:)
- deleted 4y ago[deleted]
- psychoslave 4y agoI guess that the fear, be it well founded or delusional, is fueled by the idea that a hammer for example have a lesser potential to inadvertently beat humans at civilizational level.
- hkab 4y agoreally love these hands-on tutorials, d2l and fast.ai has helped me a lot!
- emrah 4y agoPart #1: https://course.fast.ai/Lessons/lesson1.html https://course.fast.ai/Lessons/lesson1.html
- novideolike123 4y agoWould it be possible to have a transcription of the videos? I have never been able to follow a video for learning, for me the video format if like someone else controlling the way I learn.I much prefer to be able to design my own schedule in text. By using text I am able to skim quickly and select only those parts I consider important. Maybe other people are able to do that with video but I am not in that group.
- jph00 4y agoYes every lesson has a complete transcript written by our wonderful community members. You can access through the usual YouTube interface.
- crakenzak 4y agoIs there any chance that the transcripts can be published on the website? I'd love to be able to search through them. love the course btw!
- jph00 4y agoIn youtube click "show transcript". It's searchable. Click in it to jump to that part of the video.
- stevesimmons 4y agoThis would be ideal. I much prefer seeing everything in text form, in the same place, without having to jump out to YouTube. I can read many times faster than listen, and more easily skim back if I realise I don't understand something.
- jph00 4y agoThe full transcripts are available here in plain text form: https://github.com/fastai/course22p2/tree/master/summaries https://github.com/fastai/course22p2/tree/master/summaries
- 4y ago
- mattyyeung 4y agoI just finished this! My thoughts: I recommend it. I feel I can now read an arbitrary paper, frown a lot, and eventually understand what it's talking about - to the point where I can implement my own buggy version. And hey, I built my own stable diffusion!! I found the previous version of this course[1] to be a good complement: it's older (predates SD) but I feel it explains core concepts slightly better. Very understandable given how the close to the bleeding edge this new version is... Perhaps an even better complement was Karpathy's famous course[2] - similar material but builds towards GPT instead of SD. The fastai coding style is somewhat esoteric (to me) so it was helpful to contrast with Karpathy's more familiar style. I recommend doing both courses. Also I believe the fastai folks are planning a part 3 which covers LLMs; looking forward to that. Concepts from part 2 helped my hobby project, a 7-day forecast of renewable electricity and power price[3]. Feels pretty great to have built my own Stable Diffusion and GPT! I am grateful to Jeremy and Andrej. [1] https://course19.fast.ai/part2 https://course19.fast.ai/part2 [2] https://karpathy.ai/zero-to-hero.html https://karpathy.ai/zero-to-hero.html [3] https://greenforecast.au/ https://greenforecast.au/
- wodenokoto 4y agoDo you recommend doing fastai and karpathy simultaneously or one at a time?
- mattyyeung 4y agoDepends. Some would benefit from simultaneous, others sequential. I did them in "chunks" starting with fastai but that was more driven by the release schedule. Personally I'd recommend trying both and seeing which style you prefer, focus on whichever one makes your more excited to get your hands dirty and play with stuff.
- hoseja 4y agoHow long did it take you?
- mattyyeung 4y agoI didn't really track... apparently there's 30-ish hours of video, but lectures are just the beginning. The real learning happens when you play and build. The first lesson was released in October I think.
- matsemann 4y agoI just used fastai last week for an experiment at work. I have no to little DS experience, but quickly skimmed through part1. Quite easily I managed to use a pretrained resnet to do some quite astonishing things on completely unrelated pictures we have. And it only took like 10 epochs at ~4 seconds each, so less than a minute of training and it was already quite good. What I did was that we have taken pictures of things during a process. At that point, I know what the volume should be. So used ~thousand of those images to train a regression model to estimate volume of the thing in the image. Then I run the model on other images and compare what the model thinks is the volume compared to what it should be. When it's off, the algorithm is mostly correct and we have an error on our side. Quite astonished about how quick and easy this was to do with the high level api of fastai. Took me 2-3 days.
- _giorgio_ 4y agoWhy would I want to learn fast.ai... I feel that while this course is surely superb, I don't want to invest time in a new framwework. What's your opinion? We've got lightning, fast.ai, and another one that I forgot about... why?
- jph00 4y agoThis course doesn't use the fastai framework. But to answer your question -- the reason IMO to learn the fastai framework (or indeed any software which incorporates a significant amount of novel research) would be to learn about the new ideas that it introduces, in order to become a more effective and informed practitioner. And if you find you like it, you may even decide to use it.
- h4ch1 4y agoDid you create this account just to hate on the course? xd
- deleted 4y ago[deleted]
- amelius 4y agoHow much math does the reader need under their belt? Is high-school math sufficient?
- jph00 4y agoYes high school math is sufficient, as long as you did basic calculus (derivatives and the chain rule) and probability. We cover stuff that's well beyond high school math, but it's introduced when it's used, and we link to external resources for more details as needed.
- tempaccount420 4y ago> as long as you did basic calculus (derivatives and the chain rule) and probability How much additional effort would it require to incorporate these topics into your course, potentially as supplementary content? This could make the course more accessible to students from countries with less comprehensive educational systems.
- andy_ppp 4y agoThis looks awesome, however I probably need to do a bootcamp with real people to compete and have comradery with to actually retrain on the basics in this. It doesn't matter where in the world this is (I like to travel), but does anyone have a recommendation for in person bootcamps on AI?
- cubefox 4y agoUniversities are often open to sit in classes, you can act it you were an enrolled student.
- deleted 4y ago[deleted]
- tmabraham 4y agoIf you're interested in taking this course, you can check the forums and/or Discord where people will organize study groups going through the course. Some of the study groups organized are in-person, some are virtual. These are usually great opportunities to study with some real people at the same time and get that camaraderie. https://forums.fast.ai https://forums.fast.ai https://discord.gg/YHtEBwzV https://discord.gg/YHtEBwzV
- andy_ppp 4y agoFantastic, I’ll sign up and have a look! Thanks
- smusamashah 4y agoIs there a stablediffusion.cpp port like whisper.cpp and llama.cpp?
- cube2222 4y agoThis is awesome! I loved the previous FastAI courses, and the top-down approach taken in them is really fun to go along with. I guess now it's time for an LLM-themed course?
- jackallis 4y agomy nephew who is going to college this fall at east coast said, "what's the point of CS major anymore" and i asked "what do you mean" and he goes " ChatGPT". This is very cynical coming from 16 year old; however, got me thinking about impact of Chatgpt on peoples' mind about learning ML/AI if chatgpt can do it all!
- willis77 4y agoThe year is 1900. College-bound nephew says "what's the point of a physics major anymore?" I asked "what do you mean?" and he goes "Newton's Laws".
- rasmus1610 4y agoIn my opinion you still need to have a grasp of the concepts, because ChatGPT still hallucinates stuff and messes stuff up.
- knicholes 4y agoI've watched all of the fastai tutorials. I've screwed up by not going through and actually writing the code, but I hope to do it again and do it right this time. My biggest question is if this the whole part 2 course or still the 2-3 preview courses released around last November?
- rasmus1610 4y agoNo this is now the whole part 2 course, so lesson 9 - 25.
- vrglvrglvrgl 4y ago[dead]
- ttul 4y agoI audited this course - I say “audited” because I didn’t have time to do the homework; I just watched the lectures and spent what time I could playing around in Colab building some toy models. This is very much a full time university level course and you should treat it that way if you intend to complete it properly. You will need to reserve 10 hours per week for the homework - at least. I enjoyed how Jeremy and his teaching assistants step through every detail so that you can understand how these fantastically complex systems actually work, from the ground up. Nothing is glossed over or taken for granted. Much of the course material is building an AI programming framework from scratch. It’s well worth the hassle; I now at least have an intuition about how these systems work, rather than a glossed over impression with many holes. The other thing I greatly appreciated were the insights from other classmates in the forums. Some people way smarter than me took this course and Jeremy would incorporate their flashes of insight into the lectures. I give this course a 10/10 and I hope my life gets a little easier so that I can get back to the lessons and finish the homework one day.
- Avalaxy 4y ago> You will need to reserve 10 hours per week for the homework - at least. Is that for part 2? Or also for part 1? How long did it take you, doing it 10 hours per week?
- rasmus1610 4y agoFor both. Although you will be much faster in part 1 if you know the basics of deep learning. But then again you can go so much deeper on certain topics and explore maybe some tangents to the content of part 1.
- ttul 4y agoI didn't take part 1, so I can't comment intelligently on this.
- Fede_V 4y agoThe fast.ai courses are exceptionally good for getting started and learning the concepts, but, once you've mastered an intuitive understanding of the techniques, I highly encourage people to dig deeper into the concepts - specifically, reverse SDEs, stochastic calculus, optimization, score matching. Jeremy does an excellent job making the content approachable, but, if you want to go beyond being an excellent practicioner, you cannot skip the theory bits.
- idf00 4y agoFYI this course does cover a lot of intuitive understanding of techniques, but it also does cover this theory. Everything is done from the foundation level, and all of the stuff you mentioned is taught and implemented from scratch as part of the course. Of course there's loads more to do on gaining intuition, and loads more theory to be learned once the course is done! Only so much can fit in a single course :)
- crimsoneer 4y agoJeremy and Fast.AI should get bloody medals.
- sswam 4y ago[dead]