10 ms·

Practical Deep Learning for Coders 2022

423 points by tmabraham 4y ago

idf00 4y ago
Fantastic course - Fastai courses are a must for anyone looking to learn Deep Learning/ML.
jph00 4y ago
Hi folks - I'm the creator/teacher of this course. I'd be happy to answer any questions that you have about the course, learning deep learning in general, or the state of deep learning in 2022.
- knicholes 4y ago
  Thank you, so much, for enabling us mortals the power of ethical, modern AI. Your work, and the work of your colleagues, has brought so much good to this world. It wasn't until the last few years I saw people start freezing the model and just fine tuning the last layers. I've watched presenters from flamingo and imagen talk about their similar approaches. I heard it here first, at fastai.
  - jph00 4y ago
    Haha yes - it's been wonderful to see how (eventually) the deep learning world has taken to transfer learning!
    - iTokio 4y ago
      What’s your take on meta learning?
- perfopt 4y ago
  Thank you for creating this course. I started out on Tensor Flow but seeing this material I am in two minds whether I should abandon my TF book and start this one or save it for later. Most likely I am going to dive in :-)
  - jph00 4y ago
    Both the Aurélien Géron and François Chollet TF books are absolutely terrific, and everything you learn from them will be extremely useful in becoming a deep learning practitioner, regardless of what framework you end up using. So if you've started with one of those books already, keep it up! :) The fast.ai course would actually be a pretty good addition to either book, since you'll get to see a whole different way of doing things, which might be useful to understanding what's going on.
    - perfopt 4y ago
      Thank you
  - rg111 4y ago
    My suggestion would be to learn all the stuff from this course, using fast.ai library, and then gradually move towards PyTorch. fast.ai is a fantastic educational resource and a great way to approach solving problems. But the library itself is lacking, and if you are an experienced programmer, when building real-life projects, you will be frustrated with fast.ai library. The goal, IMO, should be learn from Jeremy Howard, s great instructor, communicator; learn his attitude, and then move to PyTorch (keeping the attitude, the knowledge, and the lessons with you.)
    - mloncode 4y ago
      I am an experienced ML Engineer of 10 years and have worked at several large flagship tech companies. I do not agree that fastai is not appropriate for real-life projects. If you know the fastai library well, you know its a layered api on top of pytorch, which allows you to customize things to your needs quite easily. For example, it is fairly straightforward to get any pytorch model out of a Learner object. Furthermore, lots of care has been taken to keep the apis very consistent with pytorch as well. It's also the only library I know of that consistently bakes in best practices like super convergence techniques or making things like test time augmentation very seamless. Many libraries lag behind fastai 1-2 years in this regards, and frankly it can be frustrating to use other frameworks sometimes. There is a slight learning curve, for example to learn the DataBlocks API or the callback system, but once you really understand what is happening you will understand how nice the API is and how well engineered it is. Side note: Regarding being an experienced software engineer, I highly recommend digging into how the python language was extended for this project (fastcore) and the development workflow used (nbdev), which I think could be interesting for those software engineers you mention as well as heighten your understanding of the ecosystem of tools.
- notpublic 4y ago
  First of all, a big thanks! What is your take on the current state of autonomous driving? Do you think we can achieve "full autonomy" with the technology we have currently? Any new advances in DL that you are excited about?
  - jph00 4y ago
    Honestly I'm not an expert on autonomous driving so I'm not sure I have great insights there. I do know quite a bit about computer vision however so feel qualified to comment on that bit -- I suspect the decision by Tesla to only use CV, and not LIDAR, may turn out to be a mistake. I don't see any reason why we couldn't achieve full autonomy with our current tech including LIDAR, although I don't know if it can be achieved at a practical latency and power budget. The new advances in DL I'm excited about are things I show in the class: the accessibility of modern NLP thanks to the Hugging Face ecosystem; the power of ConvNeXt for even better computer vision models; the way Gradio and HF Spaces makes it trivially easy to get a working prototype application using DL online. I'm also excited about hosted models and applications like GPT-3, DALL-E, and Codex. All the illustrations on our course website are from DALL-E, for instance!
- darepublic 4y ago
  Thanks for creating this fantastic content, I'm excited to give the 2022 course a look. It's an exciting time for AI. I'm curious about your thoughts on gpt3 and also the state of the art in computer vision, and object detection. All the best
- sooheon 4y ago
  I remember you were bullish about Swift a few years ago. What's your current view on non-python deep learning?
  - ngcc_hk 4y ago
    Interested that as well, especially the old school lisp to this new Ai.
  - jph00 4y ago
    I'm disappointed that Google shut down the Swift for Tensorflow project, because I do think Swift is a great option for deep learning. In some ways Jax is almost "non-python deep learning" since it's treating Python more like a DSL for the XLA backend. Normal Python code doesn't work in Jax. It's a pretty reasonable compromise since you still get all the benefits of the Python ecosystem. Julia seems like it has the best foundations for deep learning, since everything can be written directly in Julia. But it doesn't have a great ecosystem as a general programming tool. F# might turn out to be a good option.
- sriram_malhar 4y ago
  Thank you so much for this course. I plan to go through it properly. I have a search problem of my own and I have had a hard time applying what I have learnt (including the coursera DL specialization). The chief characteristics are: (a) It is a fuzzy search of a corpus that is in a non-English language. (b) The search should be able to run on a mobile phone _offline_. Is this possible? Can training be done elsewhere and transferred to TinyML or some such? What would be a good forum to go seeking answers?
  - devnonymous 4y ago
    If the volume of data fits on a mobile phone for it to be offline, perhaps you don't need deep learning?
    - sriram_malhar 4y ago
      Perhaps. What alternative would you suggest? The search terms are fuzzy, and there are too many variants (not exactly misspellings) for me to encode them explicitly. So I thought I'd rather learn from a crowd-sourced corpus. In my case, I don't want the model to be general. I can afford for it to be like a database index, tailored to that data.
  - leobg 4y ago
    Have you tried... a) BM25 after some preprocessing (lemmatization etc.) b) fastText / GloVe (possibly weighted by BM25) The results can be surprisingly good. Often no need to bother with big language models or GPUs.
    - sriram_malhar 4y ago
      As far as I understand, BM25 is not for fuzzy searches. For a bit more context, the search terms are in the English script, but the words are basically the closest-sounding transcriptions of sounds in various Indian languages. Different people may render the same word differently in an English transcription. But there's enough crowd-sourced data to account for the ways in which words can vary. For the same reason, GloVe is of not much use to me.
- nindalf 4y ago
  I tried the 2021 course but I didn't finish. I think the biggest friction for me was using the remote machine. I wasn't able to make steady progress like I do with my offline learning projects. How far away is the fast.ai from working on a Mac? PyTorch recently gained support (https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/ https://pytorch.org/blog/introducing-accelerated-pytorch-tra...) but that's only the start. Is this something that is being worked on?
  - jph00 4y ago
    The good news is that every lesson in this course is actually run on Kaggle Notebooks, which is a free cloud environments including GPUs. So you don't need to set up anything and it runs on any computer with a modern web browser! Mac support for all the libs used in the course will probably continue to improve in the coming months and there should be no reason you won't be able to run the stuff for the course locally on a Mac at that time. Having said that, even the M2 trains deep learning models much slower than even the free NVIDIA GPUs provided by Kaggle. So you'd only want to use local development for the smallest and simplest models. (The course shows how to train models that are fairly cutting edge and some take a while to train even on modern GPUs, so they wouldn't be a good fit for a Mac.)
    - nindalf 4y ago
      Thanks Jeremy, I'll give it another go.
    - uwuemu 4y ago
      I think last time I tried this, I kinda gave up as soon as it got to the point of hand-waving hardware and telling you to run notebooks on a third party's web service. How is that democratizing AI? It's the very opposite! Not good. What is the intended audience here? Uni-level students will learn most of these within their programs if they're interested in AI. So they're not really it. Is the intended audience "coders"? If so, most of these "coders" will have to somehow get their employer on board (typically a corporate entity very much NOT interested in "coding" something in a 3rd party ecosystem) or do it themeselves. Hence, I want to take an RTX 3080+, 64+ gb of ram, big ass SSD and I want to get through the training. Not learn some basics on somebody else's platform (come on, even from the pov of OS, running both training AND notebooks on some 3rd party's private platform is so against the idea of open source...) and call it a day. What use is that? That may be enough if you want to be a cog in somebody else's machine, but not if you want to do something useful by yourself (I say "may" because big tech generally isn't interested in your "mad AI skillz" unless you also have a student loan backed piece of paper proving you successfully learned that for the last couple of years). There will always be smart individuals and talented small teams that can successfully integrate AI into their products, but it's not thanks to the courses like this. If you're going to aim at coders, there has to be clear path demostrated from the beginning to the end. From starting up your first notebook on your local dev machine and running training on your local training machine to setting up inference in the final app (.net app or whatever)
- mkl 4y ago
  How many hours do you think this course would take for an experienced developer with plenty of applied maths but ~no machine learning? How easy is it to do the course on my own hardware rather than cloud notebooks? Would that make it closer to practical deployment?
  - gandalfgreybeer 4y ago
    Not him nor will I talk about his course, but I’ve been in the field a reasonable amount of time (both on the academia and industry side). Honestly, applied maths will get you a long way and make it easier to digest the concepts (you might just see them as repackaged problems depending on your mileage). If you have good programming skills and discipline you practically have most of what you need. Re the course, I just skimmed it and I think you can do most things on your own hardware but if you will actually use this for something practical (not just for you or a side project), being familiar with cloud tools is a big thing especially once you scale.
    - qwrshr 4y ago
      out of curiosity, how much applied math should one bone up on? (Obviously the more the better, but diminishing marginal returns and all that.)
      - cinntaile 4y ago
        None, just look things up as you go along if there is something you don't understand. You're likely not going to bother understanding how the optimization functions work or how the cost functions actually work anyway. They're implementation details in most cases.
      - gandalfgreybeer 4y ago
        Bare minimum is basic calculus, basic linear algebra and basic statistics. By basic, I probably mean first courses for those in most undergraduate programs. I disagree with needing none and just going along as needed. That’s how you have machine learning models that look like they work but you don’t understand why they work so there might actually be problems.
- sabertoothed 4y ago
  I hope you stay as humble as you have been. But you're my personal hero. It is just incredible what you have done for the world.
- Oreb 4y ago
  Is there a new version of the book? All the links I find lead to the 2020 edition.
  - jph00 4y ago
    No, the book is continually updated for each reprint, but there isn't a separate edition.
- fareesh 4y ago
  Hello Jeremy do you have any specific advice on tackling ASR using fast.ai?
- thefreeman 4y ago
  As someone who much prefers reading over watching videos, do you think I would miss much by just going through the book in the github repo? Or are those notebooks mainly supplemental to the videos.
  - jph00 4y ago
    I'd say it's the other way around - the videos are kinda supplemental to the book. The book has a lot more content, but doesn't have the interactive explanations in the course. Also the book is a couple of years old so is missing the more recent developments (but the principles haven't changed).
- cweill 4y ago
  Hey Jeremy, i just want to say that I love your course and the way you teach. I refer everyone to the Fast AI in my YouTube videos on getting started with machine learning. Please keep up the great work!
- __rito__ 4y ago
  Jeremy, hi. I have one question, and one only. Please answer: Second part, when?
- kachau 4y ago
  Will it help me more or achieve more out of learning this course as compared to just directly using GPT-3 or Dall-E as paid user?
- baladre 4y ago
  .b
- baladre 4y ago
  En
- baladre 4y ago
  .m
ben-coman 4y ago
Watching from afar the great advances in machine learning over the past few years with AlphaZero, GPT3, DALLE2 I felt it important for me to start understanding what is going on under the hood. Having just completed the private pre-release of the course run through USQ, as my first foray into machine learning this was a great introduction that had me quickly produce a working image classification system. The videos are packed really with insightful rid-bits about practical approaches to iterating quickly to understand the data better to produce better results. Very much recommend the course.
perfopt 4y ago
This is awesome. One question I have always had - is the research on applying DL for images the most developed compared to other things? Even DL used for audio processing (classification, separation etc) seems to convert audio to spectral graphs and apply DL to that. Changing a problem to be expressed as image inputs will be an advantage when using DL as a solution. Would you agree ?
- perfopt 4y ago
  Oops dumb question. Watched the first video and got my answer.
- tmabraham 4y ago
  Good question! I think a major reason for this is because of transfer learning. For computer vision, there are many good pretrained models that were trained on huge datasets (like ImageNet) that can be fine-tuned for custom tasks. Other fields often do not have such pretrained models and huge datasets to work on, so it turns out transforming a dataset into an image dataset and fine-tuning a pretrained model works better than training from scratch.
- nmfisher 4y ago
  Working with a spectrogram is definitely similar to working with an image, and it's interesting to think why that's the case. Take convolutional models, for example. Very effective for working with images because they're (a) parameter efficient, (b) learn local/spatial correlations in input features, and (c) exploit translational invariance. As an oversimplification, we can train models to visually identify "things" in images by their edges. If you think about what's going on with an audio spectrogram, you can see the same concepts at work. There's local/spatial correlation - certain sounds tend to have similar power coefficients in similar frequency buckets. These are also correlated in time (because the pitch envelope of the word "yes" tends to have the same shape), and convolutional models can also exploit time-invariance (in the sense that convolutional models can learn the word "yes" from samples where the word appears with varying amounts of silence to the left and right). That being said, the addition of the time domain makes audio quite hard to work with, and (usually) not as simple as just running a spectrogram through a vanilla image classification model. But it's definitely enlightening to think about how these models are "learning".
perfopt 4y ago
Can one do these lessons in any order? For example, do CNN first then jump back to NLP. Or skip the implementation from scratch because I have done a similar one in another course.
- jph00 4y ago
  They're designed to be done in order, but yup if you know how SGD works, for instance, you could certainly skip over that bit. The videos all have youtube timestamps, so if you drag the scrollbar you'll see what each section is about. Or you could do those bits at 2x speed in case there's some concepts there you haven't seen before. The NLP lesson could possibly work reasonably well standalone if you already know some DL basics, since it uses a different framework (Hugging Face) to the earlier lessons. The CNN lesson would probably largely make sense if you already understand multi-layer perceptrons, since it mainly shows how a convolution is just a special case of sparse matrix multiplication.
ramesh31 4y ago
You guys really are the best. Thanks for all the hard work.
alexcnwy 4y ago
The first Fast.ai course back in around 2016 changed my life. I was studying a masters in statistics and computer science that had 1 neural networks lecture and nobody knew anything about deep learning. Fast.ai and Jeremy’s teaching style helped me start playing with deep learning models really quickly and I changed my thesis topic to computer vision. I ended up consulting on the topic and doing various startups leading to the startup I’m working on now which just finished YC (AiSupervision W22). I doubt be here without fast.ai. I highly recommend and appreciate all the work that Jeremy and the rest of fast.ai do!
rg111 4y ago
There are too many poor design decisions in the fast.ai library. One should invest too much time just for the sake of learning the library's weird API, and then using it. Doing something custom is too difficult, in contrast to Jax, PyTorch, and even (poor library) TensorFlow. The coding practices are whimsical. The codebase wouldn’t pass code review in any respectable company. Variable namings are weird and super-problematic. I fully stick to what I said. Learn techniques, best practices, and, most importantly, Howard's attitude. Then take them with you and move onto something like PyTorch. Howard is great with one problem: he kinda hates math. It might also seem that he ends up promoting anti-intellectualism.
- deleted 4y ago
  [deleted]
- sabertoothed 4y ago
  I deleted an earlier, angrier comment of mine. Can you explain this last sentence (which I understand to be insulting and without basis): > Howard is great with one problem: he kinda hates math. It might also seem that he ends up promoting anti-intellectualism.
  - rg111 4y ago
    He says repeatedly "You don't need math", and stuff like that. This is not insulting. That man is my hero, and I deeply respect him. But his 2019/20 course was riddled with such statements. He repeatedly said that one doesn't need math, and showed tools like drawing math symbols on a website to learn their names and ride on that. No further math needed. It's like you can wing it in Deep Learning without learning Math. His behavior throughout the course reinforced this attitude. It is harmful for new learners. But I am fortunate that I didn't learn from that, but learned from some successful alumni example that Howard gave. One woman who was also a musician ('19/'20), she made it big, but Howard mentioned that she did the Ng course, and also read the Goodfellow book. So, I took the cue, and did DL the proper way. Anybody I know in DL made it because they know the Math. There are some influencer types in fastai community who has 10ks of followers and shills stuff and do media stuff. Other than that 1-2 people, everyone who made it in DL, did it because they knew the math. So, I think that people might get the wrong idea hearing from Howard that "you don't need math". This is one fault I find. It's not like I dislike him. I like the rest of him. I love his attitude on almost all other things. I love Jeremy Howard, and he is my hero.
jackallis 4y ago
what is/will be the state of deep learning in 2022 or next 3-5 years? you hear/read so many news/articles in HN about decline of DL. Is that so?
- davidatbu 4y ago
  I mean, just looking at OpenAI and Deepmind, they have relatively recently released break-through models for which building upon and extending can be done in relatively straightforward ways (DALLE 2, GPT-3, AlphaFold, OpenAI Codex, ...etc), so I don't think DL will "decline" any time soon ...
- kache_ 4y ago
  That's like saying "watch the decline of C++" while using javascript What does javascript run on? Most ML advancements will use DL at the core in interesting ways.
publicdaniel 4y ago
I am so grateful the FastAI team exists. It wasn't until I discovered their "Machine Learning for Coders" course that I really started to grok ML. I was in grad school trying to pivot my career from finance to data science. I didn't come from a computer science / math background and things just weren't clicking for me. I remember feeling angry, embarrassed, dumb, and overall that I wasn't smart enough to learn this stuff -- I was incredibly discouraged and felt that I didn't belong there. I was lucky enough to stumble across one of the course videos on YouTube (thanks recommendation algorithm!), and the rest is history. The amazing thing about these courses is how simple Jeremy (and team) are able to make machine learning. I didn't need to understand python dependency management in order to learn how to train an really good image classifier. Their approach helped me have lots of little wins, gave me confidence, and helped build the motivation to slog through the harder stuff when I needed to. From the bottom of my heart, thank you @jph00. You changed my life immeasurably for the better. I learned that I AM good enough, I AM smart enough, and I CAN do hard things... I just had to find the right way to learn them. Your courses completely changed my perspective on what was possible for me and opened the door to some of my life's greatest passions.
- wittycardio 4y ago
  - BeetleB 4y ago
    Lots of people who've taken the Fast.ai course have similar things to say. It's commonly said it's the fastest way to get into DL.
    - hello34 4y ago
      Yeah just wanted to say, I am also one of the persons who has immensely benefited. So it may sound like paid, yet lot of people have immensely benefited like people from India, Nigeria, etc.. Check this article[1] to know a bit about philosophy of fast.ai and why it's so popular [1] https://future.com/the-rise-of-domain-experts-in-deep-learning/ https://future.com/the-rise-of-domain-experts-in-deep-learni...
suhail 4y ago
I re-did the course with this 2022 version. Highly recommend it :)
__rito__ 4y ago
I haven’t seen this course content yet, but fully did the 2019 version. Extremely grateful to have found it. Changed the course of my life. I can vouch for it's quality. Jeremy is an excellent instructor. So much clarity in his teaching! I love that this is a hands-on course, and there are ZERO hand-wavings. I also really like the top-down approach of teaching. Now, whenever I try to communicate something or teach someone, I try to do it top-down. And I have Jeremy to thank for that. Currently, I am attending his APL study group and having a blast! Only question for @jph00 is: second part, when?
- newsoul 4y ago
  Yes, updated second part of the course when? Any chance you will shift from Swift to Julia?
anakaine 4y ago
The course uses PyTorch, which despite being a big played still has many issues running g on any of the latest AMD cards. Windows not supported on AMD cards Navi series cards not supported in general. Heavily biased towards CUDA, despite AMD cards drivers being open sourced far more than Nvidia cards. Remote machines and kaggle notebooks go some way to improving these limits for the course. I'm complaining a bit more in general here, I think.
newsoul 4y ago
Hello Jeremy. Thanks for this great content. I have gone through your entire course and learned a ton from you. Now moving on from here, do you have any resource recommendation where I can dive deeper into machine learning and deep learning theory? And also any resources to become a much much better programmer? I am currently working in as an assistant in a research lab. My coding skills are not that great.

423 points by tmabraham 4y ago

Fantastic course - Fastai courses are a must for anyone looking to learn Deep Learning/ML.

Hi folks - I'm the creator/teacher of this course. I'd be happy to answer any questions that you have about the course, learning deep learning in general, or the state of deep learning in 2022.

knicholes 4y ago
Thank you, so much, for enabling us mortals the power of ethical, modern AI. Your work, and the work of your colleagues, has brought so much good to this world. It wasn't until the last few years I saw people start freezing the model and just fine tuning the last layers. I've watched presenters from flamingo and imagen talk about their similar approaches. I heard it here first, at fastai.
- jph00 4y ago
  Haha yes - it's been wonderful to see how (eventually) the deep learning world has taken to transfer learning!
  - iTokio 4y ago
    What’s your take on meta learning?
perfopt 4y ago
Thank you for creating this course. I started out on Tensor Flow but seeing this material I am in two minds whether I should abandon my TF book and start this one or save it for later. Most likely I am going to dive in :-)
- jph00 4y ago
  Both the Aurélien Géron and François Chollet TF books are absolutely terrific, and everything you learn from them will be extremely useful in becoming a deep learning practitioner, regardless of what framework you end up using. So if you've started with one of those books already, keep it up! :) The fast.ai course would actually be a pretty good addition to either book, since you'll get to see a whole different way of doing things, which might be useful to understanding what's going on.
  - perfopt 4y ago
    Thank you
- rg111 4y ago
  My suggestion would be to learn all the stuff from this course, using fast.ai library, and then gradually move towards PyTorch. fast.ai is a fantastic educational resource and a great way to approach solving problems. But the library itself is lacking, and if you are an experienced programmer, when building real-life projects, you will be frustrated with fast.ai library. The goal, IMO, should be learn from Jeremy Howard, s great instructor, communicator; learn his attitude, and then move to PyTorch (keeping the attitude, the knowledge, and the lessons with you.)
  - mloncode 4y ago
    I am an experienced ML Engineer of 10 years and have worked at several large flagship tech companies. I do not agree that fastai is not appropriate for real-life projects. If you know the fastai library well, you know its a layered api on top of pytorch, which allows you to customize things to your needs quite easily. For example, it is fairly straightforward to get any pytorch model out of a Learner object. Furthermore, lots of care has been taken to keep the apis very consistent with pytorch as well. It's also the only library I know of that consistently bakes in best practices like super convergence techniques or making things like test time augmentation very seamless. Many libraries lag behind fastai 1-2 years in this regards, and frankly it can be frustrating to use other frameworks sometimes. There is a slight learning curve, for example to learn the DataBlocks API or the callback system, but once you really understand what is happening you will understand how nice the API is and how well engineered it is. Side note: Regarding being an experienced software engineer, I highly recommend digging into how the python language was extended for this project (fastcore) and the development workflow used (nbdev), which I think could be interesting for those software engineers you mention as well as heighten your understanding of the ecosystem of tools.
notpublic 4y ago
First of all, a big thanks! What is your take on the current state of autonomous driving? Do you think we can achieve "full autonomy" with the technology we have currently? Any new advances in DL that you are excited about?
- jph00 4y ago
  Honestly I'm not an expert on autonomous driving so I'm not sure I have great insights there. I do know quite a bit about computer vision however so feel qualified to comment on that bit -- I suspect the decision by Tesla to only use CV, and not LIDAR, may turn out to be a mistake. I don't see any reason why we couldn't achieve full autonomy with our current tech including LIDAR, although I don't know if it can be achieved at a practical latency and power budget. The new advances in DL I'm excited about are things I show in the class: the accessibility of modern NLP thanks to the Hugging Face ecosystem; the power of ConvNeXt for even better computer vision models; the way Gradio and HF Spaces makes it trivially easy to get a working prototype application using DL online. I'm also excited about hosted models and applications like GPT-3, DALL-E, and Codex. All the illustrations on our course website are from DALL-E, for instance!
darepublic 4y ago
Thanks for creating this fantastic content, I'm excited to give the 2022 course a look. It's an exciting time for AI. I'm curious about your thoughts on gpt3 and also the state of the art in computer vision, and object detection. All the best
sooheon 4y ago
I remember you were bullish about Swift a few years ago. What's your current view on non-python deep learning?
- ngcc_hk 4y ago
  Interested that as well, especially the old school lisp to this new Ai.
- jph00 4y ago
  I'm disappointed that Google shut down the Swift for Tensorflow project, because I do think Swift is a great option for deep learning. In some ways Jax is almost "non-python deep learning" since it's treating Python more like a DSL for the XLA backend. Normal Python code doesn't work in Jax. It's a pretty reasonable compromise since you still get all the benefits of the Python ecosystem. Julia seems like it has the best foundations for deep learning, since everything can be written directly in Julia. But it doesn't have a great ecosystem as a general programming tool. F# might turn out to be a good option.
sriram_malhar 4y ago
Thank you so much for this course. I plan to go through it properly. I have a search problem of my own and I have had a hard time applying what I have learnt (including the coursera DL specialization). The chief characteristics are: (a) It is a fuzzy search of a corpus that is in a non-English language. (b) The search should be able to run on a mobile phone _offline_. Is this possible? Can training be done elsewhere and transferred to TinyML or some such? What would be a good forum to go seeking answers?
- devnonymous 4y ago
  If the volume of data fits on a mobile phone for it to be offline, perhaps you don't need deep learning?
  - sriram_malhar 4y ago
    Perhaps. What alternative would you suggest? The search terms are fuzzy, and there are too many variants (not exactly misspellings) for me to encode them explicitly. So I thought I'd rather learn from a crowd-sourced corpus. In my case, I don't want the model to be general. I can afford for it to be like a database index, tailored to that data.
- leobg 4y ago
  Have you tried... a) BM25 after some preprocessing (lemmatization etc.) b) fastText / GloVe (possibly weighted by BM25) The results can be surprisingly good. Often no need to bother with big language models or GPUs.
  - sriram_malhar 4y ago
    As far as I understand, BM25 is not for fuzzy searches. For a bit more context, the search terms are in the English script, but the words are basically the closest-sounding transcriptions of sounds in various Indian languages. Different people may render the same word differently in an English transcription. But there's enough crowd-sourced data to account for the ways in which words can vary. For the same reason, GloVe is of not much use to me.
nindalf 4y ago
I tried the 2021 course but I didn't finish. I think the biggest friction for me was using the remote machine. I wasn't able to make steady progress like I do with my offline learning projects. How far away is the fast.ai from working on a Mac? PyTorch recently gained support (https://pytorch.org/blog/introducing-accelerated-pytorch-training-on-mac/ https://pytorch.org/blog/introducing-accelerated-pytorch-tra...) but that's only the start. Is this something that is being worked on?
- jph00 4y ago
  The good news is that every lesson in this course is actually run on Kaggle Notebooks, which is a free cloud environments including GPUs. So you don't need to set up anything and it runs on any computer with a modern web browser! Mac support for all the libs used in the course will probably continue to improve in the coming months and there should be no reason you won't be able to run the stuff for the course locally on a Mac at that time. Having said that, even the M2 trains deep learning models much slower than even the free NVIDIA GPUs provided by Kaggle. So you'd only want to use local development for the smallest and simplest models. (The course shows how to train models that are fairly cutting edge and some take a while to train even on modern GPUs, so they wouldn't be a good fit for a Mac.)
  - nindalf 4y ago
    Thanks Jeremy, I'll give it another go.
  - uwuemu 4y ago
    I think last time I tried this, I kinda gave up as soon as it got to the point of hand-waving hardware and telling you to run notebooks on a third party's web service. How is that democratizing AI? It's the very opposite! Not good. What is the intended audience here? Uni-level students will learn most of these within their programs if they're interested in AI. So they're not really it. Is the intended audience "coders"? If so, most of these "coders" will have to somehow get their employer on board (typically a corporate entity very much NOT interested in "coding" something in a 3rd party ecosystem) or do it themeselves. Hence, I want to take an RTX 3080+, 64+ gb of ram, big ass SSD and I want to get through the training. Not learn some basics on somebody else's platform (come on, even from the pov of OS, running both training AND notebooks on some 3rd party's private platform is so against the idea of open source...) and call it a day. What use is that? That may be enough if you want to be a cog in somebody else's machine, but not if you want to do something useful by yourself (I say "may" because big tech generally isn't interested in your "mad AI skillz" unless you also have a student loan backed piece of paper proving you successfully learned that for the last couple of years). There will always be smart individuals and talented small teams that can successfully integrate AI into their products, but it's not thanks to the courses like this. If you're going to aim at coders, there has to be clear path demostrated from the beginning to the end. From starting up your first notebook on your local dev machine and running training on your local training machine to setting up inference in the final app (.net app or whatever)
mkl 4y ago
How many hours do you think this course would take for an experienced developer with plenty of applied maths but ~no machine learning? How easy is it to do the course on my own hardware rather than cloud notebooks? Would that make it closer to practical deployment?
- gandalfgreybeer 4y ago
  Not him nor will I talk about his course, but I’ve been in the field a reasonable amount of time (both on the academia and industry side). Honestly, applied maths will get you a long way and make it easier to digest the concepts (you might just see them as repackaged problems depending on your mileage). If you have good programming skills and discipline you practically have most of what you need. Re the course, I just skimmed it and I think you can do most things on your own hardware but if you will actually use this for something practical (not just for you or a side project), being familiar with cloud tools is a big thing especially once you scale.
  - qwrshr 4y ago
    out of curiosity, how much applied math should one bone up on? (Obviously the more the better, but diminishing marginal returns and all that.)
    - cinntaile 4y ago
      None, just look things up as you go along if there is something you don't understand. You're likely not going to bother understanding how the optimization functions work or how the cost functions actually work anyway. They're implementation details in most cases.
    - gandalfgreybeer 4y ago
      Bare minimum is basic calculus, basic linear algebra and basic statistics. By basic, I probably mean first courses for those in most undergraduate programs. I disagree with needing none and just going along as needed. That’s how you have machine learning models that look like they work but you don’t understand why they work so there might actually be problems.
sabertoothed 4y ago
I hope you stay as humble as you have been. But you're my personal hero. It is just incredible what you have done for the world.
Oreb 4y ago
Is there a new version of the book? All the links I find lead to the 2020 edition.
- jph00 4y ago
  No, the book is continually updated for each reprint, but there isn't a separate edition.
fareesh 4y ago
Hello Jeremy do you have any specific advice on tackling ASR using fast.ai?
thefreeman 4y ago
As someone who much prefers reading over watching videos, do you think I would miss much by just going through the book in the github repo? Or are those notebooks mainly supplemental to the videos.
- jph00 4y ago
  I'd say it's the other way around - the videos are kinda supplemental to the book. The book has a lot more content, but doesn't have the interactive explanations in the course. Also the book is a couple of years old so is missing the more recent developments (but the principles haven't changed).
cweill 4y ago
Hey Jeremy, i just want to say that I love your course and the way you teach. I refer everyone to the Fast AI in my YouTube videos on getting started with machine learning. Please keep up the great work!
__rito__ 4y ago
Jeremy, hi. I have one question, and one only. Please answer: Second part, when?
kachau 4y ago
Will it help me more or achieve more out of learning this course as compared to just directly using GPT-3 or Dall-E as paid user?
baladre 4y ago
.b
baladre 4y ago
En
baladre 4y ago
.m

ben-coman 4y ago

Watching from afar the great advances in machine learning over the past few years with AlphaZero, GPT3, DALLE2 I felt it important for me to start understanding what is going on under the hood. Having just completed the private pre-release of the course run through USQ, as my first foray into machine learning this was a great introduction that had me quickly produce a working image classification system. The videos are packed really with insightful rid-bits about practical approaches to iterating quickly to understand the data better to produce better results. Very much recommend the course.

perfopt 4y ago

This is awesome. One question I have always had - is the research on applying DL for images the most developed compared to other things? Even DL used for audio processing (classification, separation etc) seems to convert audio to spectral graphs and apply DL to that. Changing a problem to be expressed as image inputs will be an advantage when using DL as a solution. Would you agree ?

perfopt 4y ago
Oops dumb question. Watched the first video and got my answer.
tmabraham 4y ago
Good question! I think a major reason for this is because of transfer learning. For computer vision, there are many good pretrained models that were trained on huge datasets (like ImageNet) that can be fine-tuned for custom tasks. Other fields often do not have such pretrained models and huge datasets to work on, so it turns out transforming a dataset into an image dataset and fine-tuning a pretrained model works better than training from scratch.
nmfisher 4y ago
Working with a spectrogram is definitely similar to working with an image, and it's interesting to think why that's the case. Take convolutional models, for example. Very effective for working with images because they're (a) parameter efficient, (b) learn local/spatial correlations in input features, and (c) exploit translational invariance. As an oversimplification, we can train models to visually identify "things" in images by their edges. If you think about what's going on with an audio spectrogram, you can see the same concepts at work. There's local/spatial correlation - certain sounds tend to have similar power coefficients in similar frequency buckets. These are also correlated in time (because the pitch envelope of the word "yes" tends to have the same shape), and convolutional models can also exploit time-invariance (in the sense that convolutional models can learn the word "yes" from samples where the word appears with varying amounts of silence to the left and right). That being said, the addition of the time domain makes audio quite hard to work with, and (usually) not as simple as just running a spectrogram through a vanilla image classification model. But it's definitely enlightening to think about how these models are "learning".

perfopt 4y ago

Can one do these lessons in any order? For example, do CNN first then jump back to NLP. Or skip the implementation from scratch because I have done a similar one in another course.

jph00 4y ago
They're designed to be done in order, but yup if you know how SGD works, for instance, you could certainly skip over that bit. The videos all have youtube timestamps, so if you drag the scrollbar you'll see what each section is about. Or you could do those bits at 2x speed in case there's some concepts there you haven't seen before. The NLP lesson could possibly work reasonably well standalone if you already know some DL basics, since it uses a different framework (Hugging Face) to the earlier lessons. The CNN lesson would probably largely make sense if you already understand multi-layer perceptrons, since it mainly shows how a convolution is just a special case of sparse matrix multiplication.

ramesh31 4y ago

You guys really are the best. Thanks for all the hard work.

alexcnwy 4y ago

The first Fast.ai course back in around 2016 changed my life. I was studying a masters in statistics and computer science that had 1 neural networks lecture and nobody knew anything about deep learning. Fast.ai and Jeremy’s teaching style helped me start playing with deep learning models really quickly and I changed my thesis topic to computer vision. I ended up consulting on the topic and doing various startups leading to the startup I’m working on now which just finished YC (AiSupervision W22). I doubt be here without fast.ai. I highly recommend and appreciate all the work that Jeremy and the rest of fast.ai do!

rg111 4y ago

There are too many poor design decisions in the fast.ai library. One should invest too much time just for the sake of learning the library's weird API, and then using it. Doing something custom is too difficult, in contrast to Jax, PyTorch, and even (poor library) TensorFlow. The coding practices are whimsical. The codebase wouldn’t pass code review in any respectable company. Variable namings are weird and super-problematic. I fully stick to what I said. Learn techniques, best practices, and, most importantly, Howard's attitude. Then take them with you and move onto something like PyTorch. Howard is great with one problem: he kinda hates math. It might also seem that he ends up promoting anti-intellectualism.

deleted 4y ago
[deleted]
sabertoothed 4y ago
I deleted an earlier, angrier comment of mine. Can you explain this last sentence (which I understand to be insulting and without basis): > Howard is great with one problem: he kinda hates math. It might also seem that he ends up promoting anti-intellectualism.
- rg111 4y ago
  He says repeatedly "You don't need math", and stuff like that. This is not insulting. That man is my hero, and I deeply respect him. But his 2019/20 course was riddled with such statements. He repeatedly said that one doesn't need math, and showed tools like drawing math symbols on a website to learn their names and ride on that. No further math needed. It's like you can wing it in Deep Learning without learning Math. His behavior throughout the course reinforced this attitude. It is harmful for new learners. But I am fortunate that I didn't learn from that, but learned from some successful alumni example that Howard gave. One woman who was also a musician ('19/'20), she made it big, but Howard mentioned that she did the Ng course, and also read the Goodfellow book. So, I took the cue, and did DL the proper way. Anybody I know in DL made it because they know the Math. There are some influencer types in fastai community who has 10ks of followers and shills stuff and do media stuff. Other than that 1-2 people, everyone who made it in DL, did it because they knew the math. So, I think that people might get the wrong idea hearing from Howard that "you don't need math". This is one fault I find. It's not like I dislike him. I like the rest of him. I love his attitude on almost all other things. I love Jeremy Howard, and he is my hero.

jackallis 4y ago

what is/will be the state of deep learning in 2022 or next 3-5 years? you hear/read so many news/articles in HN about decline of DL. Is that so?

davidatbu 4y ago
I mean, just looking at OpenAI and Deepmind, they have relatively recently released break-through models for which building upon and extending can be done in relatively straightforward ways (DALLE 2, GPT-3, AlphaFold, OpenAI Codex, ...etc), so I don't think DL will "decline" any time soon ...
kache_ 4y ago
That's like saying "watch the decline of C++" while using javascript What does javascript run on? Most ML advancements will use DL at the core in interesting ways.

publicdaniel 4y ago

I am so grateful the FastAI team exists. It wasn't until I discovered their "Machine Learning for Coders" course that I really started to grok ML. I was in grad school trying to pivot my career from finance to data science. I didn't come from a computer science / math background and things just weren't clicking for me. I remember feeling angry, embarrassed, dumb, and overall that I wasn't smart enough to learn this stuff -- I was incredibly discouraged and felt that I didn't belong there. I was lucky enough to stumble across one of the course videos on YouTube (thanks recommendation algorithm!), and the rest is history. The amazing thing about these courses is how simple Jeremy (and team) are able to make machine learning. I didn't need to understand python dependency management in order to learn how to train an really good image classifier. Their approach helped me have lots of little wins, gave me confidence, and helped build the motivation to slog through the harder stuff when I needed to. From the bottom of my heart, thank you @jph00. You changed my life immeasurably for the better. I learned that I AM good enough, I AM smart enough, and I CAN do hard things... I just had to find the right way to learn them. Your courses completely changed my perspective on what was possible for me and opened the door to some of my life's greatest passions.

wittycardio 4y ago
- BeetleB 4y ago
  Lots of people who've taken the Fast.ai course have similar things to say. It's commonly said it's the fastest way to get into DL.
  - hello34 4y ago
    Yeah just wanted to say, I am also one of the persons who has immensely benefited. So it may sound like paid, yet lot of people have immensely benefited like people from India, Nigeria, etc.. Check this article[1] to know a bit about philosophy of fast.ai and why it's so popular [1] https://future.com/the-rise-of-domain-experts-in-deep-learning/ https://future.com/the-rise-of-domain-experts-in-deep-learni...

suhail 4y ago

I re-did the course with this 2022 version. Highly recommend it :)

__rito__ 4y ago

I haven’t seen this course content yet, but fully did the 2019 version. Extremely grateful to have found it. Changed the course of my life. I can vouch for it's quality. Jeremy is an excellent instructor. So much clarity in his teaching! I love that this is a hands-on course, and there are ZERO hand-wavings. I also really like the top-down approach of teaching. Now, whenever I try to communicate something or teach someone, I try to do it top-down. And I have Jeremy to thank for that. Currently, I am attending his APL study group and having a blast! Only question for @jph00 is: second part, when?

newsoul 4y ago
Yes, updated second part of the course when? Any chance you will shift from Swift to Julia?

anakaine 4y ago

The course uses PyTorch, which despite being a big played still has many issues running g on any of the latest AMD cards. Windows not supported on AMD cards Navi series cards not supported in general. Heavily biased towards CUDA, despite AMD cards drivers being open sourced far more than Nvidia cards. Remote machines and kaggle notebooks go some way to improving these limits for the course. I'm complaining a bit more in general here, I think.

newsoul 4y ago

Hello Jeremy. Thanks for this great content. I have gone through your entire course and learned a ton from you. Now moving on from here, do you have any resource recommendation where I can dive deeper into machine learning and deep learning theory? And also any resources to become a much much better programmer? I am currently working in as an assistant in a research lab. My coding skills are not that great.