4 ms·
> in depth course that started right from the foundations—implementing and GPU-optimising matrix multiplications and initialisations—and covered from scratch im
by ralusek 4y ago
> in depth course that started right from the foundations—implementing and GPU-optimising matrix multiplications and initialisations—and covered from scratch implementations of all the key applications of the fastai library.
I haven't taken the course, but that sounds like a horrible place to start a course on understanding deep learning. GPU matrix operations are literally an implementation detail.
I think the proper way to teach deep learning "from scratch" would be:
1.) show simple example of regression using high level library
2.) implement same regression by writing a simple neutral network from scratch (explain going through, multiplying weights, adding biases, applying activation function, calculating loss, back propagation).
3. use NN on more complicated problem with more parameters and a larger training set, so that user sees they've hit a wall in performance, and now implementation needs to be optimized
4. at this point, say okay, our implementation of looping and multiplying can be done much faster with matrix multiplication on GPU, and even faster parallelized across GPUs on a network. If you're interested in that, here is an optional fork in the course that gets into specifics. Anything after this point will assume that implementation of NN calls will be using these techniques under the hood
5. move onto classification, q learning, GANs, transformers.
95% should have skipped step 4 and only revisited if they become interested in this for a specific reason. To start with it is crazy. It's like starting a course about flying by explaining how certain composites allowed us to transition from propellers to jets, and let's dive into how those composites are made.
- idf00 4y ago> I haven't taken the course, but that sounds like a horrible place to start a course on understanding deep learning. Sounds like you support the course author's decision to make this part 2 of the course series to be taken after part 1 is completed!
- ralusek 4y agoSorry for failing to clarify: > Three years ago we pioneered Deep Learning from the Foundations, an in depth course that started right from the foundations—implementing and GPU-optimising matrix multiplications and initialisations They're talking about how Part 1 starts
- Yenrabbit 4y ago'Deep learning from the Foundations' was the previous version of this new course, but both built on their respective part 1s. So part 1 is 'practical deep learning for coders', very good intro to DL for getting things done that starts with big wins using pretrained models and then digs into them enough that you know how to make them better. Part 2 is for those wanting to go all the way to implementing the ideas 'from scratch', reading papers and diving into advanced topics.
- idf00 4y agoDeep Learning from the Foundations IS part 2. They are not talking about how part 1 starts. They are talking about how part 2 starts. This mirrors how their book is written, with the content typically in Deep Learning for Foundations being the second half of the book. I've taken the courses every year, it's always been like that. You should consider taking a look at part 1 of the course if you want to understand the starting curriculum: https://course.fast.ai/ https://course.fast.ai/
- boloust 4y agoThe course basically goes through the exact steps you described. The main "GPU optimisation" is to basically say "okay, now that we know how to implement matrix multiplication, let's use the optimised pytorch implementation instead".