5 ms·

You need to have a basic understanding of matrix products (from linear algebra) and the chain rule (from calculus). Both can be learnt on demand using Khan Acad

by jph00 10y ago

You need to have a basic understanding of matrix products (from linear algebra) and the chain rule (from calculus). Both can be learnt on demand using Khan Academy - no need to go study math first, but just study what you need when you need it. We give brief explanations of both when they come up. The wiki (http://wiki.fast.ai http://wiki.fast.ai) has links to necessary learning resources for each lesson.

bogomipz 10y ago
So these are the two subsets of those maths needed for this course but in general is Deep Learning largely based in just Calculus and Linear Algebra? Are there any others you would recommend learning and/or brushing up on in a more general sense for ML/DL? Thanks for making this available by the way!
- jph00 10y ago
  Deep learning is pretty much entirely linear algebra and calculus, at a mathematical level. But I really think that a code-first approach is much better. So I'd suggest just getting started working through the course, and really dive into the Jupyter notebooks for each lesson. Do lots of experiments.
  - bogomipz 10y ago
    >"But I really think that a code-first approach is much better." This is really refreshing to hear Most of the books I have looked at seem to be mostly math-first and very densely so at that. As much as I appreciate math it's not always the warmest welcome. I am looking forward to working through your course and seeing part 2 as well. Cheers.
- Cybiote 10y ago
  If you want a solid understanding of DL⊂ML, you'll also need a good understanding of probability theory and information theory. While the mechanics require just matrix math and some vector calc, you'll also want to know linear algebra proper and these days, function spaces especially. A small helping of analysis, for measure theory, will not hurt either.
  - bogomipz 10y ago
    Interesting that you mentioned Information Theory, when I think of this I think of Shannon, encoding symbols, bauds, bit, noise and channels in the Telecom network sense. But I guess this all applies quite similarly in the neural network sense in DL/ML as well?
    - jph00 10y ago
      There's a 1:1 relationship between likelihood functions and entropy, so a background in information theory is likely to make your understanding of loss functions come quicker.
      - bogomipz 10y ago
        Thanks for the insight.
  - jph00 10y ago
    I strongly disagree. This has been necessary for many previous approaches, but it's not fundamental to the topic. Sometimes reading and understanding papers requires that background, I guess.
    - Cybiote 10y ago
      First of all, let me say that what you all have put together is truly excellent. It covers the sort of things toy treatments and papers leave out or gloss over if you actually want to get something working in the real world. That said, I strongly disagree with your disagreement. There was a recent paper whose abstract I read that made me think of homotopies between convolutional networks. Unfortunately I lost the paper behind some stream and never got to read it proper. In the context of that, I realized that the search for convnet design will likely soon be highly automatable, obsoleting much of the work that many DLers are doing now. What will be future proof is understanding information theory so that loss functions become less magical. Information theory is needed to understand what aspects of the current approaches to Reinforcement learning are likely dead ends (typically to do with getting a good exploration strategy, also related to creativity). Concentration of measure is vital to understanding so many properties we find in optimization, dimensionality reduction and learning. Understanding learning stability and ideal properties of a learner/convergence means being comfortable with concepts like Jacobians and semi-positive definiteness for a start. Probability theory is needed for the newer variational methods, whether in the context of autoencoders or a library like Edward (whose like I think is the future). Functionals and the variational calculus is becoming more important, in both deep learning and for understanding the brain. There's lots of work in game theory of dynamical systems (think evolutionary game theory) that can help contextualize GANs as a special case of a broader category of strategies. Much to the contrary, the topics I mentioned are both the future of deep learning and future proof in general. This blog post by Ferenc captures my sentiment on the matter: http://www.inference.vc/deep-learning-is-easy/ http://www.inference.vc/deep-learning-is-easy/

by jph00 10y ago

bogomipz 10y ago

So these are the two subsets of those maths needed for this course but in general is Deep Learning largely based in just Calculus and Linear Algebra? Are there any others you would recommend learning and/or brushing up on in a more general sense for ML/DL? Thanks for making this available by the way!

jph00 10y ago
Deep learning is pretty much entirely linear algebra and calculus, at a mathematical level. But I really think that a code-first approach is much better. So I'd suggest just getting started working through the course, and really dive into the Jupyter notebooks for each lesson. Do lots of experiments.
- bogomipz 10y ago
  >"But I really think that a code-first approach is much better." This is really refreshing to hear Most of the books I have looked at seem to be mostly math-first and very densely so at that. As much as I appreciate math it's not always the warmest welcome. I am looking forward to working through your course and seeing part 2 as well. Cheers.
Cybiote 10y ago
If you want a solid understanding of DL⊂ML, you'll also need a good understanding of probability theory and information theory. While the mechanics require just matrix math and some vector calc, you'll also want to know linear algebra proper and these days, function spaces especially. A small helping of analysis, for measure theory, will not hurt either.
- bogomipz 10y ago
  Interesting that you mentioned Information Theory, when I think of this I think of Shannon, encoding symbols, bauds, bit, noise and channels in the Telecom network sense. But I guess this all applies quite similarly in the neural network sense in DL/ML as well?
  - jph00 10y ago
    There's a 1:1 relationship between likelihood functions and entropy, so a background in information theory is likely to make your understanding of loss functions come quicker.
    - bogomipz 10y ago
      Thanks for the insight.
- jph00 10y ago
  I strongly disagree. This has been necessary for many previous approaches, but it's not fundamental to the topic. Sometimes reading and understanding papers requires that background, I guess.
  - Cybiote 10y ago
    First of all, let me say that what you all have put together is truly excellent. It covers the sort of things toy treatments and papers leave out or gloss over if you actually want to get something working in the real world. That said, I strongly disagree with your disagreement. There was a recent paper whose abstract I read that made me think of homotopies between convolutional networks. Unfortunately I lost the paper behind some stream and never got to read it proper. In the context of that, I realized that the search for convnet design will likely soon be highly automatable, obsoleting much of the work that many DLers are doing now. What will be future proof is understanding information theory so that loss functions become less magical. Information theory is needed to understand what aspects of the current approaches to Reinforcement learning are likely dead ends (typically to do with getting a good exploration strategy, also related to creativity). Concentration of measure is vital to understanding so many properties we find in optimization, dimensionality reduction and learning. Understanding learning stability and ideal properties of a learner/convergence means being comfortable with concepts like Jacobians and semi-positive definiteness for a start. Probability theory is needed for the newer variational methods, whether in the context of autoencoders or a library like Edward (whose like I think is the future). Functionals and the variational calculus is becoming more important, in both deep learning and for understanding the brain. There's lots of work in game theory of dynamical systems (think evolutionary game theory) that can help contextualize GANs as a special case of a broader category of strategies. Much to the contrary, the topics I mentioned are both the future of deep learning and future proof in general. This blog post by Ferenc captures my sentiment on the matter: http://www.inference.vc/deep-learning-is-easy/ http://www.inference.vc/deep-learning-is-easy/