6 ms·

I know it says graduate level math not required but it does need some math correct? What is the minimum?

by throwaway2016a 10y ago

jph00 10y ago
You need to have a basic understanding of matrix products (from linear algebra) and the chain rule (from calculus). Both can be learnt on demand using Khan Academy - no need to go study math first, but just study what you need when you need it. We give brief explanations of both when they come up. The wiki (http://wiki.fast.ai http://wiki.fast.ai) has links to necessary learning resources for each lesson.
- bogomipz 10y ago
  So these are the two subsets of those maths needed for this course but in general is Deep Learning largely based in just Calculus and Linear Algebra? Are there any others you would recommend learning and/or brushing up on in a more general sense for ML/DL? Thanks for making this available by the way!
  - jph00 10y ago
    Deep learning is pretty much entirely linear algebra and calculus, at a mathematical level. But I really think that a code-first approach is much better. So I'd suggest just getting started working through the course, and really dive into the Jupyter notebooks for each lesson. Do lots of experiments.
    - bogomipz 10y ago
      >"But I really think that a code-first approach is much better." This is really refreshing to hear Most of the books I have looked at seem to be mostly math-first and very densely so at that. As much as I appreciate math it's not always the warmest welcome. I am looking forward to working through your course and seeing part 2 as well. Cheers.
  - Cybiote 10y ago
    If you want a solid understanding of DL⊂ML, you'll also need a good understanding of probability theory and information theory. While the mechanics require just matrix math and some vector calc, you'll also want to know linear algebra proper and these days, function spaces especially. A small helping of analysis, for measure theory, will not hurt either.
    - bogomipz 10y ago
      Interesting that you mentioned Information Theory, when I think of this I think of Shannon, encoding symbols, bauds, bit, noise and channels in the Telecom network sense. But I guess this all applies quite similarly in the neural network sense in DL/ML as well?
      - jph00 10y ago
        There's a 1:1 relationship between likelihood functions and entropy, so a background in information theory is likely to make your understanding of loss functions come quicker.
        bogomipz 10y ago
        Thanks for the insight.
    - jph00 10y ago
      I strongly disagree. This has been necessary for many previous approaches, but it's not fundamental to the topic. Sometimes reading and understanding papers requires that background, I guess.
      - Cybiote 10y ago
        First of all, let me say that what you all have put together is truly excellent. It covers the sort of things toy treatments and papers leave out or gloss over if you actually want to get something working in the real world. That said, I strongly disagree with your disagreement. There was a recent paper whose abstract I read that made me think of homotopies between convolutional networks. Unfortunately I lost the paper behind some stream and never got to read it proper. In the context of that, I realized that the search for convnet design will likely soon be highly automatable, obsoleting much of the work that many DLers are doing now. What will be future proof is understanding information theory so that loss functions become less magical. Information theory is needed to understand what aspects of the current approaches to Reinforcement learning are likely dead ends (typically to do with getting a good exploration strategy, also related to creativity). Concentration of measure is vital to understanding so many properties we find in optimization, dimensionality reduction and learning. Understanding learning stability and ideal properties of a learner/convergence means being comfortable with concepts like Jacobians and semi-positive definiteness for a start. Probability theory is needed for the newer variational methods, whether in the context of autoencoders or a library like Edward (whose like I think is the future). Functionals and the variational calculus is becoming more important, in both deep learning and for understanding the brain. There's lots of work in game theory of dynamical systems (think evolutionary game theory) that can help contextualize GANs as a special case of a broader category of strategies. Much to the contrary, the topics I mentioned are both the future of deep learning and future proof in general. This blog post by Ferenc captures my sentiment on the matter: http://www.inference.vc/deep-learning-is-easy/ http://www.inference.vc/deep-learning-is-easy/
deepnotderp 10y ago
How the hell is deep learning graduate level math?
- jph00 10y ago
  Some courses are, eg https://www.cs.ox.ac.uk/people/nando.defreitas/machinelearning/ https://www.cs.ox.ac.uk/people/nando.defreitas/machinelearni... .
  - deepnotderp 10y ago
    There's no way that's graduate level math. I've gone through Nando's lectures and although they're more mathematically involved than most courses, they aren't grad level math.
    - jph00 10y ago
      IIRC the course details suggest it is most appropriate for a graduate level. I may be misremembering however.
      - master_yoda_1 10y ago
        You might be right. The level of education is different for every individual and also people come from different country. In America you can be a graduate in computer science without knowing calculus, and in some other country they teach calculus/probability in high school. So the above person might have studied the chain rule and entropy in high school. So its not exactly graduate level math for everyone.
        vinay427 10y ago
        > In America you can be a graduate in computer science without knowing calculus, and in some other country they teach calculus/probability in high school. This certainly isn't true at my university in the US. As a matter of fact, I'm not sure why you named a country at all seeing as this varies by university. Anyway, I would definitely agree that on the scale of reputable international universities this is not grad-level math. A graduate school in a well-qualified university would expect students to either know this material or be able to learn it on their own. They are intro classes for the math major at my (and many other) universities.

I know it says graduate level math not required but it does need some math correct? What is the minimum?

jph00 10y ago

You need to have a basic understanding of matrix products (from linear algebra) and the chain rule (from calculus). Both can be learnt on demand using Khan Academy - no need to go study math first, but just study what you need when you need it. We give brief explanations of both when they come up. The wiki (http://wiki.fast.ai http://wiki.fast.ai) has links to necessary learning resources for each lesson.

bogomipz 10y ago
So these are the two subsets of those maths needed for this course but in general is Deep Learning largely based in just Calculus and Linear Algebra? Are there any others you would recommend learning and/or brushing up on in a more general sense for ML/DL? Thanks for making this available by the way!
- jph00 10y ago
  Deep learning is pretty much entirely linear algebra and calculus, at a mathematical level. But I really think that a code-first approach is much better. So I'd suggest just getting started working through the course, and really dive into the Jupyter notebooks for each lesson. Do lots of experiments.
  - bogomipz 10y ago
    >"But I really think that a code-first approach is much better." This is really refreshing to hear Most of the books I have looked at seem to be mostly math-first and very densely so at that. As much as I appreciate math it's not always the warmest welcome. I am looking forward to working through your course and seeing part 2 as well. Cheers.
- Cybiote 10y ago
  If you want a solid understanding of DL⊂ML, you'll also need a good understanding of probability theory and information theory. While the mechanics require just matrix math and some vector calc, you'll also want to know linear algebra proper and these days, function spaces especially. A small helping of analysis, for measure theory, will not hurt either.
  - bogomipz 10y ago
    Interesting that you mentioned Information Theory, when I think of this I think of Shannon, encoding symbols, bauds, bit, noise and channels in the Telecom network sense. But I guess this all applies quite similarly in the neural network sense in DL/ML as well?
    - jph00 10y ago
      There's a 1:1 relationship between likelihood functions and entropy, so a background in information theory is likely to make your understanding of loss functions come quicker.
      - bogomipz 10y ago
        Thanks for the insight.
  - jph00 10y ago
    I strongly disagree. This has been necessary for many previous approaches, but it's not fundamental to the topic. Sometimes reading and understanding papers requires that background, I guess.
    - Cybiote 10y ago
      First of all, let me say that what you all have put together is truly excellent. It covers the sort of things toy treatments and papers leave out or gloss over if you actually want to get something working in the real world. That said, I strongly disagree with your disagreement. There was a recent paper whose abstract I read that made me think of homotopies between convolutional networks. Unfortunately I lost the paper behind some stream and never got to read it proper. In the context of that, I realized that the search for convnet design will likely soon be highly automatable, obsoleting much of the work that many DLers are doing now. What will be future proof is understanding information theory so that loss functions become less magical. Information theory is needed to understand what aspects of the current approaches to Reinforcement learning are likely dead ends (typically to do with getting a good exploration strategy, also related to creativity). Concentration of measure is vital to understanding so many properties we find in optimization, dimensionality reduction and learning. Understanding learning stability and ideal properties of a learner/convergence means being comfortable with concepts like Jacobians and semi-positive definiteness for a start. Probability theory is needed for the newer variational methods, whether in the context of autoencoders or a library like Edward (whose like I think is the future). Functionals and the variational calculus is becoming more important, in both deep learning and for understanding the brain. There's lots of work in game theory of dynamical systems (think evolutionary game theory) that can help contextualize GANs as a special case of a broader category of strategies. Much to the contrary, the topics I mentioned are both the future of deep learning and future proof in general. This blog post by Ferenc captures my sentiment on the matter: http://www.inference.vc/deep-learning-is-easy/ http://www.inference.vc/deep-learning-is-easy/

deepnotderp 10y ago

How the hell is deep learning graduate level math?

jph00 10y ago
Some courses are, eg https://www.cs.ox.ac.uk/people/nando.defreitas/machinelearning/ https://www.cs.ox.ac.uk/people/nando.defreitas/machinelearni... .
- deepnotderp 10y ago
  There's no way that's graduate level math. I've gone through Nando's lectures and although they're more mathematically involved than most courses, they aren't grad level math.
  - jph00 10y ago
    IIRC the course details suggest it is most appropriate for a graduate level. I may be misremembering however.
    - master_yoda_1 10y ago
      You might be right. The level of education is different for every individual and also people come from different country. In America you can be a graduate in computer science without knowing calculus, and in some other country they teach calculus/probability in high school. So the above person might have studied the chain rule and entropy in high school. So its not exactly graduate level math for everyone.
      - vinay427 10y ago
        > In America you can be a graduate in computer science without knowing calculus, and in some other country they teach calculus/probability in high school. This certainly isn't true at my university in the US. As a matter of fact, I'm not sure why you named a country at all seeing as this varies by university. Anyway, I would definitely agree that on the scale of reputable international universities this is not grad-level math. A graduate school in a well-qualified university would expect students to either know this material or be able to learn it on their own. They are intro classes for the math major at my (and many other) universities.