3 ms·

I'm almost completing my phd in math, and as much as it pains me to say this, I think a great amount of deep learning can be accessible without "the math". And

I'm almost completing my phd in math, and as much as it pains me to say this, I think a great amount of deep learning can be accessible without "the math". And as tempting as it is to tell everyone to pay their mathematical dues by carrying the proverbial buckets of water up the steps, you can get by very well in the field with only the very basic basics of mathematical knowledge. This is how i'd describe it. Deep learning is a set of tinker toys. Lego blocks if you will that you can sculpt with data into some very interesting models. Its an art, where the brushstrokes are matrices. Place an attention module here, and a convolution net there. And throw in a tensor with a softmax, and viola. Now I love math. and part of me really wants to see deep learning become a mathematical discipline. Deep in the backwaters there are parts of deep learning involve some math (think variational inference, bayesian models, etc). And I do want deep learning to be about condition numbers and combinatorics. But if you want to be perfectly honest with a newbie in the field, if you want to get your feet wet in deep learning, don't waste 3 months on a class on advanced optimization or measure theory or probability. Just dive in

nilkn 9y ago
I pretty much agree with you. I have a bachelor's degree in pure mathematics from a top 15 university -- I even published research as an undergraduate. Deep learning strikes me as an experimental science and even a subset of ordinary programming more than it does a subset of mathematics or statistics or statistical learning. I've read the entirety of the Deep Learning book by Goodfellow et. al. It's a fantastic book, but I was shocked at the lack of robust theory and the relative simplicity of the math. Even an introductory analysis textbook like Rudin is considerably harder, and Rudin is regularly tackled by undergraduates at any highly ranked university. The only reason to fret about this in my opinion is if you're a PhD machine learning engineer who doesn't want the field to open up to non-PhDs. I think data scientists are used to being able to say, "hey, if you don't have a PhD, you really can't do or understand what I do" -- deep learning represents potentially a huge culture shock to that attitude. But even the Google Brain research team has some non-PhDs now. I do think deep learning practitioners should learn the math. I just don't think there's actually that much math to learn. Certainly, if you read through the TensorFlow MNIST tutorial and you have no idea what cross-entropy is and you don't understand what the softmax layer is for, you need to go back to the basics. But these are concepts that anyone with any reasonable engineering degree can pick up relatively quickly. As an example, I submit the five articles on Distill, a new online machine learning journal: http://distill.pub/ http://distill.pub/ Notice that only the first has any real math, and even there the math is just not very advanced -- it's undergraduate-level.
- gabrielgoh 9y ago
  hey, I authored the first article :D
- minkzilla 9y ago
  Do you have any recommendations for books to get into deep learning? I want some theory and math, but not too much because I do not have that much mathamitical background. I have only gone through calculus 2. (I will be taking calc 3 this coming semester). I am a computer science major. I have dabbled in writing a super simple neural network to solve the MNIST. Using an example written in python and porting it to go, so that I couldn't copy and paste. I had to see what each step did. It very rapidly went to about 30% accuracy and stuck there. So I know I did something wrong. But I abandoned it after not being able to figure out what.
  - nilkn 9y ago
    Once you take calculus 3, I'd recommend diving right into the Deep Learning book: http://www.deeplearningbook.org/ http://www.deeplearningbook.org/ It's definitely the best reference on the subject. With only calculus 3 under your belt the math won't be trivial, but it should overall be fairly approachable and certainly much more so than something like "The Elements of Statistical Learning".

nilkn 9y ago

I pretty much agree with you. I have a bachelor's degree in pure mathematics from a top 15 university -- I even published research as an undergraduate. Deep learning strikes me as an experimental science and even a subset of ordinary programming more than it does a subset of mathematics or statistics or statistical learning. I've read the entirety of the Deep Learning book by Goodfellow et. al. It's a fantastic book, but I was shocked at the lack of robust theory and the relative simplicity of the math. Even an introductory analysis textbook like Rudin is considerably harder, and Rudin is regularly tackled by undergraduates at any highly ranked university. The only reason to fret about this in my opinion is if you're a PhD machine learning engineer who doesn't want the field to open up to non-PhDs. I think data scientists are used to being able to say, "hey, if you don't have a PhD, you really can't do or understand what I do" -- deep learning represents potentially a huge culture shock to that attitude. But even the Google Brain research team has some non-PhDs now. I do think deep learning practitioners should learn the math. I just don't think there's actually that much math to learn. Certainly, if you read through the TensorFlow MNIST tutorial and you have no idea what cross-entropy is and you don't understand what the softmax layer is for, you need to go back to the basics. But these are concepts that anyone with any reasonable engineering degree can pick up relatively quickly. As an example, I submit the five articles on Distill, a new online machine learning journal: http://distill.pub/ http://distill.pub/ Notice that only the first has any real math, and even there the math is just not very advanced -- it's undergraduate-level.

gabrielgoh 9y ago
hey, I authored the first article :D
minkzilla 9y ago
Do you have any recommendations for books to get into deep learning? I want some theory and math, but not too much because I do not have that much mathamitical background. I have only gone through calculus 2. (I will be taking calc 3 this coming semester). I am a computer science major. I have dabbled in writing a super simple neural network to solve the MNIST. Using an example written in python and porting it to go, so that I couldn't copy and paste. I had to see what each step did. It very rapidly went to about 30% accuracy and stuck there. So I know I did something wrong. But I abandoned it after not being able to figure out what.
- nilkn 9y ago
  Once you take calculus 3, I'd recommend diving right into the Deep Learning book: http://www.deeplearningbook.org/ http://www.deeplearningbook.org/ It's definitely the best reference on the subject. With only calculus 3 under your belt the math won't be trivial, but it should overall be fairly approachable and certainly much more so than something like "The Elements of Statistical Learning".