4 ms·

I am just finishing up my MS in Statistics, and at my university they don't have any course(yet) that goes into deep learning. The only time the course briefly

I am just finishing up my MS in Statistics, and at my university they don't have any course(yet) that goes into deep learning. The only time the course briefly talk about neural net is in the context of Logit regression. And I have always wonder if the time spent proving Gauss Markov theorem is better spent somewhere else. Do you know any source(book or otherwise) that continues to build on top of topics taught in a Regression and Linear algebra course(such as Multiple Linear, Weighted Least square, Logistic, generalized least squares, principle component regression, singular value decomposition) and slowly move towards the current state deep learning?

konjin 6y ago
There is no theory in current state of the art deep learning. It works, but no one knows why, and no one knows how to fix a model that breaks.
trevett 6y ago
Sounds like you might like this book: https://www.deeplearningbook.org/ https://www.deeplearningbook.org/
sjg007 6y ago
>And I have always wonder if the time spent proving Gauss Markov theorem is better spent somewhere else. No.. learning how to prove it will serve you well in the future. All deep learning is is figuring out different ways to combine all of this prior art inside of a computation graph. Step N is linear regression, SVD etc... step N+1 is deep learning.
- wcarss 6y ago
  I wish I knew what it was like to have the math background, and don't at all mean to suggest one should not bother, but as a person on a team of people who have made a few production NN models over the past year: it is completely possible to conceptually and practically grasp iterated reverse differentiation with convolutions well enough to use deep learning to do novel work, without having the deep background math knowledge. For example, I barely know what a Support Vector Machine is, nor could I do a linear regression without a lot of hand holding. But I can design a passable tensorflow model and improve it. It would definitely be very hard to do any meaningful research from this position, but I know enough to be useful (er, or dangerous) and can read papers and code to keep up with recent advancements, and try things out like different convolution designs, layer designs, gates, functions, feedback, etc. (On the topic of reading: Karpathy[1] and Colah[2]'s posts have a wealth of introductory conceptual information and images in them, and helped our team discussions a lot while we've learned. 1 - https://karpathy.github.io/ https://karpathy.github.io/ 2 - https://colah.github.io/ https://colah.github.io/)
  - sjg007 6y ago
    Sure makes sense. There are multiple layers to deep learning: The math and proofs. The algorithms. The software tools/frameworks Building NNs with those tools. All of those have different levels of conceptualization and utility. As some extreme examples: Anyone can build an iPhone image classifier drag and drop today. That’s one level. Developing an alternate to backprop is another level.
    - sjg007 6y ago
      Another example would be databases.. there's the theory, the tools and the applications.
  - iamcreasy 6y ago
    I think it will vary for person to person - but when I am looking at a new idea(i.e. principle component analysis) that is built on top of old ideas(PCA is based on Singular value decomposition) - familiarity with old ideas makes me more comfortable/confident in thinking that I'll be able to understand the new idea thoroughly and hopefully I'll also see the incoming pitfalls of the new idea. And I really enjoy the process of adding new concept/idea in an existing larger picture.
jahewson 6y ago
SVMs are still relevant and well worth learning if you want ML of the non-neural variety.
deleted 6y ago
[deleted]
TrackerFF 6y ago
Last time I took a Uni. course, on the side of work, I enrolled in a brand new Deep Learning course. We used Ian Goodfellows book (https://www.deeplearningbook.org/ https://www.deeplearningbook.org/), as well as supplementary research papers.
sgt101 6y ago
I liked "Hands on Machine Learning" by A. Geron. It's quite practical - you won't find careful progressive proofs, but it is very good at introducing the methods of ML and relating them to real problems. It provides you with a way to get your hands round the techniques - with an MSc in Stats you should then be well placed to understand where the gaps and problems are and to use things wisely. It isn't quite state of the art, and it is very light in terms of how to evaluate and deliver the solutions. But I thought it was good. My team at work has gone through it selecting one chapter at a time this year in our book club - it's provided a good catch up and share forum for us. Some of the folks in the team are Engineers and so haven't a strong ML background, a couple are Data Scientists and therefore were in the same place as you (+/- some use of deep learning in a few engagements) and a couple of the folks are diehard ML researchers, but focused on different things (time series, NLP, evaluation) so there was a lot of crossover for everyone.

konjin 6y ago

There is no theory in current state of the art deep learning. It works, but no one knows why, and no one knows how to fix a model that breaks.

trevett 6y ago

Sounds like you might like this book: https://www.deeplearningbook.org/ https://www.deeplearningbook.org/

sjg007 6y ago

>And I have always wonder if the time spent proving Gauss Markov theorem is better spent somewhere else. No.. learning how to prove it will serve you well in the future. All deep learning is is figuring out different ways to combine all of this prior art inside of a computation graph. Step N is linear regression, SVD etc... step N+1 is deep learning.

wcarss 6y ago
I wish I knew what it was like to have the math background, and don't at all mean to suggest one should not bother, but as a person on a team of people who have made a few production NN models over the past year: it is completely possible to conceptually and practically grasp iterated reverse differentiation with convolutions well enough to use deep learning to do novel work, without having the deep background math knowledge. For example, I barely know what a Support Vector Machine is, nor could I do a linear regression without a lot of hand holding. But I can design a passable tensorflow model and improve it. It would definitely be very hard to do any meaningful research from this position, but I know enough to be useful (er, or dangerous) and can read papers and code to keep up with recent advancements, and try things out like different convolution designs, layer designs, gates, functions, feedback, etc. (On the topic of reading: Karpathy[1] and Colah[2]'s posts have a wealth of introductory conceptual information and images in them, and helped our team discussions a lot while we've learned. 1 - https://karpathy.github.io/ https://karpathy.github.io/ 2 - https://colah.github.io/ https://colah.github.io/)
- sjg007 6y ago
  Sure makes sense. There are multiple layers to deep learning: The math and proofs. The algorithms. The software tools/frameworks Building NNs with those tools. All of those have different levels of conceptualization and utility. As some extreme examples: Anyone can build an iPhone image classifier drag and drop today. That’s one level. Developing an alternate to backprop is another level.
  - sjg007 6y ago
    Another example would be databases.. there's the theory, the tools and the applications.
- iamcreasy 6y ago
  I think it will vary for person to person - but when I am looking at a new idea(i.e. principle component analysis) that is built on top of old ideas(PCA is based on Singular value decomposition) - familiarity with old ideas makes me more comfortable/confident in thinking that I'll be able to understand the new idea thoroughly and hopefully I'll also see the incoming pitfalls of the new idea. And I really enjoy the process of adding new concept/idea in an existing larger picture.

jahewson 6y ago

SVMs are still relevant and well worth learning if you want ML of the non-neural variety.

deleted 6y ago

[deleted]

TrackerFF 6y ago

Last time I took a Uni. course, on the side of work, I enrolled in a brand new Deep Learning course. We used Ian Goodfellows book (https://www.deeplearningbook.org/ https://www.deeplearningbook.org/), as well as supplementary research papers.

sgt101 6y ago

I liked "Hands on Machine Learning" by A. Geron. It's quite practical - you won't find careful progressive proofs, but it is very good at introducing the methods of ML and relating them to real problems. It provides you with a way to get your hands round the techniques - with an MSc in Stats you should then be well placed to understand where the gaps and problems are and to use things wisely. It isn't quite state of the art, and it is very light in terms of how to evaluate and deliver the solutions. But I thought it was good. My team at work has gone through it selecting one chapter at a time this year in our book club - it's provided a good catch up and share forum for us. Some of the folks in the team are Engineers and so haven't a strong ML background, a couple are Data Scientists and therefore were in the same place as you (+/- some use of deep learning in a few engagements) and a couple of the folks are diehard ML researchers, but focused on different things (time series, NLP, evaluation) so there was a lot of crossover for everyone.