5 ms·

Does anyone know of good resources for studying machine learning or data science given a strong mathematical background? I'm transitioning careers from pure mat

Does anyone know of good resources for studying machine learning or data science given a strong mathematical background? I'm transitioning careers from pure math research into industry. I know very little about machine learning, but I know the crap out of linear algebra and real analysis (and other, less relevant fields). It'd be great to have some sources that leverage that without assuming much prior CS knowledge.

srean 8y ago
Given your math bent expecially analysis, I would recommend you start with Luc Devroye, follow it up with Vapnik. These are a bit dated but still very relevant. After these two go with Alex Smola and Bernard Scholkopf. The books that have been suggested to you so far aren't bad but with high fluff to meat ratio, not meant for a reader like you. If you have function analysis in your bag you will feel right at home with the books I suggested. Deep NN are much in the news, but on the math side of the practice of DNN, there is not much to it apart from chain rule for differentiation.
- poster123 8y ago
  Thanks for the recommendations. Devroye wrote several books, and I assume the one being mentioned is "A Probabilistic Theory of Pattern Recognition".
  - srean 8y ago
    Yes that's the one I had in mind. His other books are also very good, but are more specialized. You may also enjoy graycat's comments on HN. He know his math but is contrarian about machine learning but its good to have that view point.
- poster123 8y ago
  The book "A Probabilistic Theory of Pattern Recognition" by Devroye, Györfi and Lugosi is available from the site of Gyorfi at http://www.szit.bme.hu/~gyorfi/pbook.pdf http://www.szit.bme.hu/~gyorfi/pbook.pdf .
Iwan-Zotow 8y ago
Sure, Goodfellow book http://www.deeplearningbook.org/ http://www.deeplearningbook.org/
mlthoughts2018 8y ago
I made the same transition earlier in my career. One book on deep learning that meets your requirements is [0]. It’s readable, covers a broad set of modern topics, and has pragmatic tips for real use cases. For general machine learning, there are many, many books. A good intro is [1] and a more comprehensive, reference sort of book is [2]. Frankly, by this point, even reading the documentation and user guide of scikit-learn has a fairly good mathematical presentation of many algorithms. Another good reference book is [3]. Finally, I would also recommend supplementing some of that stuff with Bayesian analysis, which can address many of the same problems, or be intermixed with machine learning algorithms, but which is important for a lot of other reasons too (MCMC sampling, hierarchical regression, small data problems). For that I would recommend [4] and [5]. Stay away from bootcamps or books or lectures that seem overly branded with “data science.” This usually means more focus on data pipeline tooling, data cleaning, shallow details about a specific software package, and side tasks like wrapping something in a webservice. That stuff is extremely easy to learn on the job and usually needs to be tailored differently for every different project or employer, so it’s a relative waste of time unless it is the only way you can get a job. [0]: < https://www.amazon.com/Deep-Learning-Adaptive-Computation-Machine/dp/0262035618 https://www.amazon.com/Deep-Learning-Adaptive-Computation-Ma... > [1]: < https://www.amazon.com/Pattern-Classification-Pt-1-Richard-Duda/dp/0471056693 https://www.amazon.com/Pattern-Classification-Pt-1-Richard-D... > [2]: < https://www.amazon.com/Pattern-Recognition-Learning-Information-Statistics/dp/0387310738 https://www.amazon.com/Pattern-Recognition-Learning-Informat... > [3]: < http://www.web.stanford.edu/~hastie/ElemStatLearn/ http://www.web.stanford.edu/~hastie/ElemStatLearn/ > [4]: < http://www.stat.columbia.edu/~gelman/book/ http://www.stat.columbia.edu/~gelman/book/ > [5]: < http://www.stat.columbia.edu/~gelman/arm/ http://www.stat.columbia.edu/~gelman/arm/ >
- soVeryTired 8y ago
  +1 for Gelman, but I hate Bishop's book [2]. It was an early go-to reference in the field, but there are better books out there now.
  - hikarudo 8y ago
    What do you hate about Bishop's book? I'm genuinely curious.
soVeryTired 8y ago
For non-deep learning, read David Barber's book: http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/090310.pdf http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/090310.pdf Some sections may be less relevant, depending on what you want to do, but Section III is a very good introduction to machine learning methods. Do the exercises as you're reading. Theory is one thing, but in ML my rule of thumb is that you don't really understand a model until you've coded it up. A collection of written exercises would be a good way to impress an interviewer, too.
- glitch 8y ago
  Note: 090310.pdf (9 March 2010) is an older version of the textbook. The latest revision of the textbook PDF (091117.pdf, 9 November 2017) is available at: → http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/091117.pdf http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/091117.pdf Aside: ⸰ Textbook homepage: http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=Brml.HomePage http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=... ⸰ Online version homepage (should always contain link to latest revision): http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=Brml.Online http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=... ⸰ Directory sorted by date: http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/?C=M;O=D http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/?C=M;O=D
- nightski 8y ago
  It really is a great book. However my friend & I attempted to make it through and faced quite a bit of struggle. Eventually we'd figure things out but it felt like it would of been significantly easier with the help of a teacher/mentor to ask questions to. Unfortunately it's actually kind of hard to find colleges nearby that even have courses in Bayesian statistics (from an inference perspective). That was frustrating.
  - man-and-laptop 8y ago
    It's far from state of the art, but reading Computer Vision by Simon J D Prince [1] alongside David Barber's BRML can really help. [1] - http://www.computervisionmodels.com/ http://www.computervisionmodels.com/
thanatropism 8y ago
There are many new methods cropping up that most people in the data science hype train will be full-on unable to access, including methods on manifolds (even kernel methods on manifolds) and algebraic-topological methods (persistent homology) with enough maths to give Kagglers the screaming meemies. I'm using some of those for $(redacted, the plan is to make money). Don't follow the crowd.
- mjw 8y ago
  When I started out in ML I was really keen to learn about the most 'mathsy' approaches out there. I think with hindsight, it's great to have a broad spectrum of methods available to you, but if you focus too much on methods at the hard-math end of the spectrum just for the sake of an intellectual challenge, you can end up fixated on an exotic solution looking for a problem while the rest of the field moves on, rather than doing useful engineering people care about. Maybe you find a niche where something exotic really helps, maybe you don't -- maybe for research this is a risk worth taking. But just something to keep in mind. IMO: breadth is good. Mathematical maturity helps. If one sticks around one finds uses for interesting maths eventually, but not worth trying to force it. Another avenue for people who want to use some hardcore math: try and use it to find some good theory around why things which work well, work well. Not an easy task either by any means.
- spitfire 8y ago
  Any papers or books to get started with this? I'm in ML but not on the DL hype train - disappointed by its limits.
- tzahola 8y ago
  Math? Meh. Just throw more GPUs at it bro. Gradient descent 4ever! xD /s
RobertDeNiro 8y ago
You don't need much CS knowledge for deep learning, in fact pure math/stats is more useful. I'd say CS knowledge is only useful if you do RL stuff and even then you don't really need it that much.
- hypersoar 8y ago
  Tell that to my potential employers :|
  - RobertDeNiro 8y ago
    Depends what jobs you're applying to. But I've seen it first hand traditional HR doesn't know how to filter for ML resumes.
- tachyonbeam 8y ago
  That really depends on what you're doing. I've seen people coming from a pure math background really struggle. If you've never programmed before, then using something like TensowFlow isn't exactly going to feel natural. Familiarity with Python and the Linux ecosystem would definitely be quite helpful. Yes, I'm sure you could run ML models on a Windows machine, but at some point, you might want to perform experiments on a cluster, and it's going to be running Linux.
  - RobertDeNiro 8y ago
    True. I was assuming programing experience but no data structure and algorithms. I can attest that a lot of people in my ML class couldn't program and they sucked at the assignments we had.
ilzmastr 8y ago
Are you only interested in ML practicum? What did you take after real analysis? What are your thoughts/interests on analysis for ML, like the approximation theory that branched from Fourier and Wavelet analysis catching on with Cybenko [1], continuing with among others Mhaskar [2], most recently added to by Bolcskei [3]. And then there are other areas where analysis applies like studying the optimization of such networks... (And of course, this is just for NNs. There are other areas of research where analysis comes into ML. And of course, real analysis lays the foundations for probability and statistics, and is not abstracted away in many research areas in these fields.) [1](https://pdfs.semanticscholar.org/05ce/b32839c26c8d2cb38d5529cf7720a68c3fab.pdf https://pdfs.semanticscholar.org/05ce/b32839c26c8d2cb38d5529...) [2](https://pdfs.semanticscholar.org/694a/d455c119c0d07036792b80abbf5488a9a4ca.pdf https://pdfs.semanticscholar.org/694a/d455c119c0d07036792b80...) [3](https://arxiv.org/abs/1705.01714 https://arxiv.org/abs/1705.01714)
mindviews 8y ago
For Machine Learning, I found "Learning From Data" http://amlbook.com/ http://amlbook.com/ to be a very strong, foundational book. It does the best job of any reference I've seen of answering the question: "Is learning possible and feasible?" Provides and excellent mathematical foundation for understanding when and why ML systems should succeed and fail. Associated MOOC from Caltech at https://work.caltech.edu/telecourse.html https://work.caltech.edu/telecourse.html
- saamm 8y ago
  My ML professor used this book, and I loved it. Some presentations of ML make it seem like a bag of tricks; this feels like the opposite of that.
- Eridrus 8y ago
  I'm a big fan of the MOOC, even though it didn't explicitly cover it, it basically explained what people mean when they say "we don't know why neural nets work", which is that our previous generalization bounds would imply that neural nets would not generalize well, but in practice they do.
cs702 8y ago
If you have at least some coding experience and you are interested in the practical aspects of ML/DL (i.e., you want to learn the how-to, not the why or the whence), my recommendation is to start with the fast.ai courses by Jeremy Howard (co-author of this "Matrix Calculus" cheat sheet) and Rachel Thomas[a]: * fast.ai ML course: http://forums.fast.ai/t/another-treat-early-access-to-intro-to-machine-learning-videos/6826 http://forums.fast.ai/t/another-treat-early-access-to-intro-... * fast.ai DL course: part 1: http://course.fast.ai/ http://course.fast.ai/ part 2: http://course.fast.ai/part2.html http://course.fast.ai/part2.html The fast.ai courses spend very little time on theory, and you can follow the videos at your own pace. Books: * The best books on ML (excluding DL), in my view, are "An Introduction to Statistical Learning" by James, Witten, Hastie and Tibshirani, and "The Elements of Statistical Learning" by Hastie, Tibshirani and Friedman. The Elements arguably belongs on every ML practitioner's bookshelf -- it's a fantastic reference manual.[b] * The only book on DL that I'm aware of is "Deep Learning," by Goodfellow, Bengio and Courville. It's a good book, but I suggest holding off on reading it until you've had a chance to experiment with a range of deep learning models. Otherwise, you will get very little useful out of it.[c] Good luck! [a] Scroll down on this page for their bios: http://course.fast.ai/about.html http://course.fast.ai/about.html [b] Introduction to Statistical Learning: http://www-bcf.usc.edu/~gareth/ISL/ http://www-bcf.usc.edu/~gareth/ISL/ The Elements of Statistical Learning: https://web.stanford.edu/~hastie/ElemStatLearn/ https://web.stanford.edu/~hastie/ElemStatLearn/ [c] http://www.deeplearningbook.org/ http://www.deeplearningbook.org/
utk09 8y ago
https://github.com/llSourcell/Learn_Machine_Learning_in_3_Months https://github.com/llSourcell/Learn_Machine_Learning_in_3_Mo... This is also a good resource for MachineLearning

srean 8y ago

Given your math bent expecially analysis, I would recommend you start with Luc Devroye, follow it up with Vapnik. These are a bit dated but still very relevant. After these two go with Alex Smola and Bernard Scholkopf. The books that have been suggested to you so far aren't bad but with high fluff to meat ratio, not meant for a reader like you. If you have function analysis in your bag you will feel right at home with the books I suggested. Deep NN are much in the news, but on the math side of the practice of DNN, there is not much to it apart from chain rule for differentiation.

poster123 8y ago
Thanks for the recommendations. Devroye wrote several books, and I assume the one being mentioned is "A Probabilistic Theory of Pattern Recognition".
- srean 8y ago
  Yes that's the one I had in mind. His other books are also very good, but are more specialized. You may also enjoy graycat's comments on HN. He know his math but is contrarian about machine learning but its good to have that view point.
poster123 8y ago
The book "A Probabilistic Theory of Pattern Recognition" by Devroye, Györfi and Lugosi is available from the site of Gyorfi at http://www.szit.bme.hu/~gyorfi/pbook.pdf http://www.szit.bme.hu/~gyorfi/pbook.pdf .

Iwan-Zotow 8y ago

Sure, Goodfellow book http://www.deeplearningbook.org/ http://www.deeplearningbook.org/

mlthoughts2018 8y ago

I made the same transition earlier in my career. One book on deep learning that meets your requirements is [0]. It’s readable, covers a broad set of modern topics, and has pragmatic tips for real use cases. For general machine learning, there are many, many books. A good intro is [1] and a more comprehensive, reference sort of book is [2]. Frankly, by this point, even reading the documentation and user guide of scikit-learn has a fairly good mathematical presentation of many algorithms. Another good reference book is [3]. Finally, I would also recommend supplementing some of that stuff with Bayesian analysis, which can address many of the same problems, or be intermixed with machine learning algorithms, but which is important for a lot of other reasons too (MCMC sampling, hierarchical regression, small data problems). For that I would recommend [4] and [5]. Stay away from bootcamps or books or lectures that seem overly branded with “data science.” This usually means more focus on data pipeline tooling, data cleaning, shallow details about a specific software package, and side tasks like wrapping something in a webservice. That stuff is extremely easy to learn on the job and usually needs to be tailored differently for every different project or employer, so it’s a relative waste of time unless it is the only way you can get a job. [0]: < https://www.amazon.com/Deep-Learning-Adaptive-Computation-Machine/dp/0262035618 https://www.amazon.com/Deep-Learning-Adaptive-Computation-Ma... > [1]: < https://www.amazon.com/Pattern-Classification-Pt-1-Richard-Duda/dp/0471056693 https://www.amazon.com/Pattern-Classification-Pt-1-Richard-D... > [2]: < https://www.amazon.com/Pattern-Recognition-Learning-Information-Statistics/dp/0387310738 https://www.amazon.com/Pattern-Recognition-Learning-Informat... > [3]: < http://www.web.stanford.edu/~hastie/ElemStatLearn/ http://www.web.stanford.edu/~hastie/ElemStatLearn/ > [4]: < http://www.stat.columbia.edu/~gelman/book/ http://www.stat.columbia.edu/~gelman/book/ > [5]: < http://www.stat.columbia.edu/~gelman/arm/ http://www.stat.columbia.edu/~gelman/arm/ >

soVeryTired 8y ago
+1 for Gelman, but I hate Bishop's book [2]. It was an early go-to reference in the field, but there are better books out there now.
- hikarudo 8y ago
  What do you hate about Bishop's book? I'm genuinely curious.

soVeryTired 8y ago

For non-deep learning, read David Barber's book: http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/090310.pdf http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/090310.pdf Some sections may be less relevant, depending on what you want to do, but Section III is a very good introduction to machine learning methods. Do the exercises as you're reading. Theory is one thing, but in ML my rule of thumb is that you don't really understand a model until you've coded it up. A collection of written exercises would be a good way to impress an interviewer, too.

glitch 8y ago
Note: 090310.pdf (9 March 2010) is an older version of the textbook. The latest revision of the textbook PDF (091117.pdf, 9 November 2017) is available at: → http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/091117.pdf http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/091117.pdf Aside: ⸰ Textbook homepage: http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=Brml.HomePage http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=... ⸰ Online version homepage (should always contain link to latest revision): http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=Brml.Online http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=... ⸰ Directory sorted by date: http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/?C=M;O=D http://web4.cs.ucl.ac.uk/staff/D.Barber/textbook/?C=M;O=D
nightski 8y ago
It really is a great book. However my friend & I attempted to make it through and faced quite a bit of struggle. Eventually we'd figure things out but it felt like it would of been significantly easier with the help of a teacher/mentor to ask questions to. Unfortunately it's actually kind of hard to find colleges nearby that even have courses in Bayesian statistics (from an inference perspective). That was frustrating.
- man-and-laptop 8y ago
  It's far from state of the art, but reading Computer Vision by Simon J D Prince [1] alongside David Barber's BRML can really help. [1] - http://www.computervisionmodels.com/ http://www.computervisionmodels.com/

thanatropism 8y ago

There are many new methods cropping up that most people in the data science hype train will be full-on unable to access, including methods on manifolds (even kernel methods on manifolds) and algebraic-topological methods (persistent homology) with enough maths to give Kagglers the screaming meemies. I'm using some of those for $(redacted, the plan is to make money). Don't follow the crowd.

mjw 8y ago
When I started out in ML I was really keen to learn about the most 'mathsy' approaches out there. I think with hindsight, it's great to have a broad spectrum of methods available to you, but if you focus too much on methods at the hard-math end of the spectrum just for the sake of an intellectual challenge, you can end up fixated on an exotic solution looking for a problem while the rest of the field moves on, rather than doing useful engineering people care about. Maybe you find a niche where something exotic really helps, maybe you don't -- maybe for research this is a risk worth taking. But just something to keep in mind. IMO: breadth is good. Mathematical maturity helps. If one sticks around one finds uses for interesting maths eventually, but not worth trying to force it. Another avenue for people who want to use some hardcore math: try and use it to find some good theory around why things which work well, work well. Not an easy task either by any means.
spitfire 8y ago
Any papers or books to get started with this? I'm in ML but not on the DL hype train - disappointed by its limits.
tzahola 8y ago
Math? Meh. Just throw more GPUs at it bro. Gradient descent 4ever! xD /s

RobertDeNiro 8y ago

You don't need much CS knowledge for deep learning, in fact pure math/stats is more useful. I'd say CS knowledge is only useful if you do RL stuff and even then you don't really need it that much.

hypersoar 8y ago
Tell that to my potential employers :|
- RobertDeNiro 8y ago
  Depends what jobs you're applying to. But I've seen it first hand traditional HR doesn't know how to filter for ML resumes.
tachyonbeam 8y ago
That really depends on what you're doing. I've seen people coming from a pure math background really struggle. If you've never programmed before, then using something like TensowFlow isn't exactly going to feel natural. Familiarity with Python and the Linux ecosystem would definitely be quite helpful. Yes, I'm sure you could run ML models on a Windows machine, but at some point, you might want to perform experiments on a cluster, and it's going to be running Linux.
- RobertDeNiro 8y ago
  True. I was assuming programing experience but no data structure and algorithms. I can attest that a lot of people in my ML class couldn't program and they sucked at the assignments we had.

ilzmastr 8y ago

Are you only interested in ML practicum? What did you take after real analysis? What are your thoughts/interests on analysis for ML, like the approximation theory that branched from Fourier and Wavelet analysis catching on with Cybenko [1], continuing with among others Mhaskar [2], most recently added to by Bolcskei [3]. And then there are other areas where analysis applies like studying the optimization of such networks... (And of course, this is just for NNs. There are other areas of research where analysis comes into ML. And of course, real analysis lays the foundations for probability and statistics, and is not abstracted away in many research areas in these fields.) [1](https://pdfs.semanticscholar.org/05ce/b32839c26c8d2cb38d5529cf7720a68c3fab.pdf https://pdfs.semanticscholar.org/05ce/b32839c26c8d2cb38d5529...) [2](https://pdfs.semanticscholar.org/694a/d455c119c0d07036792b80abbf5488a9a4ca.pdf https://pdfs.semanticscholar.org/694a/d455c119c0d07036792b80...) [3](https://arxiv.org/abs/1705.01714 https://arxiv.org/abs/1705.01714)

mindviews 8y ago

For Machine Learning, I found "Learning From Data" http://amlbook.com/ http://amlbook.com/ to be a very strong, foundational book. It does the best job of any reference I've seen of answering the question: "Is learning possible and feasible?" Provides and excellent mathematical foundation for understanding when and why ML systems should succeed and fail. Associated MOOC from Caltech at https://work.caltech.edu/telecourse.html https://work.caltech.edu/telecourse.html

saamm 8y ago
My ML professor used this book, and I loved it. Some presentations of ML make it seem like a bag of tricks; this feels like the opposite of that.
Eridrus 8y ago
I'm a big fan of the MOOC, even though it didn't explicitly cover it, it basically explained what people mean when they say "we don't know why neural nets work", which is that our previous generalization bounds would imply that neural nets would not generalize well, but in practice they do.

cs702 8y ago

If you have at least some coding experience and you are interested in the practical aspects of ML/DL (i.e., you want to learn the how-to, not the why or the whence), my recommendation is to start with the fast.ai courses by Jeremy Howard (co-author of this "Matrix Calculus" cheat sheet) and Rachel Thomas[a]: * fast.ai ML course: http://forums.fast.ai/t/another-treat-early-access-to-intro-to-machine-learning-videos/6826 http://forums.fast.ai/t/another-treat-early-access-to-intro-... * fast.ai DL course: part 1: http://course.fast.ai/ http://course.fast.ai/ part 2: http://course.fast.ai/part2.html http://course.fast.ai/part2.html The fast.ai courses spend very little time on theory, and you can follow the videos at your own pace. Books: * The best books on ML (excluding DL), in my view, are "An Introduction to Statistical Learning" by James, Witten, Hastie and Tibshirani, and "The Elements of Statistical Learning" by Hastie, Tibshirani and Friedman. The Elements arguably belongs on every ML practitioner's bookshelf -- it's a fantastic reference manual.[b] * The only book on DL that I'm aware of is "Deep Learning," by Goodfellow, Bengio and Courville. It's a good book, but I suggest holding off on reading it until you've had a chance to experiment with a range of deep learning models. Otherwise, you will get very little useful out of it.[c] Good luck! [a] Scroll down on this page for their bios: http://course.fast.ai/about.html http://course.fast.ai/about.html [b] Introduction to Statistical Learning: http://www-bcf.usc.edu/~gareth/ISL/ http://www-bcf.usc.edu/~gareth/ISL/ The Elements of Statistical Learning: https://web.stanford.edu/~hastie/ElemStatLearn/ https://web.stanford.edu/~hastie/ElemStatLearn/ [c] http://www.deeplearningbook.org/ http://www.deeplearningbook.org/

utk09 8y ago

https://github.com/llSourcell/Learn_Machine_Learning_in_3_Months https://github.com/llSourcell/Learn_Machine_Learning_in_3_Mo... This is also a good resource for MachineLearning