6 ms·

Machine Learning Crash Course: Part 2

378 points by llazzaro 10y ago

deleted 10y ago
[deleted]
michaelsbradley 10y ago
Here's Part 1: https://ml.berkeley.edu/blog/2016/11/06/tutorial-1/ https://ml.berkeley.edu/blog/2016/11/06/tutorial-1/
supremus_58 10y ago
does anyone have recommendation on a text that is mathematically heavy but also looks at modern approaches/appleications?
- argonaut 10y ago
  The main reason we see all these relatively information-less blog posts / tutorials is because most people on HN are afraid of the math. If you're not afraid of the math (or have a relatively strong math background), then honestly you can jump into the CS 229 Stanford lecture notes and homework assignments. If that's a little too heavy for you, you can also start with the MOOC version of that course (which is a very watered down version of CS 229).
- davmre 10y ago
  You can order the standard machine learning texts from most to least math-y, and least to most modern: - Pattern Recognition and Machine Learning (Bishop 2007) - Machine Learning: A Probabilistic Perspective (Murphy 2012) - Deep Learning (Goodfellow, Bengio, Courville 2016) If you want cutting-edge material, read the Deep Learning book (which is still quite technical, though some of its content may be outdated in a few years). If you want timeless mathematical foundations very clearly presented, read Bishop. Murphy is a good middle ground. If you're self-teaching and have trouble focusing on a textbook for long periods, the Stanford CS229 lectures combined with Andrew Ng's course notes and assignments are probably the best resource. They are still quite rigorous and working through them will give you a solid foundation, after which you'll be more prepared to understand the deeper content in any of the texts above.
  - gtani 10y ago
    Besides the DL book, there's other excellent texts that are freely available/open content on the web: - Elements statistical Learning, Hastie et al - Shalev-Shwartz and Ben-David: http://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning/understanding-machine-learning-theory-algorithms.pdf http://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning... - the late David MacKay's Info Theory - Bayesian Reasoning in ML, Barber - Hopcroft/Kannan (this is an older version, you can google latest: http://www.cs.cornell.edu/jeh/book112013.pdf http://www.cs.cornell.edu/jeh/book112013.pdf
    - nilknarf 10y ago
      I didn't expect to see Hopcroft/Kannan in that list. I used it in a course taught by Kannan and back then it was called "Computer Science Theory for the Information Age". Apparently the book is now called "Foundations of Data Science". I always thought the old naming was terrible but compared to the new title it actually describes the content a lot better since the book is mainly about CS theory and mathematical foundations for them. It was one of my favorite courses but I would not classify it as ML book. [1]https://www.cs.cmu.edu/~venkatg/teaching/CStheory-infoage/ https://www.cs.cmu.edu/~venkatg/teaching/CStheory-infoage/ [2]https://www.cs.cornell.edu/jeh/book2016June9.pdf https://www.cs.cornell.edu/jeh/book2016June9.pdf
  - wjn0 10y ago
    Where would you place "Elements of Statistical Learning" in relation to these, if you know?
    - platz 10y ago
      You want ISLr (http://www-bcf.usc.edu/~gareth/ISL/ http://www-bcf.usc.edu/~gareth/ISL/), not ESL. ESL is the prototype for the former.
      - argonaut 10y ago
        Not sure why this is downvoted. ISLR is widely considered the "easier to read" / more pedagogical version of ESL. It's still mathematical but is closer to what the poster wants - a textbook that teaches them.
- lindbergh 10y ago
  Also consider Foundations of Machine Learning (2012) by Meryar Mohri. It is a very good book with a statistical learning approach to ML. If you're interested by statistical guarantees to ML algorithms, eg. why the SVM algorithm works and its sample complexity (the expected out of sample error in function of the sample size), it's the book you need.
wiradikusuma 10y ago
Now that there's a bunch of AI/ML-related links in the front page, probably now is the best time to ask: As I learn deep learning, from the practical point-of-view, I found that the idea is simply to feed some "black box" with labeled data so next time it can give you correct label given unlabeled data. In essence, it's pattern recognition. What do you think? And then, as I try to find use cases for ML (you know, finding problem for the solution), I found that actually, many problems that can be solved with ML can actually be solved with rules. For example, detecting transaction fraud. You just need to find the right rules/formula. Forget ML, if you can't hardcode if-else, just use rules engine. What do you think? So, I'm starting to think that ML is good for solving problems where (1) we're too lazy to formulate the rules, or (2) the data is too complex/big to analyze by rules (as in, understanding image or voice). What do you think?
- mmanfrin 10y ago
  As I understand it, ML is about the finding of those rules. The big example I can think of is stocks -- what signals are important in determining the fiscal health of a company or stock? Every trader or investor has their own ruleset they believe in, ML (and specifically Quant) is meant to dig through as much data as possible to find those signals that affect the as-best-understood quality of a pick. For example, a human can think 'oh I will look at numbers like P/E and growth and...' whereas with ML you can feed it all those, and things like number of times the CEO has tweeted in the past week or the number of PR articles or even the language in PRs released to see if there is some strong correlative signal in the bunch, or if it is all just noise.
- pouta 10y ago
  I can't wait to read the answer to this!
- tnecniv 10y ago
  Figuring out the rules is the hardest part of programming (after choosing good names, haha).
- cttet 10y ago
  You are absolute right about most field of applications of ML. But in some fields like computer vision, we humans fail(at least till now) to make rules better than black box neural networks. Also, there are another group of ML people deal with rules+data, I quite agree with this article about the two types of ML: http://projecteuclid.org/download/pdf_1/euclid.ss/1009213726%20 http://projecteuclid.org/download/pdf_1/euclid.ss/1009213726...
Keyframe 10y ago
I also have a question regarding ML. Are there resources where I can see how could I treat video sequences (series of images / spatial/temporal continuity) as inputs? Trying to find a starting point for learning and use case I have in mind.
- windexh8er 10y ago
  I may be off base here but wouldn't CV be part of what you're interested in? If so I've found Adrian Rosebrock has some great paid for and free options: https://www.pyimagesearch.com https://www.pyimagesearch.com (free tutorials in the blog) https://www.pyimagesearch.com/pyimagesearch-gurus/ https://www.pyimagesearch.com/pyimagesearch-gurus/ (paid for "guru" course)
  - Keyframe 10y ago
    You're right, computer vision is/would be definitely a part of it. Thanks for the link. Guru seems a bit pricey for a hobby, but maybe I'll go with it.
- lgessler 10y ago
  There's a really straightforward approach to turning video into feature vectors which you can readily plug into any old ML algorithm. You can turn every decoded frame (which is essentially a PNG, right?) into a (Width*Height)x1 vector, where each cell is an RGB pixel. You can then compose these vectors into a matrix, or perform further operations on them, e.g. SVD. Whether or not this is a good approach, though, will depend of course on your application. For face detection, for example, this is not adequate, and you'll need more sophisticated algorithms. For a good overview of that, see Jason Lawrence's slides[1] and also take a look at some of the other stuff he covered in his Computer Vision course[2]. Sorry if that's too basic for what you were asking. If you want to see some messy code that does this using OpenCV, here's some I wrote a while back with a friend, starting on line 127: https://github.com/sprestwood/CompVisionS2015/blob/master/test_scripts/people_detector.py#L127 https://github.com/sprestwood/CompVisionS2015/blob/master/te... [1] http://www.cs.virginia.edu/~gfx/Courses/2015/TopicsVision/lectures/lecture07_recognition.pdf http://www.cs.virginia.edu/~gfx/Courses/2015/TopicsVision/le... [2] http://www.cs.virginia.edu/~gfx/Courses/2015/TopicsVision/schedule.html http://www.cs.virginia.edu/~gfx/Courses/2015/TopicsVision/sc...
BrandoElFollito 10y ago
A comment I made within this post: Fitting a formula to data implies that there are reasons for the formula to be this and not something else. For example one can prove, within the current model, that the speed of a falling object is v(t) = 1/2 g t^2 and it is reasonable to fit experimental data to this formula to determine g. It may be a false assumption (it is, actually) and this is the whole idea of modelling and understanding limitations of a model. In contrast, the article assumes that a linear expression will be fitted to any kind of data as if they behaved, by miracle, in a linear fashion. Any kind of deduction from this will be false, except if, again by miracle, the data actually behaves linearly. I am a big fan of clusterisation and data behaviour discovery - the process which highlights relationship between data we do not know anything about. I believe this is a huge win in ML. Fitting something (1D, 2D, ...) to data without a model and drawing conclusions is at least perilous.

378 points by llazzaro 10y ago

deleted 10y ago

[deleted]

michaelsbradley 10y ago

Here's Part 1: https://ml.berkeley.edu/blog/2016/11/06/tutorial-1/ https://ml.berkeley.edu/blog/2016/11/06/tutorial-1/

supremus_58 10y ago

does anyone have recommendation on a text that is mathematically heavy but also looks at modern approaches/appleications?

argonaut 10y ago
The main reason we see all these relatively information-less blog posts / tutorials is because most people on HN are afraid of the math. If you're not afraid of the math (or have a relatively strong math background), then honestly you can jump into the CS 229 Stanford lecture notes and homework assignments. If that's a little too heavy for you, you can also start with the MOOC version of that course (which is a very watered down version of CS 229).
davmre 10y ago
You can order the standard machine learning texts from most to least math-y, and least to most modern: - Pattern Recognition and Machine Learning (Bishop 2007) - Machine Learning: A Probabilistic Perspective (Murphy 2012) - Deep Learning (Goodfellow, Bengio, Courville 2016) If you want cutting-edge material, read the Deep Learning book (which is still quite technical, though some of its content may be outdated in a few years). If you want timeless mathematical foundations very clearly presented, read Bishop. Murphy is a good middle ground. If you're self-teaching and have trouble focusing on a textbook for long periods, the Stanford CS229 lectures combined with Andrew Ng's course notes and assignments are probably the best resource. They are still quite rigorous and working through them will give you a solid foundation, after which you'll be more prepared to understand the deeper content in any of the texts above.
- gtani 10y ago
  Besides the DL book, there's other excellent texts that are freely available/open content on the web: - Elements statistical Learning, Hastie et al - Shalev-Shwartz and Ben-David: http://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning/understanding-machine-learning-theory-algorithms.pdf http://www.cs.huji.ac.il/~shais/UnderstandingMachineLearning... - the late David MacKay's Info Theory - Bayesian Reasoning in ML, Barber - Hopcroft/Kannan (this is an older version, you can google latest: http://www.cs.cornell.edu/jeh/book112013.pdf http://www.cs.cornell.edu/jeh/book112013.pdf
  - nilknarf 10y ago
    I didn't expect to see Hopcroft/Kannan in that list. I used it in a course taught by Kannan and back then it was called "Computer Science Theory for the Information Age". Apparently the book is now called "Foundations of Data Science". I always thought the old naming was terrible but compared to the new title it actually describes the content a lot better since the book is mainly about CS theory and mathematical foundations for them. It was one of my favorite courses but I would not classify it as ML book. [1]https://www.cs.cmu.edu/~venkatg/teaching/CStheory-infoage/ https://www.cs.cmu.edu/~venkatg/teaching/CStheory-infoage/ [2]https://www.cs.cornell.edu/jeh/book2016June9.pdf https://www.cs.cornell.edu/jeh/book2016June9.pdf
- wjn0 10y ago
  Where would you place "Elements of Statistical Learning" in relation to these, if you know?
  - platz 10y ago
    You want ISLr (http://www-bcf.usc.edu/~gareth/ISL/ http://www-bcf.usc.edu/~gareth/ISL/), not ESL. ESL is the prototype for the former.
    - argonaut 10y ago
      Not sure why this is downvoted. ISLR is widely considered the "easier to read" / more pedagogical version of ESL. It's still mathematical but is closer to what the poster wants - a textbook that teaches them.
lindbergh 10y ago
Also consider Foundations of Machine Learning (2012) by Meryar Mohri. It is a very good book with a statistical learning approach to ML. If you're interested by statistical guarantees to ML algorithms, eg. why the SVM algorithm works and its sample complexity (the expected out of sample error in function of the sample size), it's the book you need.

wiradikusuma 10y ago

Now that there's a bunch of AI/ML-related links in the front page, probably now is the best time to ask: As I learn deep learning, from the practical point-of-view, I found that the idea is simply to feed some "black box" with labeled data so next time it can give you correct label given unlabeled data. In essence, it's pattern recognition. What do you think? And then, as I try to find use cases for ML (you know, finding problem for the solution), I found that actually, many problems that can be solved with ML can actually be solved with rules. For example, detecting transaction fraud. You just need to find the right rules/formula. Forget ML, if you can't hardcode if-else, just use rules engine. What do you think? So, I'm starting to think that ML is good for solving problems where (1) we're too lazy to formulate the rules, or (2) the data is too complex/big to analyze by rules (as in, understanding image or voice). What do you think?

mmanfrin 10y ago
As I understand it, ML is about the finding of those rules. The big example I can think of is stocks -- what signals are important in determining the fiscal health of a company or stock? Every trader or investor has their own ruleset they believe in, ML (and specifically Quant) is meant to dig through as much data as possible to find those signals that affect the as-best-understood quality of a pick. For example, a human can think 'oh I will look at numbers like P/E and growth and...' whereas with ML you can feed it all those, and things like number of times the CEO has tweeted in the past week or the number of PR articles or even the language in PRs released to see if there is some strong correlative signal in the bunch, or if it is all just noise.
pouta 10y ago
I can't wait to read the answer to this!
tnecniv 10y ago
Figuring out the rules is the hardest part of programming (after choosing good names, haha).
cttet 10y ago
You are absolute right about most field of applications of ML. But in some fields like computer vision, we humans fail(at least till now) to make rules better than black box neural networks. Also, there are another group of ML people deal with rules+data, I quite agree with this article about the two types of ML: http://projecteuclid.org/download/pdf_1/euclid.ss/1009213726%20 http://projecteuclid.org/download/pdf_1/euclid.ss/1009213726...

Keyframe 10y ago

I also have a question regarding ML. Are there resources where I can see how could I treat video sequences (series of images / spatial/temporal continuity) as inputs? Trying to find a starting point for learning and use case I have in mind.

windexh8er 10y ago
I may be off base here but wouldn't CV be part of what you're interested in? If so I've found Adrian Rosebrock has some great paid for and free options: https://www.pyimagesearch.com https://www.pyimagesearch.com (free tutorials in the blog) https://www.pyimagesearch.com/pyimagesearch-gurus/ https://www.pyimagesearch.com/pyimagesearch-gurus/ (paid for "guru" course)
- Keyframe 10y ago
  You're right, computer vision is/would be definitely a part of it. Thanks for the link. Guru seems a bit pricey for a hobby, but maybe I'll go with it.
lgessler 10y ago
There's a really straightforward approach to turning video into feature vectors which you can readily plug into any old ML algorithm. You can turn every decoded frame (which is essentially a PNG, right?) into a (Width*Height)x1 vector, where each cell is an RGB pixel. You can then compose these vectors into a matrix, or perform further operations on them, e.g. SVD. Whether or not this is a good approach, though, will depend of course on your application. For face detection, for example, this is not adequate, and you'll need more sophisticated algorithms. For a good overview of that, see Jason Lawrence's slides[1] and also take a look at some of the other stuff he covered in his Computer Vision course[2]. Sorry if that's too basic for what you were asking. If you want to see some messy code that does this using OpenCV, here's some I wrote a while back with a friend, starting on line 127: https://github.com/sprestwood/CompVisionS2015/blob/master/test_scripts/people_detector.py#L127 https://github.com/sprestwood/CompVisionS2015/blob/master/te... [1] http://www.cs.virginia.edu/~gfx/Courses/2015/TopicsVision/lectures/lecture07_recognition.pdf http://www.cs.virginia.edu/~gfx/Courses/2015/TopicsVision/le... [2] http://www.cs.virginia.edu/~gfx/Courses/2015/TopicsVision/schedule.html http://www.cs.virginia.edu/~gfx/Courses/2015/TopicsVision/sc...

BrandoElFollito 10y ago

A comment I made within this post: Fitting a formula to data implies that there are reasons for the formula to be this and not something else. For example one can prove, within the current model, that the speed of a falling object is v(t) = 1/2 g t^2 and it is reasonable to fit experimental data to this formula to determine g. It may be a false assumption (it is, actually) and this is the whole idea of modelling and understanding limitations of a model. In contrast, the article assumes that a linear expression will be fitted to any kind of data as if they behaved, by miracle, in a linear fashion. Any kind of deduction from this will be false, except if, again by miracle, the data actually behaves linearly. I am a big fan of clusterisation and data behaviour discovery - the process which highlights relationship between data we do not know anything about. I believe this is a huge win in ML. Fitting something (1D, 2D, ...) to data without a model and drawing conclusions is at least perilous.