5 ms·

How to Machine Learn

136 points by gwulf 12y ago

mattbettinson 12y ago
I'm a freshman in university right now, how is linear algebra helpful in computer science? I'm finding it hard to stay motivated as I can't think of any uses outside of graphics. Maybe I'm just not far along in the course though.
- rhgraysonii 12y ago
  A good starting point is singular value decomposition [0] 0. http://en.wikipedia.org/wiki/Singular_value_decomposition http://en.wikipedia.org/wiki/Singular_value_decomposition
- hiddencost 12y ago
  http://math.stackexchange.com/questions/344879/how-does-linear-algebra-help-with-computer-science http://math.stackexchange.com/questions/344879/how-does-line... Pretty much all of machine learning relies on linear algebra. Most scientific computing relies on lI near algebra. Graph algorithms often use linear algebra. If you just want to be a Web Dev or an app developer, it probably doesn't matter, tho. But if you just want to do Web Dev or mobile apps then you don't really need most of computer science.
  - fa 12y ago
    Well, if I remember my university's beginner linear algebra course, there were many topics on the syllabus only due to historic accident, ancestor worship, and theoretical necessities: I remember parallelepipeds, Cramer's rule, solving eigensystems by solving for a polynomial's zeros... Let me tell you how many times I've used parallelepipeds, Cramer's rule, or found eigenvalues via the quadratic formula in the 12 years since linear algebra (and a career in statistical signal processing and machine learning): zero, zero, and zero. Most things in college are important only to the extent that they're gatekeepers for what's really important. What hiddencost says is true, but very little of what you're learning in class is relevant to those important and interesting things. Sorry: college sucks.
    - j2kun 12y ago
      I can point to people who used eigenvalues to make billions and change the world. So... maybe they're not useless.
      - dchapp 12y ago
        It's not that eigenvalues per se are useless--they're plainly not--but that no one finds eigenvalues in practice by computing the characteristic polynomial and solving for its roots. Unfortunately, that computation is often found in HW and exams in US undergraduate linear algebra courses.
        j2kun 12y ago
        People also don't only solve for eigenvalues computationally. Knowing about all the perspectives of eigenvalues helps. I agree the computations are dumb, but if you propose an intro linear algebra class with no computations to soak up test scores you will get far more protests.
        fa 12y ago
        This is especially tragic because matrix factorization algorithms are so deep and interesting, theory and programming-wise! LU, Cholesky, QR, eigendecomposition, SVD, mmmm. Round-off error tolerance, convergence criteria, stability, yum. Characteristic qualities and root finding: bleh.
        dchapp 12y ago
        Not to mention that when your matrix is 5x5 or more, there aren't even general solutions for roots if for some reason you're still insisting on the matrix->polynomial->eigenvalues route.
        Chinjut 12y ago
        Sure there are (in many reasonable senses). Polynomial root extraction just isn't expressible in terms of addition, subtraction, multiplication, division, and nth roots alone. But that's ok; there's nothing magic about that particular set of operations, so as to make it the end-all, be-all.
- deleted 12y ago
  [deleted]
- jostmey 12y ago
  A lot of people don't appreciate the utility of Linear Algebra because they fail to see how the core theorems generalize beyond Cartesian coordinates. Sure, the basic operators of a vector space such as "+" and "*" look boringly familiar, but these operations can be overloaded to carry out other calculations just like in computer programming. The fact that you can represent any "vector space" with a basis set and that every possible basis set for that vector space will have the same dimensionality is pretty cool and useful.
- gms7777 12y ago
  Linear algebra comes up all the time in advanced courses. Certainly in graphics, but also if you do any machine learning, optimization, probabilistic algorithms, any sort of scientific computation. I never took a linear algebra course, convincing myself that I could just pick it up as I went along and getting into grad level computer science courses was ultimately a rather painful experience because of this. If I were you, I'd try to take it seriously and really try to develop an intuitive feel for linear algebra because (depending on what courses you want to take), it can really save you time and headaches in the future.
- powerset 12y ago
  A nice little pep talk motivating study of linear algebra and why it's useful (e.g. pagerank): http://ocw.mit.edu/courses/mathematics/18-06sc-linear-algebra-fall-2011/Syllabus/ http://ocw.mit.edu/courses/mathematics/18-06sc-linear-algebr...
- j2kun 12y ago
  I write a blog on math and programming and I see linear algebra applied every day. 1. Ranking in search engines (more generally, any kind of random walk analysis) [1] 2. Fourier analysis, and as a consequence most signal processing involves some understanding of linear algebra because integrals are linear. [2] 3. Regression [3] and more generally linear modeling of anything. 4. Facial recognition [4] 5. Community detection [5], where most leading methods analyze the spectrum of a graph to find communities. In fact, applied network science in general has a ton of linear algebra. 6. Greedy algorithms are characterized by a kind of generalization of linear systems [6] 7. Linear programming, perhaps the most applied piece of mathematics ever, needs a strong foundation of linear algebra [7] 8. All of quantum computing is literally just linear algebra [8]. 9. Cryptography has a ton of linear algebra in it, and a large portion of the techniques are reasoned about with linear algebra. 10. Most of calculus relies on linear algebra, most importantly optimization [9] 11. Recent data analysis techniques based on topology do so through linear algebra [10] 12. Coding theory, including the algorithms used to correct errors on DVDs. Basically, any time you want to encode data so that you can recover from white noise, you're going to use a linear code. [11] This includes compression techniques. 13. Of course graphics. I could go on... [1]: http://jeremykun.com/2011/06/12/googles-pagerank-introduction/ http://jeremykun.com/2011/06/12/googles-pagerank-introductio... [2]: http://jeremykun.com/2012/07/18/the-fast-fourier-transform/ http://jeremykun.com/2012/07/18/the-fast-fourier-transform/ [3]: http://jeremykun.com/2013/08/18/linear-regression/ http://jeremykun.com/2013/08/18/linear-regression/ [4]: http://jeremykun.com/2011/07/27/eigenfaces/ http://jeremykun.com/2011/07/27/eigenfaces/ [5]: http://jeremykun.com/2014/05/19/community-detection-in-graphs-a-casual-tour/ http://jeremykun.com/2014/05/19/community-detection-in-graph... [6]: http://jeremykun.com/2014/08/26/when-greedy-algorithms-are-perfect-the-matroid/ http://jeremykun.com/2014/08/26/when-greedy-algorithms-are-p... [7]: http://jeremykun.com/2014/06/02/linear-programming-and-the-most-affordable-healthy-diet-part-1/ http://jeremykun.com/2014/06/02/linear-programming-and-the-m... [8]: http://jeremykun.com/2014/12/08/a-motivation-for-quantum-computing/ http://jeremykun.com/2014/12/08/a-motivation-for-quantum-com... [9]: http://jeremykun.com/2013/11/30/lagrangians-for-the-amnesiac/ http://jeremykun.com/2013/11/30/lagrangians-for-the-amnesiac... [10]: http://jeremykun.com/2013/04/10/computing-homology/ http://jeremykun.com/2013/04/10/computing-homology/ [11]: http://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction http://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_corr...
anindyabd 12y ago
The website for the project linked to on the first line is based on this Bootstrap template: http://startbootstrap.com/template-overviews/grayscale/ http://startbootstrap.com/template-overviews/grayscale/. At least they could have changed the background photo!
- gwulf 12y ago
  Good find! The goal was to wrap the feature engineer scripts up with a front end, so design was an after thought.
raz32dust 12y ago
Useful list. I would recommend adding one more resource for linear algebra/machine learning: I absolutely enjoyed learning Linear Algebra from these beautiful lectures by Prof. Gilbert Strang (MIT): https://www.youtube.com/watch?v=ZK3O402wf1c&list=PLE7DDD91010BC51F8 https://www.youtube.com/watch?v=ZK3O402wf1c&list=PLE7DDD9101... Seriously, I gained a new found appreciation for Linear algebra after going through these lectures. You should go over some of these lectures even if you already know linear algebra - it might give you insights you never had before (it did, for me). Absolute must-watch if you are into machine learning or related areas.

136 points by gwulf 12y ago

I'm a freshman in university right now, how is linear algebra helpful in computer science? I'm finding it hard to stay motivated as I can't think of any uses outside of graphics. Maybe I'm just not far along in the course though.

rhgraysonii 12y ago
A good starting point is singular value decomposition [0] 0. http://en.wikipedia.org/wiki/Singular_value_decomposition http://en.wikipedia.org/wiki/Singular_value_decomposition
hiddencost 12y ago
http://math.stackexchange.com/questions/344879/how-does-linear-algebra-help-with-computer-science http://math.stackexchange.com/questions/344879/how-does-line... Pretty much all of machine learning relies on linear algebra. Most scientific computing relies on lI near algebra. Graph algorithms often use linear algebra. If you just want to be a Web Dev or an app developer, it probably doesn't matter, tho. But if you just want to do Web Dev or mobile apps then you don't really need most of computer science.
- fa 12y ago
  Well, if I remember my university's beginner linear algebra course, there were many topics on the syllabus only due to historic accident, ancestor worship, and theoretical necessities: I remember parallelepipeds, Cramer's rule, solving eigensystems by solving for a polynomial's zeros... Let me tell you how many times I've used parallelepipeds, Cramer's rule, or found eigenvalues via the quadratic formula in the 12 years since linear algebra (and a career in statistical signal processing and machine learning): zero, zero, and zero. Most things in college are important only to the extent that they're gatekeepers for what's really important. What hiddencost says is true, but very little of what you're learning in class is relevant to those important and interesting things. Sorry: college sucks.
  - j2kun 12y ago
    I can point to people who used eigenvalues to make billions and change the world. So... maybe they're not useless.
    - dchapp 12y ago
      It's not that eigenvalues per se are useless--they're plainly not--but that no one finds eigenvalues in practice by computing the characteristic polynomial and solving for its roots. Unfortunately, that computation is often found in HW and exams in US undergraduate linear algebra courses.
      - j2kun 12y ago
        People also don't only solve for eigenvalues computationally. Knowing about all the perspectives of eigenvalues helps. I agree the computations are dumb, but if you propose an intro linear algebra class with no computations to soak up test scores you will get far more protests.
      - fa 12y ago
        This is especially tragic because matrix factorization algorithms are so deep and interesting, theory and programming-wise! LU, Cholesky, QR, eigendecomposition, SVD, mmmm. Round-off error tolerance, convergence criteria, stability, yum. Characteristic qualities and root finding: bleh.
        dchapp 12y ago
        Not to mention that when your matrix is 5x5 or more, there aren't even general solutions for roots if for some reason you're still insisting on the matrix->polynomial->eigenvalues route.
        Chinjut 12y ago
        Sure there are (in many reasonable senses). Polynomial root extraction just isn't expressible in terms of addition, subtraction, multiplication, division, and nth roots alone. But that's ok; there's nothing magic about that particular set of operations, so as to make it the end-all, be-all.
deleted 12y ago
[deleted]
jostmey 12y ago
A lot of people don't appreciate the utility of Linear Algebra because they fail to see how the core theorems generalize beyond Cartesian coordinates. Sure, the basic operators of a vector space such as "+" and "*" look boringly familiar, but these operations can be overloaded to carry out other calculations just like in computer programming. The fact that you can represent any "vector space" with a basis set and that every possible basis set for that vector space will have the same dimensionality is pretty cool and useful.
gms7777 12y ago
Linear algebra comes up all the time in advanced courses. Certainly in graphics, but also if you do any machine learning, optimization, probabilistic algorithms, any sort of scientific computation. I never took a linear algebra course, convincing myself that I could just pick it up as I went along and getting into grad level computer science courses was ultimately a rather painful experience because of this. If I were you, I'd try to take it seriously and really try to develop an intuitive feel for linear algebra because (depending on what courses you want to take), it can really save you time and headaches in the future.
powerset 12y ago
A nice little pep talk motivating study of linear algebra and why it's useful (e.g. pagerank): http://ocw.mit.edu/courses/mathematics/18-06sc-linear-algebra-fall-2011/Syllabus/ http://ocw.mit.edu/courses/mathematics/18-06sc-linear-algebr...
j2kun 12y ago
I write a blog on math and programming and I see linear algebra applied every day. 1. Ranking in search engines (more generally, any kind of random walk analysis) [1] 2. Fourier analysis, and as a consequence most signal processing involves some understanding of linear algebra because integrals are linear. [2] 3. Regression [3] and more generally linear modeling of anything. 4. Facial recognition [4] 5. Community detection [5], where most leading methods analyze the spectrum of a graph to find communities. In fact, applied network science in general has a ton of linear algebra. 6. Greedy algorithms are characterized by a kind of generalization of linear systems [6] 7. Linear programming, perhaps the most applied piece of mathematics ever, needs a strong foundation of linear algebra [7] 8. All of quantum computing is literally just linear algebra [8]. 9. Cryptography has a ton of linear algebra in it, and a large portion of the techniques are reasoned about with linear algebra. 10. Most of calculus relies on linear algebra, most importantly optimization [9] 11. Recent data analysis techniques based on topology do so through linear algebra [10] 12. Coding theory, including the algorithms used to correct errors on DVDs. Basically, any time you want to encode data so that you can recover from white noise, you're going to use a linear code. [11] This includes compression techniques. 13. Of course graphics. I could go on... [1]: http://jeremykun.com/2011/06/12/googles-pagerank-introduction/ http://jeremykun.com/2011/06/12/googles-pagerank-introductio... [2]: http://jeremykun.com/2012/07/18/the-fast-fourier-transform/ http://jeremykun.com/2012/07/18/the-fast-fourier-transform/ [3]: http://jeremykun.com/2013/08/18/linear-regression/ http://jeremykun.com/2013/08/18/linear-regression/ [4]: http://jeremykun.com/2011/07/27/eigenfaces/ http://jeremykun.com/2011/07/27/eigenfaces/ [5]: http://jeremykun.com/2014/05/19/community-detection-in-graphs-a-casual-tour/ http://jeremykun.com/2014/05/19/community-detection-in-graph... [6]: http://jeremykun.com/2014/08/26/when-greedy-algorithms-are-perfect-the-matroid/ http://jeremykun.com/2014/08/26/when-greedy-algorithms-are-p... [7]: http://jeremykun.com/2014/06/02/linear-programming-and-the-most-affordable-healthy-diet-part-1/ http://jeremykun.com/2014/06/02/linear-programming-and-the-m... [8]: http://jeremykun.com/2014/12/08/a-motivation-for-quantum-computing/ http://jeremykun.com/2014/12/08/a-motivation-for-quantum-com... [9]: http://jeremykun.com/2013/11/30/lagrangians-for-the-amnesiac/ http://jeremykun.com/2013/11/30/lagrangians-for-the-amnesiac... [10]: http://jeremykun.com/2013/04/10/computing-homology/ http://jeremykun.com/2013/04/10/computing-homology/ [11]: http://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction http://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_corr...

anindyabd 12y ago

The website for the project linked to on the first line is based on this Bootstrap template: http://startbootstrap.com/template-overviews/grayscale/ http://startbootstrap.com/template-overviews/grayscale/. At least they could have changed the background photo!

gwulf 12y ago
Good find! The goal was to wrap the feature engineer scripts up with a front end, so design was an after thought.

raz32dust 12y ago

Useful list. I would recommend adding one more resource for linear algebra/machine learning: I absolutely enjoyed learning Linear Algebra from these beautiful lectures by Prof. Gilbert Strang (MIT): https://www.youtube.com/watch?v=ZK3O402wf1c&list=PLE7DDD91010BC51F8 https://www.youtube.com/watch?v=ZK3O402wf1c&list=PLE7DDD9101... Seriously, I gained a new found appreciation for Linear algebra after going through these lectures. You should go over some of these lectures even if you already know linear algebra - it might give you insights you never had before (it did, for me). Absolute must-watch if you are into machine learning or related areas.