14 ms·

Introduction to Applied Linear Algebra: Vectors, Matrices, and Least Squares

938 points by yarapavan 8y ago

VikingCoder 8y ago
I'm going to complain about this every chance I get. A 2D vector, we generally store as [x, y, 0]. What's the extra 0? The homogeneous coordinate. A 2D point, we generally store as [x, y, 1]. That extra 1 is the homogeneous coordinate, and since it's there, it means "and apply translations!" If I have a 2D transform, I put the translation component in the last row or column, depending on if you pre-multiply or post-multiply (I can never remember which). When I transform a vector by that matrix, the 0 in the homogeneous coordinate means translation doesn't apply. Perfect! But what if I have a 3D vector? Well... I end up with [x, y, z, 0], right? Ugh. If instead, we stored the homogeneous coordinate in the FIRST position, [0, x, y] for 2D, and [0, x, y, z] for 3D, etc. then it's just a sparse vector! Set the values you want to! [0] is the 0-vector in any number of dimensions! [1] is the origin point in any number of dimensions! Why did we put the homogeneous coordinate last in all our internal representations? It was so dumb!
- twtw 8y ago
  I don't follow. What do we gain by moving the homogeneous coordinate from last position to first? I don't understand this: > then it's just a sparse vector! Set the variables you want to! Or this: > [1] is the origin point in any number of dimensions. Could you clarify? Also, I don't think this book even discusses homogeneous coordinates. It would be sort of unusual for this type of general text and the only mention of "homogeneous" in the index is "homogeneous equation."
  - edflsafoiewq 8y ago
    I think what's meant is that by putting it first, you can always treat any point in P^n as a point in P^m by just ignoring the extra numbers if m < n or by treating the missing numbers as all being 0 if m > n. That is the point in P^2, [1 x y], can also be regarded as the point [1 x] in P^1, the point [1 x y 0] in P^3, the point [1 x y 0 0] in P^4, etc. This is in contrast to putting it last, where if you have [x y 1] in P^2 and you want the point in P^1 you need to allocate a new list [x 1], etc. The vector is sparse in the sense that you can regard a point as being an infinitely long list of numbers of which we are sparsely giving only that prefix that is non-zero (like how you can regard a decimal numeral as being an infinitely long list of digits, all the ones that are missing being 0). [1] is the origin point in any dimension because it is [1] in P^0, [1 0] in P^1, [1 0 0] in P^2, etc.
- deleted 8y ago
  [deleted]
- starmole 8y ago
  I hope you are just trolling. Homogeneous coordinates are for projection, not for selecting translation.
  - twtw 8y ago
    Homogeneous coordinates allow for affine transformations to be represented and performed with a matrix multiply. This includes perspective projection and translation. One of the use cases for homogeneous coordinates is certainly to be able to achieve translation.
- chengiz 8y ago
  With due respect, who is "we" and what are you talking about? The book does not as much as mention a homogeneous coordinate and uses 2-arrays for 2D vectors.
  - VikingCoder 8y ago
    I'm off on a tangent unrelated to the book. "we" is computer programmers who deal with vectors and points. Mostly graphics, physics, games, etc.
    - joppy 8y ago
      Technically the projective coordinate (3,2,1) should be exactly the same as (6,4,2), and every nonzero multiple thereof. So it’s not really correct to say that (x, y, 0) represents a vector, or that adding these projective coordinates is vector addition. Vector addition is represented by an identity matrix with the x, y coordinates in the rightmost column.
muhneesh 8y ago
I'm e-learning Linear Algebra right now to have a good math foundation for Machine Learning. I was a History and Sociology major in college - so I didn't take any math. If you are like me, and working off an initial base of high school math, I would recommend the following (all free): Linear Algebra Foundations to Frontiers (UT Austin) Course: https://www.edx.org/course/linear-algebra-foundations-to-frontiers https://www.edx.org/course/linear-algebra-foundations-to-fro... Comments: This was a great starting place for me. Good interactive HW exercises, very clear instruction and time-efficient. Linear Algebra (MIT OpenCourseware) Course: https://ocw.mit.edu/courses/mathematics/18-06-linear-algebra-spring-2010/ https://ocw.mit.edu/courses/mathematics/18-06-linear-algebra... Comments: This course is apparently the holy grail course for Intro Linear Algebra. One of my colleagues, who did an MS in EE at MIT, said Gilbert Strang was the best teacher he had. I started off with this but had to rewind to the UT class because I didn't have some of the fundamentals (e.g. how to calc a dot product). I'm personally 15% through this, but enjoying it. Linear Algebra Review PDF (Stanford CS229) Link: http://cs229.stanford.edu/section/cs229-linalg.pdf http://cs229.stanford.edu/section/cs229-linalg.pdf Comments: This is the set of Linear Algebra review materials they go over at the beginning of Stanford's machine learning class (CS229). This is my workback to know I'm tracking to the right set of knowledge, and thus far, the courses have done a great job of doing so.
- flor1s 8y ago
  Don't forget to review calculus as well. Khan Academy is a good start for learning about single variable calculus (http://www.khanacademy.org http://www.khanacademy.org), but their content on multivariable calculus is a bit lacking (neural networks / deep learning use the concept of the derivatives and the gradient a lot). A good supplement for multivariable calculus would be Terence Parr and Jeremy Howard's article on "All the matrix calculus you need for deep learning": https://explained.ai/matrix-calculus/index.html https://explained.ai/matrix-calculus/index.html
  - muhneesh 8y ago
    Thanks - I am doing that as well! I've been using MIT OpenCourseware for single variable calculus (and will do the same for multivariable). I fenced the parent post to Linear Algebra to not go too far away from the OP. I will certainly check out the Terrence Parr / Jeremy Howard site, and am super familiar with Khan Academy.
flor1s 8y ago
After reading this book (or during), take a look at the author's (Boyd) video lectures on linear dynamical systems: https://see.stanford.edu/Course/EE263/ https://see.stanford.edu/Course/EE263/ There is a lot of overlap between the book and the course.
- chrispeel 8y ago
  Also take a look at Boyd and Vandenberghe's book on convex optimization: https://web.stanford.edu/~boyd/cvxbook/ https://web.stanford.edu/~boyd/cvxbook/
StefanKarpinski 8y ago
This is a beautiful book and a great intro to the basics of linear algebra. All the figures in the book are generated in Julia and there’s a companion book with Julia code for computational examples: http://vmls-book.stanford.edu/vmls-julia-companion.pdf http://vmls-book.stanford.edu/vmls-julia-companion.pdf
chobytes 8y ago
Just finished my intro linear algebra class yesterday. The class was a bit more abstract in nature, so some of the chapters in this look like they could be nice application oriented follow ups to it!
herostratus101 8y ago
How much overlap is there between this and Stanford's EE263? Can someone in the know comment on the differences between what is covered?
- vasili111 8y ago
  Comment below: https://news.ycombinator.com/item?id=18678713 https://news.ycombinator.com/item?id=18678713
TobiasA 8y ago
I'm a self taught programmer with a very weak maths background. What's the best learning path for me if I want to be able to understand and create ML based applications?
- hackermailman 8y ago
  There's a practical course for this http://www.datasciencecourse.org/lectures/ http://www.datasciencecourse.org/lectures/ anything you don't know, like linear algebra, look up the topics here for a 1-2hr crash course https://www.youtube.com/playlist?list=PLm3J0oaFux3aafQm568blS9blxtA_EWQv https://www.youtube.com/playlist?list=PLm3J0oaFux3aafQm568bl... There's a playlist for a math background in ML for anybody who wants to try a more rigorous ML course https://www.youtube.com/playlist?list=PL7y-1rk2cCsA339crwXMWUaBRuLBvPBCg https://www.youtube.com/playlist?list=PL7y-1rk2cCsA339crwXMW... More information, including recommended texts https://canvas.cmu.edu/courses/603/assignments/syllabus https://canvas.cmu.edu/courses/603/assignments/syllabus but don't let that list of prereqs discourage you, can easily look them up directly. You don't have to understand all of Linear Algebra to do matrix multiplication. There's plenty of ML books, papers and playlists on youtube for a full course in ML from dozens of universities https://www.cs.cmu.edu/~roni/10601/ https://www.cs.cmu.edu/~roni/10601/ (click on 2017 lectures) Note never trust YouTube or any other resource to be around forever, make sure you archive everything before you start taking it as lectures tend to disappear (then seed them for others ^^ ) If you have a really weak background go through this free book, refuse to not be able to complete it https://infinitedescent.xyz/ https://infinitedescent.xyz/ There's no answers because the author gives thanks to a grad course in evidenced based teaching where he claims the only way to really know something and remember it is to figure it out for yourself. Math stackexchange can help too.
  - Aromaisn 8y ago
    Fantastic selection of material here. Also used that YouTube playlist myself and it was incredibly valuable. +1 for archiving these resources.
  - TobiasA 8y ago
    Thank you so much, this is perfect!
mmmmpancakes 8y ago
A book on applied linear algebra with a focus on regression and no mention anywhere of the singular value decomposition??
- herostratus101 8y ago
  He does cover SVD in EE263, and all of the lectures for that course are freely available online.
- IgniteTheSun 8y ago
  This book has a lot of very interesting applications and seems to cover information not normally found in first books on Linear Algebra (e.g., makes use of calculus, Taylor series, etc) and the authors are EEs, not mathematicians. It doesn't, however, cover several topics normally covered in the first year of linear algebra (e.g., vector spaces, subspaces, nullspace, eigenvalues, singular values; see pp 461-462). As with most engineering books, there are no solutions provided. An excellent supplement to other Linear Algebra textbooks. Given its focus on applications, will hold the interest of engineers and other technical folk but may not be loved by mathematicians who may prefer a more rigorous approach.
deleted 8y ago
[deleted]
thrwmlaccnt 8y ago
Is theoretical linear algebra at the level of axler helpful for machine learning? If so, in what ways?
nicebill8 8y ago
The 3B1B series on Linear Algebra is by far the most welcoming and informative introduction to the topic I've ever seen: https://www.youtube.com/watch?v=fNk_zzaMoSs&list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab https://www.youtube.com/watch?v=fNk_zzaMoSs&list=PLZHQObOWTQ...
- aidos 8y ago
  Any linear algebra post I come here to comment the same (if someone else hasn’t done it already). Seriously, this series is absolutely wonderful.
graycat 8y ago
I went though the slides. Super fun material! I've seen all the methods long ago, and much deeper than in the slides, and published on some of the most advanced material, and much more, but, still, it was fun material because of the many examples and really good graphs. From their other books, clearly they are real experts. The slides, then, are a careful path where minimal theory gives a LOT of nice applications. The theory they give is nearly always so simple that they are able, in just a few lines, to give essentially the proofs, nearly always. E.g., I never saw any mention of convexity, and these two guys are right at the top of experts on theory and applications of convexity, so that it is clear that they tried hard to get lots of applications from minimal theory. They did next to nothing on numerical stability -- some mention might have been good. There's a still easier derivation of the least squares normal equations based on perpendicular projections -- they might have included that. That is, if drop a golf ball to the floor, the line to the floor and the shortest distance to the floor is the line perpendicular to the floor. This fact generalizes. They have illustrated a nice, general lesson: Can do such applications with just finite dimensions and/or discreteness. Can do more theory with continuous instead of discrete values and infinite instead of finite dimensions. But, then, even with the extra theory, often challenging, commonly for the computing are back to discreteness and finiteness. Sooooo, just omit the more advanced theory and just stay discrete and finite throughout -- that's one of the themes in the slides. With this theme, the slides are able to do at least something interesting and potentially valuable from stacks of texts in pure and applied math, statistics, and more with just a few slides, simple math, nice graphs, and a few words. Nice. E.g., they did a lot of applied statistics without mentioning probability theory! How'd they do that? They just stayed with the data and omitted describing the probabilistic context from which the data was samples or estimates. Cute. But, readers, be warned -- the probabilistic context should not be neglected; eventually should learn that, too. Another cute omission of theory -- vector subspaces and the, really, the axioms of a vector space. E.g., that "floor" I mentioned above is such a subspace. How'd they do that? They just stayed with the basic example vector spaces they had in mind and managed to avoid talking about subspaces. At one point they touched on determinants for the 2 x 2 case, mentioned that that result is important (should be remembered or some such), that there is a more general approach that don't have to remember!!! Determinants have some value here and there, e.g., show some continuity results right away and have some nice connections with volumes, but they are tricky to explain and CAN be omitted!!! Uh, there is an easier proof of the Schwartz inequality based on Bessel's inequality. Since they did enough with orthogonality to do Bessel's inequality, they could have used that approach to the Schwartz inequality -- I first saw in P. Halmos. They didn't make clear the close connections among inner products, covariance, and correlation -- maybe some readers will see those connections from what is in the slides. They did the QT decomposition -- nice -- that is, for square matrix A, we can write A = QT where Q is orthogonal and T is triangular. They used that to solve systems of linear equations but omitted Gauss elimination and the associated approaches to numerical stability. For the Q, they emphasized the Gram-Schmidt process but neglected to mention that it's numerically unstable -- no wonder since are commonly subtracting large numbers whose difference is small, the basic sin in numerical analysis. Of course, the authors are EE profs. Then it is interesting that another theme in the slides is getting close to much of the work in what computer science calls machine learning. E.g., their few slides on using classification to recognize digits 0-9 in handwriting is really cute, especially the graph that shows the sizes of the coefficients on top of the square that has the input data of hand written digits so that see which parts of the input data are the most relevant to the calculation. Cute. Of course, there's much more to those fields that they omitted than included, but that's true also for even the best 5 star hotel luncheon buffet!!! More fun stuff at https://news.ycombinator.com/item?id=18648999 https://news.ycombinator.com/item?id=18648999
k__ 8y ago
Somehow I found linear algebra easier than calculus, but I don't know why. I did both at the same time in university, but failed calculus 3 times and aced linear algebra at the first try. I'd expect being either good or bad at math, not both at the same time
- impendia 8y ago
  Math professor here --- Quality of teaching might have something to do with it. But, also, calculus is much harder to understand at a rigorous, formal level than at an informal level. On one level you can try to understand what the main concepts are about, be able to compute derivatives and integrals, solve optimization and related rates problems, and so on. I'd recommend Silvanus Thompson's Calculus Made Easy over any mainstream calculus book for this. In my opinion, the book succeeds amazingly at fulfilling the promise of its title. But suppose you really try to read any mainstream calculus book, and understand everything. For example: - Why are limits defined the way they are (with epsilons and deltas)? - The book will probably touch lightly upon the Mean Value Theorem -- why is this important? What's the point? - Why is the chain rule true? It reads dy/dx = (dy/du) (du/dx). Yay! This is just cancelling fractions, right? Any "respectable" calculus book will insist that it's not, but most students will cheerfully ignore this, still get correct answers to the homework problems, and sleep fine at night. - Consider the function e^x. How is it defined? The informal way is to say e = 2.71828... and we define exponents "as usual". Most students are perfectly happy with this. But does this really make sense if x is irrational? Your calculus book might bend over backwards to define everything properly (e^x is the inverse to ln(x), which is defined as a definite integral), and it takes a lot of work to appreciate why. In my experience, these sorts of issues mostly don't pop up in linear algebra, where the proofs tend to parallel the handwavy heuristics. I wonder if this had anything to do with your experience?
  - k__ 8y ago
    I was really bad at rearanging terms/formulars and this came up in calculus exams all the time, but not so much in linear algebra exams.
    - 8y ago
CamperBob2 8y ago
Kind of inauspicious to see this kind of thing on page 5: A (standard) unit vector is a vector with all elements equal to zero, except one element which is equal to one Huh? I've never heard the term "standard vector" before, and a "unit vector" is a vector whose magnitude is one. There is no requirement that one element be equal to one. Maybe that's why the download is free...
- RossBencina 8y ago
  > A (standard) unit vector... I know this as "an element of the standard basis," B = {e_1, e_2, ...), where e_1 = (1,0,0,...), e_2 = (0,1,0,0,...). You could view it as inauspicious that the treatment doesn't begin with abstract vector spaces, but there is always Axler. For what it's worth, I find it inauspicious that after taking three (pure-math oriented) Linear Algebra courses I never saw least squares nor the SVD. I'm looking forward to taking a look at Prof Boyd's book.
  - CamperBob2 8y ago
    Point being, their definition is just plain wrong. If that's how the authors describe a unit vector, I don't think this is the book you want to use to learn about SVD.
    - Koshkin 8y ago
      Well, the way I see it, when you teach applied science you often want to sacrifice some rigor so that your students could focus on what was intended to be learned in the first place.
- umanwizard 8y ago
  But there is the requirement that the other elements be equal to zero. It's right there in the sentence you quoted.
  - CamperBob2 8y ago
    It's right there in the sentence you quoted. True enough, thanks -- fixed. Still a completely bogus definition.
nyc111 8y ago
> 2-vector (x1,x2) can represent a location or a displacement in 2-D... Isn’t this fundamentally faulty? Same notation describing a point and displacement. From this, we may conclude that, a point and a displacement are the same thing because they are described by the same notation. Shouldn’t mathematics be free of such contextual interpretation?
- Koshkin 8y ago
  My advice would be not to get stuck on these “philosophical” questions, if your goal is to actually learn math, and instead just press on and keep learning and solving real problems. Eventually the fog will dissolve by itself, and these kinds of questions will seem to you either naive or devoid any real substance, or just uninteresting compared to everything else that you have learned.
  - nyc111 8y ago
    No. This advice does not apply to me. I don't want to learn mathematics. I'm more interested in learning parts of mathematics that interest me at the moment. And I think philosophy comes before mathematics. Or mathematics is the philosophy of quantities. Both philosopnies are based on definitions.
- earthicus 8y ago
  There's no issue with the notation; I think you've misunderstood the mathematical idea. Consider a more familiar algebraic object, a real number, x. This can model a length, an area, volume, time, time interval, temperature, weight, speed, physical constant, geometric ratio, fractional dimension, etc... In mathematics, we abstract by forgetting about what the things are, and retain information about how they behave, and about what abstract properties they satisfy. The insight is that 2d locations and 2d displacements have the same abstract properties, which are modeled by a certain algebraic object: 2-vectors.
  - nyc111 8y ago
    Thanks for the explanation. Makes sense. Something else I noticed, vector notation does not specify a coordinate system. V = (1, 2) is just an array of two numbers. The cartesian coordinate interpretation is a choice we make. Correct?
syntaxing 8y ago
Applied linear algebra is such a great idea. Linear algebra is relatively easy to understand and used everywhere. But the material is so damn boring since it's a lot of arthimetic. Even the homework problem is boring since there is no specific purpose.
- cultus 8y ago
  Typical LA courses in math departments have a bizarre focus on being able to do Gaussian elimination by hand and stuff like that. It's not particularly useful or even mathematically interesting. LA courses would be so much more useful if they just stuck to theory and only had computer applications.
  - all2 8y ago
    I found knowing the mechanics of Gaussian elimination to be really helpful when learning about algorithms like SVD or using Householder transformations. Knowing how the matrices became triangularized gave me something to hang the new information on.
pylus 8y ago
Ijust flipped some pages and saw it is great book. Wished my school offered this.
aaaaaaaaaab 8y ago
Prof. Boyd is a great teacher! I highly recommend his course on linear dynamical systems [0] and convex optimization [1] too. The former is beginner stuff, and while convex optimization is more advanced, both are very engaging and clearly explained, with lots of anecdotes and practical examples! [0] https://see.stanford.edu/Course/EE263 https://see.stanford.edu/Course/EE263 [1] https://www.youtube.com/playlist?list=PL3940DD956CDF0622 https://www.youtube.com/playlist?list=PL3940DD956CDF0622
mipmap04 8y ago
Linear algebra is probably my favorite part of math from a practicality standpoint. I'm not in a math heavy field, but knowing how to use matrices to solve optimization problems has been very helpful.
prince005 8y ago
Proceed with caution..
wpmoradi 8y ago
Great Resource! But a good primer to Linear Algebra would be Gilbert Strangs course at MIT OCW.
squidgyhead 8y ago
You might be interested in the Open Textbook Initiative by the American Institute of Mathematics: https://aimath.org/textbooks/ https://aimath.org/textbooks/
gxs 8y ago
Sort of off topic - but I was a math major in college and I always had a broad, casual categorization of the types of classes I took. Linear Algebra - felt like it required you to be able to hold really long trains of thought in your head Probability - felt like you had to be clever Analysis - felt like you just had to think critically and approach things from all angles I always preferred Algebra - felt like I was writing essays not doing math

938 points by yarapavan 8y ago

I'm going to complain about this every chance I get. A 2D vector, we generally store as [x, y, 0]. What's the extra 0? The homogeneous coordinate. A 2D point, we generally store as [x, y, 1]. That extra 1 is the homogeneous coordinate, and since it's there, it means "and apply translations!" If I have a 2D transform, I put the translation component in the last row or column, depending on if you pre-multiply or post-multiply (I can never remember which). When I transform a vector by that matrix, the 0 in the homogeneous coordinate means translation doesn't apply. Perfect! But what if I have a 3D vector? Well... I end up with [x, y, z, 0], right? Ugh. If instead, we stored the homogeneous coordinate in the FIRST position, [0, x, y] for 2D, and [0, x, y, z] for 3D, etc. then it's just a sparse vector! Set the values you want to! [0] is the 0-vector in any number of dimensions! [1] is the origin point in any number of dimensions! Why did we put the homogeneous coordinate last in all our internal representations? It was so dumb!

twtw 8y ago
I don't follow. What do we gain by moving the homogeneous coordinate from last position to first? I don't understand this: > then it's just a sparse vector! Set the variables you want to! Or this: > [1] is the origin point in any number of dimensions. Could you clarify? Also, I don't think this book even discusses homogeneous coordinates. It would be sort of unusual for this type of general text and the only mention of "homogeneous" in the index is "homogeneous equation."
- edflsafoiewq 8y ago
  I think what's meant is that by putting it first, you can always treat any point in P^n as a point in P^m by just ignoring the extra numbers if m < n or by treating the missing numbers as all being 0 if m > n. That is the point in P^2, [1 x y], can also be regarded as the point [1 x] in P^1, the point [1 x y 0] in P^3, the point [1 x y 0 0] in P^4, etc. This is in contrast to putting it last, where if you have [x y 1] in P^2 and you want the point in P^1 you need to allocate a new list [x 1], etc. The vector is sparse in the sense that you can regard a point as being an infinitely long list of numbers of which we are sparsely giving only that prefix that is non-zero (like how you can regard a decimal numeral as being an infinitely long list of digits, all the ones that are missing being 0). [1] is the origin point in any dimension because it is [1] in P^0, [1 0] in P^1, [1 0 0] in P^2, etc.
deleted 8y ago
[deleted]
starmole 8y ago
I hope you are just trolling. Homogeneous coordinates are for projection, not for selecting translation.
- twtw 8y ago
  Homogeneous coordinates allow for affine transformations to be represented and performed with a matrix multiply. This includes perspective projection and translation. One of the use cases for homogeneous coordinates is certainly to be able to achieve translation.
chengiz 8y ago
With due respect, who is "we" and what are you talking about? The book does not as much as mention a homogeneous coordinate and uses 2-arrays for 2D vectors.
- VikingCoder 8y ago
  I'm off on a tangent unrelated to the book. "we" is computer programmers who deal with vectors and points. Mostly graphics, physics, games, etc.
  - joppy 8y ago
    Technically the projective coordinate (3,2,1) should be exactly the same as (6,4,2), and every nonzero multiple thereof. So it’s not really correct to say that (x, y, 0) represents a vector, or that adding these projective coordinates is vector addition. Vector addition is represented by an identity matrix with the x, y coordinates in the rightmost column.

muhneesh 8y ago

I'm e-learning Linear Algebra right now to have a good math foundation for Machine Learning. I was a History and Sociology major in college - so I didn't take any math. If you are like me, and working off an initial base of high school math, I would recommend the following (all free): Linear Algebra Foundations to Frontiers (UT Austin) Course: https://www.edx.org/course/linear-algebra-foundations-to-frontiers https://www.edx.org/course/linear-algebra-foundations-to-fro... Comments: This was a great starting place for me. Good interactive HW exercises, very clear instruction and time-efficient. Linear Algebra (MIT OpenCourseware) Course: https://ocw.mit.edu/courses/mathematics/18-06-linear-algebra-spring-2010/ https://ocw.mit.edu/courses/mathematics/18-06-linear-algebra... Comments: This course is apparently the holy grail course for Intro Linear Algebra. One of my colleagues, who did an MS in EE at MIT, said Gilbert Strang was the best teacher he had. I started off with this but had to rewind to the UT class because I didn't have some of the fundamentals (e.g. how to calc a dot product). I'm personally 15% through this, but enjoying it. Linear Algebra Review PDF (Stanford CS229) Link: http://cs229.stanford.edu/section/cs229-linalg.pdf http://cs229.stanford.edu/section/cs229-linalg.pdf Comments: This is the set of Linear Algebra review materials they go over at the beginning of Stanford's machine learning class (CS229). This is my workback to know I'm tracking to the right set of knowledge, and thus far, the courses have done a great job of doing so.

flor1s 8y ago
Don't forget to review calculus as well. Khan Academy is a good start for learning about single variable calculus (http://www.khanacademy.org http://www.khanacademy.org), but their content on multivariable calculus is a bit lacking (neural networks / deep learning use the concept of the derivatives and the gradient a lot). A good supplement for multivariable calculus would be Terence Parr and Jeremy Howard's article on "All the matrix calculus you need for deep learning": https://explained.ai/matrix-calculus/index.html https://explained.ai/matrix-calculus/index.html
- muhneesh 8y ago
  Thanks - I am doing that as well! I've been using MIT OpenCourseware for single variable calculus (and will do the same for multivariable). I fenced the parent post to Linear Algebra to not go too far away from the OP. I will certainly check out the Terrence Parr / Jeremy Howard site, and am super familiar with Khan Academy.

flor1s 8y ago

After reading this book (or during), take a look at the author's (Boyd) video lectures on linear dynamical systems: https://see.stanford.edu/Course/EE263/ https://see.stanford.edu/Course/EE263/ There is a lot of overlap between the book and the course.

chrispeel 8y ago
Also take a look at Boyd and Vandenberghe's book on convex optimization: https://web.stanford.edu/~boyd/cvxbook/ https://web.stanford.edu/~boyd/cvxbook/

StefanKarpinski 8y ago

This is a beautiful book and a great intro to the basics of linear algebra. All the figures in the book are generated in Julia and there’s a companion book with Julia code for computational examples: http://vmls-book.stanford.edu/vmls-julia-companion.pdf http://vmls-book.stanford.edu/vmls-julia-companion.pdf

chobytes 8y ago

Just finished my intro linear algebra class yesterday. The class was a bit more abstract in nature, so some of the chapters in this look like they could be nice application oriented follow ups to it!

herostratus101 8y ago

How much overlap is there between this and Stanford's EE263? Can someone in the know comment on the differences between what is covered?

vasili111 8y ago
Comment below: https://news.ycombinator.com/item?id=18678713 https://news.ycombinator.com/item?id=18678713

TobiasA 8y ago

I'm a self taught programmer with a very weak maths background. What's the best learning path for me if I want to be able to understand and create ML based applications?

hackermailman 8y ago
There's a practical course for this http://www.datasciencecourse.org/lectures/ http://www.datasciencecourse.org/lectures/ anything you don't know, like linear algebra, look up the topics here for a 1-2hr crash course https://www.youtube.com/playlist?list=PLm3J0oaFux3aafQm568blS9blxtA_EWQv https://www.youtube.com/playlist?list=PLm3J0oaFux3aafQm568bl... There's a playlist for a math background in ML for anybody who wants to try a more rigorous ML course https://www.youtube.com/playlist?list=PL7y-1rk2cCsA339crwXMWUaBRuLBvPBCg https://www.youtube.com/playlist?list=PL7y-1rk2cCsA339crwXMW... More information, including recommended texts https://canvas.cmu.edu/courses/603/assignments/syllabus https://canvas.cmu.edu/courses/603/assignments/syllabus but don't let that list of prereqs discourage you, can easily look them up directly. You don't have to understand all of Linear Algebra to do matrix multiplication. There's plenty of ML books, papers and playlists on youtube for a full course in ML from dozens of universities https://www.cs.cmu.edu/~roni/10601/ https://www.cs.cmu.edu/~roni/10601/ (click on 2017 lectures) Note never trust YouTube or any other resource to be around forever, make sure you archive everything before you start taking it as lectures tend to disappear (then seed them for others ^^ ) If you have a really weak background go through this free book, refuse to not be able to complete it https://infinitedescent.xyz/ https://infinitedescent.xyz/ There's no answers because the author gives thanks to a grad course in evidenced based teaching where he claims the only way to really know something and remember it is to figure it out for yourself. Math stackexchange can help too.
- Aromaisn 8y ago
  Fantastic selection of material here. Also used that YouTube playlist myself and it was incredibly valuable. +1 for archiving these resources.
- TobiasA 8y ago
  Thank you so much, this is perfect!

mmmmpancakes 8y ago

A book on applied linear algebra with a focus on regression and no mention anywhere of the singular value decomposition??

herostratus101 8y ago
He does cover SVD in EE263, and all of the lectures for that course are freely available online.
IgniteTheSun 8y ago
This book has a lot of very interesting applications and seems to cover information not normally found in first books on Linear Algebra (e.g., makes use of calculus, Taylor series, etc) and the authors are EEs, not mathematicians. It doesn't, however, cover several topics normally covered in the first year of linear algebra (e.g., vector spaces, subspaces, nullspace, eigenvalues, singular values; see pp 461-462). As with most engineering books, there are no solutions provided. An excellent supplement to other Linear Algebra textbooks. Given its focus on applications, will hold the interest of engineers and other technical folk but may not be loved by mathematicians who may prefer a more rigorous approach.

deleted 8y ago

[deleted]

thrwmlaccnt 8y ago

Is theoretical linear algebra at the level of axler helpful for machine learning? If so, in what ways?

nicebill8 8y ago

The 3B1B series on Linear Algebra is by far the most welcoming and informative introduction to the topic I've ever seen: https://www.youtube.com/watch?v=fNk_zzaMoSs&list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab https://www.youtube.com/watch?v=fNk_zzaMoSs&list=PLZHQObOWTQ...

aidos 8y ago
Any linear algebra post I come here to comment the same (if someone else hasn’t done it already). Seriously, this series is absolutely wonderful.

graycat 8y ago

I went though the slides. Super fun material! I've seen all the methods long ago, and much deeper than in the slides, and published on some of the most advanced material, and much more, but, still, it was fun material because of the many examples and really good graphs. From their other books, clearly they are real experts. The slides, then, are a careful path where minimal theory gives a LOT of nice applications. The theory they give is nearly always so simple that they are able, in just a few lines, to give essentially the proofs, nearly always. E.g., I never saw any mention of convexity, and these two guys are right at the top of experts on theory and applications of convexity, so that it is clear that they tried hard to get lots of applications from minimal theory. They did next to nothing on numerical stability -- some mention might have been good. There's a still easier derivation of the least squares normal equations based on perpendicular projections -- they might have included that. That is, if drop a golf ball to the floor, the line to the floor and the shortest distance to the floor is the line perpendicular to the floor. This fact generalizes. They have illustrated a nice, general lesson: Can do such applications with just finite dimensions and/or discreteness. Can do more theory with continuous instead of discrete values and infinite instead of finite dimensions. But, then, even with the extra theory, often challenging, commonly for the computing are back to discreteness and finiteness. Sooooo, just omit the more advanced theory and just stay discrete and finite throughout -- that's one of the themes in the slides. With this theme, the slides are able to do at least something interesting and potentially valuable from stacks of texts in pure and applied math, statistics, and more with just a few slides, simple math, nice graphs, and a few words. Nice. E.g., they did a lot of applied statistics without mentioning probability theory! How'd they do that? They just stayed with the data and omitted describing the probabilistic context from which the data was samples or estimates. Cute. But, readers, be warned -- the probabilistic context should not be neglected; eventually should learn that, too. Another cute omission of theory -- vector subspaces and the, really, the axioms of a vector space. E.g., that "floor" I mentioned above is such a subspace. How'd they do that? They just stayed with the basic example vector spaces they had in mind and managed to avoid talking about subspaces. At one point they touched on determinants for the 2 x 2 case, mentioned that that result is important (should be remembered or some such), that there is a more general approach that don't have to remember!!! Determinants have some value here and there, e.g., show some continuity results right away and have some nice connections with volumes, but they are tricky to explain and CAN be omitted!!! Uh, there is an easier proof of the Schwartz inequality based on Bessel's inequality. Since they did enough with orthogonality to do Bessel's inequality, they could have used that approach to the Schwartz inequality -- I first saw in P. Halmos. They didn't make clear the close connections among inner products, covariance, and correlation -- maybe some readers will see those connections from what is in the slides. They did the QT decomposition -- nice -- that is, for square matrix A, we can write A = QT where Q is orthogonal and T is triangular. They used that to solve systems of linear equations but omitted Gauss elimination and the associated approaches to numerical stability. For the Q, they emphasized the Gram-Schmidt process but neglected to mention that it's numerically unstable -- no wonder since are commonly subtracting large numbers whose difference is small, the basic sin in numerical analysis. Of course, the authors are EE profs. Then it is interesting that another theme in the slides is getting close to much of the work in what computer science calls machine learning. E.g., their few slides on using classification to recognize digits 0-9 in handwriting is really cute, especially the graph that shows the sizes of the coefficients on top of the square that has the input data of hand written digits so that see which parts of the input data are the most relevant to the calculation. Cute. Of course, there's much more to those fields that they omitted than included, but that's true also for even the best 5 star hotel luncheon buffet!!! More fun stuff at https://news.ycombinator.com/item?id=18648999 https://news.ycombinator.com/item?id=18648999

k__ 8y ago

Somehow I found linear algebra easier than calculus, but I don't know why. I did both at the same time in university, but failed calculus 3 times and aced linear algebra at the first try. I'd expect being either good or bad at math, not both at the same time

impendia 8y ago
Math professor here --- Quality of teaching might have something to do with it. But, also, calculus is much harder to understand at a rigorous, formal level than at an informal level. On one level you can try to understand what the main concepts are about, be able to compute derivatives and integrals, solve optimization and related rates problems, and so on. I'd recommend Silvanus Thompson's Calculus Made Easy over any mainstream calculus book for this. In my opinion, the book succeeds amazingly at fulfilling the promise of its title. But suppose you really try to read any mainstream calculus book, and understand everything. For example: - Why are limits defined the way they are (with epsilons and deltas)? - The book will probably touch lightly upon the Mean Value Theorem -- why is this important? What's the point? - Why is the chain rule true? It reads dy/dx = (dy/du) (du/dx). Yay! This is just cancelling fractions, right? Any "respectable" calculus book will insist that it's not, but most students will cheerfully ignore this, still get correct answers to the homework problems, and sleep fine at night. - Consider the function e^x. How is it defined? The informal way is to say e = 2.71828... and we define exponents "as usual". Most students are perfectly happy with this. But does this really make sense if x is irrational? Your calculus book might bend over backwards to define everything properly (e^x is the inverse to ln(x), which is defined as a definite integral), and it takes a lot of work to appreciate why. In my experience, these sorts of issues mostly don't pop up in linear algebra, where the proofs tend to parallel the handwavy heuristics. I wonder if this had anything to do with your experience?
- k__ 8y ago
  I was really bad at rearanging terms/formulars and this came up in calculus exams all the time, but not so much in linear algebra exams.
  - 8y ago

CamperBob2 8y ago

Kind of inauspicious to see this kind of thing on page 5: A (standard) unit vector is a vector with all elements equal to zero, except one element which is equal to one Huh? I've never heard the term "standard vector" before, and a "unit vector" is a vector whose magnitude is one. There is no requirement that one element be equal to one. Maybe that's why the download is free...

RossBencina 8y ago
> A (standard) unit vector... I know this as "an element of the standard basis," B = {e_1, e_2, ...), where e_1 = (1,0,0,...), e_2 = (0,1,0,0,...). You could view it as inauspicious that the treatment doesn't begin with abstract vector spaces, but there is always Axler. For what it's worth, I find it inauspicious that after taking three (pure-math oriented) Linear Algebra courses I never saw least squares nor the SVD. I'm looking forward to taking a look at Prof Boyd's book.
- CamperBob2 8y ago
  Point being, their definition is just plain wrong. If that's how the authors describe a unit vector, I don't think this is the book you want to use to learn about SVD.
  - Koshkin 8y ago
    Well, the way I see it, when you teach applied science you often want to sacrifice some rigor so that your students could focus on what was intended to be learned in the first place.
umanwizard 8y ago
But there is the requirement that the other elements be equal to zero. It's right there in the sentence you quoted.
- CamperBob2 8y ago
  It's right there in the sentence you quoted. True enough, thanks -- fixed. Still a completely bogus definition.

nyc111 8y ago

> 2-vector (x1,x2) can represent a location or a displacement in 2-D... Isn’t this fundamentally faulty? Same notation describing a point and displacement. From this, we may conclude that, a point and a displacement are the same thing because they are described by the same notation. Shouldn’t mathematics be free of such contextual interpretation?

Koshkin 8y ago
My advice would be not to get stuck on these “philosophical” questions, if your goal is to actually learn math, and instead just press on and keep learning and solving real problems. Eventually the fog will dissolve by itself, and these kinds of questions will seem to you either naive or devoid any real substance, or just uninteresting compared to everything else that you have learned.
- nyc111 8y ago
  No. This advice does not apply to me. I don't want to learn mathematics. I'm more interested in learning parts of mathematics that interest me at the moment. And I think philosophy comes before mathematics. Or mathematics is the philosophy of quantities. Both philosopnies are based on definitions.
earthicus 8y ago
There's no issue with the notation; I think you've misunderstood the mathematical idea. Consider a more familiar algebraic object, a real number, x. This can model a length, an area, volume, time, time interval, temperature, weight, speed, physical constant, geometric ratio, fractional dimension, etc... In mathematics, we abstract by forgetting about what the things are, and retain information about how they behave, and about what abstract properties they satisfy. The insight is that 2d locations and 2d displacements have the same abstract properties, which are modeled by a certain algebraic object: 2-vectors.
- nyc111 8y ago
  Thanks for the explanation. Makes sense. Something else I noticed, vector notation does not specify a coordinate system. V = (1, 2) is just an array of two numbers. The cartesian coordinate interpretation is a choice we make. Correct?

syntaxing 8y ago

Applied linear algebra is such a great idea. Linear algebra is relatively easy to understand and used everywhere. But the material is so damn boring since it's a lot of arthimetic. Even the homework problem is boring since there is no specific purpose.

cultus 8y ago
Typical LA courses in math departments have a bizarre focus on being able to do Gaussian elimination by hand and stuff like that. It's not particularly useful or even mathematically interesting. LA courses would be so much more useful if they just stuck to theory and only had computer applications.
- all2 8y ago
  I found knowing the mechanics of Gaussian elimination to be really helpful when learning about algorithms like SVD or using Householder transformations. Knowing how the matrices became triangularized gave me something to hang the new information on.

pylus 8y ago

Ijust flipped some pages and saw it is great book. Wished my school offered this.

aaaaaaaaaab 8y ago

Prof. Boyd is a great teacher! I highly recommend his course on linear dynamical systems [0] and convex optimization [1] too. The former is beginner stuff, and while convex optimization is more advanced, both are very engaging and clearly explained, with lots of anecdotes and practical examples! [0] https://see.stanford.edu/Course/EE263 https://see.stanford.edu/Course/EE263 [1] https://www.youtube.com/playlist?list=PL3940DD956CDF0622 https://www.youtube.com/playlist?list=PL3940DD956CDF0622

mipmap04 8y ago

Linear algebra is probably my favorite part of math from a practicality standpoint. I'm not in a math heavy field, but knowing how to use matrices to solve optimization problems has been very helpful.

prince005 8y ago

Proceed with caution..

wpmoradi 8y ago

Great Resource! But a good primer to Linear Algebra would be Gilbert Strangs course at MIT OCW.

squidgyhead 8y ago

You might be interested in the Open Textbook Initiative by the American Institute of Mathematics: https://aimath.org/textbooks/ https://aimath.org/textbooks/

gxs 8y ago

Sort of off topic - but I was a math major in college and I always had a broad, casual categorization of the types of classes I took. Linear Algebra - felt like it required you to be able to hold really long trains of thought in your head Probability - felt like you had to be clever Analysis - felt like you just had to think critically and approach things from all angles I always preferred Algebra - felt like I was writing essays not doing math