9 ms·
An Intuitive Guide to Linear Algebra
- textminer 14y agoGreat post. A suggested follow-up: "Banach Spaces, The Spectral Theorem, and Your Changing Body: A Personal Introduction."
- deleted 14y ago[deleted]
- btown 14y agoAs an electrical-engineer turned machine-learning-grad-student, linear transformations have been involved in most everything I do since my first year of undergrad. But all this time, I've done matrix multiplication the way I was taught in high school: "The (i,j) element of AB is what you get by walking right across the i'th row of A while you walk down the j'th column of B, taking the sum of products as you go." It works, but there's no connection between that process and the intuition of a linear transformation; it's just a rote computation. And checking a long string of matrix multiplications to see if they intuitively make sense (shouldn't everything intuitively make sense?) is especially aggravating when you constantly have to interrupt your intuition to switch to a rote calculation. I never thought to think of the columns of B as vectors that physically travel through A; to think of a dataflow or pipeline from right to left on the page. Sure, it's not a cure-all, but it'll be a useful mental tool to have. Oh, and it's also an excellent introduction to the subject, although the Linear Operations section gets a bit muddled... first something's not a linear operation, and then it is, wat? Still, an excellent post.
- wging 14y agoYou might get some mileage out of thinking of a row in the matrix product as a dot product, with the intuition that goes along with that. The dot product is perhaps easier to tie into the geometric intuition you have for linear transformations.
- btilly 14y agoI hope that this matches how some people think enough to help them. For me it is too computational. I prefer understanding the topic from first principles as described at http://news.ycombinator.com/item?id=4086325 http://news.ycombinator.com/item?id=4086325. (Then again I don't particularly like spreadsheets either.)
- elteto 14y ago"The eigenvectors are the axes of the transformation" = mind blown. After several engineering courses, studying eigenvectors in an advanced math class and still no one could put it this simple. This guy is amazing.
- textminer 14y agoHis description of the determinant, too. I hadn't heard that explanation until a second semester of real analysis, when learning the proof of the Inverse Function Theorem (an amazing thing to study, by the way, connects many the dots between linear algebra and calculus). Even then, it was a question I had to ask my brilliant, constantly-pissed looking young professor. "Hey, uh, the Jacobian... what does the determinant mean, uh, geometrically?". He looked at me like a slug, before explaining it was the measure of the newly mapped unit square. Fireworks went off in my head. Two linear algebra classes before only ever explained it by its algorithm or its usefulness (e.g., ∃ A^(-1) for A \in R^{n,n} iff det(A) != 0) Side note: that professor had the most effective teaching style for pure math I've ever seen. Besides lectures that expanded on the contents of Rudin and interesting problem sets, he gave us a list of a hundred theorems, propositions, and exercises. Told us the final exam would be six problems, four of which would come from that list, another of which would be a clever new one, the last something truly hard. Never learned analysis better than when sitting down and working through (not memorizing) each of those proofs and theorems for possible later recapitulation.
- wicker 14y agoAny chance you know where I could find that list or something like it?
- kens 14y agoDeterminant as volume of the transformation is cool, but I still don't "get" the determinant intuitively despite many years of math. In particular, why does the determinant work to solve linear equations (i.e. Cramer's rule)? And what's the motivation behind the formula for the determinant? (I realize these questions are a bit vague, but I'm hoping for a more intuitive answer than "that's just the way the math works out".)
- mturmon 14y agoNice work. It's a richly geometric area. Other things that might be nice to include are dot products (projecting one vector onto another as a measure of co-linearity) and rotation matrices (you could keep it to 3x3).
- valgaze 14y agoStrang from MIT has an extraordinary set of lectures (his textbook is even better): http://ocw.mit.edu/courses/mathematics/18-06-linear-algebra-spring-2010/video-lectures/ http://ocw.mit.edu/courses/mathematics/18-06-linear-algebra-...
- fferen 14y agoI'm watching them now, and they are simply superb, probably the best math class I've ever "taken". I particularly liked his "4 ways of matrix multiplication" (lecture 3) and explanation of fourier series (lecture 24), but his exposition of every concept from determinants to eigenvalues has been intuitive and memorable.
- jackfoxy 14y agoCan't say enough good things about Prof. Gilbert Strang http://www-math.mit.edu/~gs/ http://www-math.mit.edu/~gs/ He is one of my heros, and yeah, I have copied his Linear Algebra lectures to CD so I can speed them up, stop them, and in case they ever disappear from the net. Not to take away from the OP, who I think has some good ideas for making this stuff intuitive.
- hdivider 14y agoFunny, I'm actually working on a game that intends to teach some of this stuff to a non-mathematical audience. Linear algebra is so far-reaching, I find it surprising that other branches of mathematics often seem to get preferential treatment (usually normal algebra and geometry), in spite of the fact that linear algebra is both: a) fairly advanced (i.e. not often taught in school, at least not the deeper stuff) b) not very difficult to learn (unlike lots of other 'introductory' topics in mathematics). Perhaps there is something about matrices (being mere tables of numbers for most folks) that people find unattractive, almost statistics-like. (On the other hand, it could be a simple extension of the symbol barrier [1], given those long vertical brackets.) [1] Prof Keith Devlin introduces this concept here: http://profkeithdevlin.org/2012/02/22/how-to-design-video-games-that-support-good-math-learning-level-3/ http://profkeithdevlin.org/2012/02/22/how-to-design-video-ga...
- sadga 14y agoI think an intuitive guide should have more diagrams of affine transformations. Don't just visualize the matrix, visualize the transformation.
- gtani 14y agoThese're good too http://linear.ups.edu/xml/latest/fcla-xml-latest.xml#fcla-xml-latestli14.xml http://linear.ups.edu/xml/latest/fcla-xml-latest.xml#fcla-xm... http://joshua.smcvt.edu/linearalgebra/book.pdf http://joshua.smcvt.edu/linearalgebra/book.pdf Also, the common 1st year texts (Anton, Lang, Hoffman/Kunze and Friedberg/insel/Spence) can be found easily for cheap, used. The old edition of Strang I used to have was good too, but some people react really strongly when you bring it up. There's lots of ways ot sequence LA and the needs of EE's, econometrics/game theory, prob/stats and applied mathies are different from physics/math majors. Look at ToC's and read the Holyoke prof's writeup: http://www.amazon.com/Introduction-Linear-Algebra-Fourth-Gilbert/product-reviews/0980232716/ http://www.amazon.com/Introduction-Linear-Algebra-Fourth-Gil... (i think they're 1st year texts, my Dad's a physics prof and he started talking about determinants around 7th grade)
- eru 14y agoWhat the author uses as his strawman is a linear algebra course for engineers. As mathematicians, we didn't do any of this matrix and vector stuff with numbers when introducing linear algebra in university. There were a bunch of axioms, and you proved things. That's how you know. What the author sees as abstract "(2d vectors! 3d vectors!)" was way more applied than the stuff we dealt in. But, granted, the purpose wasn't learning about how to get mini-spreadsheets for equations. It was about how to rigorously navigate a useful axiomatic setting. (Later on, we proved that you can find a base, and write down your linear transformation as a bunch of numbers and call that a matrix; same for points and vectors. But we always saw that as somewhat ugly, and anyway limited to the finite dimensional case.)
- marshallp 14y agoAre there practical applications for proving theorems? It seems like it's the full employment act for pencil pushers. edit: downvoters downvote instead of providing refutation. A lot in common with fundamental religionists.
- klodolph 14y agoIf you can't prove a theorem, then it's not a theorem, it's a conjecture. Conjectures are not as useful. For example, the Shannon-Hartley theorem tells us under what circumstances it is possible to decode a radio signal. That theorem lets us design, e.g., cell phone networks. Theorems are useful in all sorts of unexpected ways. At the turn of the 20th century group theory was considered a worthless corner of mathematics, but understanding group theory allows modern chemical instrumentation to work. edit: > It seems like it's the full employment act for pencil pushers > downvoters downvote instead of providing refutation. A lot in common with fundamental religionists You've proven the downvoters right by piling one ad hominem attack upon another.
- marshallp 14y agoThe example you've described (radio signals) is just a method of winning an argument. It doesn't actually help you, you practically do it by writing a computer program that tries out different parameters to get the right one.
- verroq 14y agoI think it's much clearer to define everything in terms of vector spaces and linear operators.
- backprojection 14y agoI entirely agree. People tend to run away from abstract math, with the impression that it's difficult. But the whole point is that a more abstract view of these things makes them easier to understand, not harder.
- gizmo686 14y agoThe problem is that abstractions are designed to make hard problems harder. Someone who has never seen the type of problem that would make a given abstraction useful would not be able to understand it easily. If you work you're way up, then it becomes obvious.
- adeelk 14y ago> Someone who has never seen the type of problem that would make a given abstraction useful would not be able to understand it easily. Whether or not someone is sufficiently motivated to study something should not affect their ability to understand it. Vector spaces and linear maps are actually much easier to understand, and I think that at some point in mathematics (probably your first analysis course) your motivation has to come from the beauty of the theory itself rather than some real-world application.
- ced 14y agoTerry Tao has a great set of lecture notes available online, too. http://www.math.ucla.edu/~tao/resource/general/115a.3.02f/ http://www.math.ucla.edu/~tao/resource/general/115a.3.02f/
- achompas 14y agoThis should be at the top of this HN thread. A shame that it's buried down here--these look extremely valuable. Thanks for the link!
- dsrguru 14y agoEero Simoncelli's guide to linear algebra is very good too and likely more concise, although it doesn't cover eigenvectors and eigenvalues. http://www.cns.nyu.edu/~eero/NOTES/geomLinAlg.pdf http://www.cns.nyu.edu/~eero/NOTES/geomLinAlg.pdf
- marshallp 14y agoI might be missing something big, but it seems like linear algebra is then just an overly complicated way of describing what should be a simple toolbox of spreadsheet manipulation functions, kind of like an extra module for python. Why is it given such special emphasis then, it would be like teaching the datetime or sched modules.
- Evbn 14y agoWhat does linear algebra have to do with a spreadsheet? What are you imagining when you hear "linear algebra"?
- marshallp 14y agoDid you read the article?
- binarysolo 14y agoThe other way around -- Linear Algebra is a (computationally) simple way to describe a (computationally) complicated toolbox of manipulation functions across a huge data set/space.
- marshallp 14y agoLearning a few functions that deal with spreadsheets is a lot more simpler and faster (and hence useful to a lot more people) then doing linear algebra courses complete with proofs and 18th century symbols. What will have more societal impact? Should anyone wishing to use a wordprocessor be required to understand turing machines and computer architecture?
- weichi 14y agoI hope the author is reading your comments, because they make it clear that at least one of his readers is coming away with seriously incorrect ideas about linear algebra! The thing that you are missing is that linear algebra is about much more than matrix multiplication. It is about the properties of linear transformations in general. While matrix multiplication may have simple implementations in spreadsheets, many other concepts in linear algebra don't, for example calculations of and properties of eigenvectors and the corresponding eigenvalues, or the various matrix decompositions (themselves closely related to eigenvalues and eigenvectors). The matrix decompositions in particular give results that not only are mathematically interesting, but are enormously important computationally. In terms of societal impact, other than calculus, no field of mathematics has had a greater impact than linear algebra. From quantum mechanics to the google search engine to machine learning, linear algebra is fundamental. Spreadsheets are awesome and a brilliant idea, but they are aimed at solving a different set of problems. Important though they may be, they pale in comparison to linear algebra.
- lomendil 14y agoI found "Linear Algebra Done Right" to be a much more intuitive introduction to the subject. It doesn't get to determinants until the end. http://linear.axler.net/ http://linear.axler.net/ After going through that course I finally understood things like eigenvectors, null spaces, and projections. Now I see them everywhere (unless you think that's a curse)
- Evbn 14y agoEvery now and again I think my highfalutin college courses were an overpriced waste, and then a conversation like this comes along and I see that the fundamentals of my courses are apparently radical and unheard of most other places. Do schools really not teach the underlying concepts of math, or do people just fail to understand them the first time through and then blame their teachers? /took linear algebra in the math department, then TAed it.
- marshallp 14y agoThey were a waste. You paid an arm and a leg to learn some spreadsheet manipulation functions.
- shardling 14y ago>Do schools really not teach the underlying concepts of math, or do people just fail to understand them the first time through and then blame their teachers? It is absolutely the case that a great many students will not remember seeing material that they definitely were exposed to. As someone who's TA'd physics classes for a few years, I'd often ask a recitation whether they'd covered some particular subject yet in lecture, and they'd often say (as a class) no. I'd ask the professor later, and of course they had -- but students rarely go into lecture having done the reading or prep work, and so have a piss-poor retention rate.
- snogglethorpe 14y ago> students rarely go into lecture having done the reading or prep work, and so have a piss-poor retention rate. I've found that even if do the reading, my brain is often, well, a bit leaky... TT
- klodolph 14y ago> Linear algebra gives you mini-spreadsheets for your math equations. Okay, that lets you visualize it (in the finite case) but it's a terrible way to sell it. Spreadsheets are booooring. Did you know that functions are vectors? Okay, better. Did you know that quantum mechanics is all about linear algebra? Okay, sold! 1. Almost any time you work in more than one dimension you will want linear algebra in your toolbox. There are a zillion methods for solving (non-linear) equations out there, and in more than one dimension, they use linear algebra. Newton's method? Incredibly useful in practice due (quadratic convergence rocks!), and with some linear algebra sauce BOOM you have Newton's method in as many dimensions as you can sneeze at. 2. Oh, by the way... did you know that the Fourier transform is linear? 3. Back to quantum mechanics... there's a thing you can do with a linear operator (a matrix is a kind of linear operator) where you get the "spectrum" of the linear operator. It's useful for making sense of big matrices. But in QM, the wavefunctions for electrons are described as eigenfunctions of a linear operator, and taking the "spectrum" of the linear operator gives you the actual spectrum of light that the chemical under study emits. Hence the name, "spectral theorem". It may be linear algebra on paper, but it's laser beams and semiconductors in the real world. 4. Oh hey, want to learn about infinite-dimensional vector spaces? Maybe some other time.. 5. It's hella useful for modeling. Any model is wrong, but Markov processes are useful. Say you run an agency that rents out moving vans, and you have facilities in 30 cities. Vans rented in city A have a 10% chance of being dropped off in city B, 7% in city C, 9.2% in city D, etc. At this rate, how long till you run out of vans in city F? It's a differential equations problem with like 30 different equations! Or you could rewrite it as a single equation with matrices. You'll end up with weird things like 'e^(A*t)' where A is a matrix, and you thought "no way I can exponentiate to the power of a matrix" but spectral decomposition is like "yes way!" and you can solve the equation by diagonalization. Radical! (Basically, linear algebra rescues differential equations from the pits of intractability. I'm using rental vans as an example, but it could be a chemical reaction or a nuclear reaction or a million subway riders or whatever you want.) So the question is: Do you find economics, quantum physics, chemistry, engineering, classical mechanics, machine learning, statistics, etc. useful? Then get some linear algebra in you!
- marshallp 14y agoHate to do this (but will have to anyway because you're misrepresenting that authors article), - you've mixed in with nonlinear phenomena (most physical processes) with linear, and there's less and less reason to pretend linearity with increasing computer speeds (although in fairness you alluded to models being wrong) - poo pooed spreadsheets without explaining why (you alluded to QM etc. but they don't care about useless proofs in linear algebra unless they're pencil pushing time wasters) - implied infinite-dimensions is practically useful when it isn't (unless you're a mathematician/theoretical-somethingist seeking to extract tax-payer money) There's little need to get linear algebra in you unless you want to waste time.
- pfortuny 14y agoOK can you explain kernels and their role in transformations (f.e. the dimension formula)? No you can't this way. And it truly has an INTUITIVE explanation (so much more than the intuitive ess of a spreadsheet!).
- marshallp 14y agoWhat's the practical use for understanding that intuitively (for the 1000s of people who have to go through it every year)? Genuinely curious.
- pfortuny 14y agoWhy there are things that, for example, block other things when watching TV (2d TV) and where they are. Why they are all "in the same direction"... This is the first, simplest, off-hand example I can come up with. Things are blocked "in parallel" (assuming the camera is far away from the objects) because that projection is a linear map (I repeat, assuming "far away focus") and has a one dimensional kernel: 2d + 1d = 3d, which means (flat TV + orthogonal lines = 3d world), that is the dimension formula.
- reyan 14y agoInterestingly this is more or less like the way I was taught linear algebra in high school.
- soapdog 14y agoFor the mathematicians here. I am taking Linear Algebra classes as a CS undergrad. Can someone recommend a very good book? I am looking for the kind of book that will make you fell in love with Linear Algebra. For Calculus, I used Piskunovs Differential and Integral Calculus which was miles away from what my classmates were using. That along with Maple help to double check that my stuff was correct proved a good combo. My current Linear Algebra book is an honest book but it is a boring book, it fails to entertain or to amaze or to give you those moments of insight that puts a grin in your face. I think I just wish I had better books as an undergrad :-(
- ktf 14y agoObligatory plug for the Manga Guide to Linear Algebra: http://nostarch.com/linearalgebra http://nostarch.com/linearalgebra :D