8 ms·

An Intuitive Guide to Linear Algebra

371 points by nkurz 12y ago

aswanson 12y ago
I wish someone would better explain Lie algebra.
- throwaway5752 12y ago
  I'll would request the same for p-adic numbers, while they're at it!
  - avalonalex 12y ago
    Obligatory Schemes and Syzygies
  - sukilot 12y ago
    p-adic numbers are measure of necesary- resolution to describe. A pair of p-adics is very close when you can use a very coarse ruler (of tick size p^n, large n) to measure the exact difference (1 vs 10000000001). p-adic numbers are far away when you need a very fine grid (1.0 vs 10.000000000001) to measure the exact difference.
  - sillysaurus3 12y ago
    I'd request the same for whatever mathematical concepts Quants use for stock market trading. Or at least a list of concepts useful for such an endeavor.
  - Chinjut 12y ago
    p-adic numbers are actually quite simple; so simple, I've actually seen children invent it on their own!! They're just like ordinary decimal notation numbers, except instead of allowing these to extend infinitely to the right (as in 3.14159...), you allow them to extend infinitely to the left. So, in addition to familiar numbers like 45 or 67.8, you also have numbers like ...95141.3 or ...999 or what have you. You add and multiply these according to the same rules you're already familiar with. So, for example, if you add 1 to ...999, the last digit of the output is 0, and you get a carry. Making the second to last digit 0, with another carry. And so on and so on, making the result ...0000 over all. Thus, ...999 acts like -1. (If we were working in base two, this last example would be just like the "two's complement" you are perhaps familiar with from computer arithmetic!) In fact, most discussion of p-adic numbers isn't done in base ten. Instead, people typically focus on p-adics in a prime base (hence the p). Why? Because in a prime base, you will find that every nonzero p-adic number has a multiplicative inverse, which is very convenient (while in a composite base, you will find that sometimes nonzero numbers multiply to zero). But there's nothing actually stopping you from making use of the notion for non-prime base, should you be interested in doing so; the notion is still perfectly coherent. (That having been said, another reason mathematicians focus on p-adics in prime bases is that the Chinese Remainder Theorem essentially allows one to reduce the study of all other bases to this case.) There's a lot of beautiful further theory to explore here, but again, the basic idea is hopefully quite simple. Let me know if you have any questions and I'll be happy to try explaining further or more clearly.
    - zwegner 12y ago
      Interesting. Very similar to the way ints are represented in Python (since they have arbitrary length, negative numbers are represented with infinitely many 1's in the most significant bits).
  - cottonseed 12y ago
    You might enjoy the r/elimath subreddit: https://www.reddit.com/r/elimath/ https://www.reddit.com/r/elimath/ Someone already asked about the p-adics: https://www.reddit.com/r/elimath/comments/2rhavm/explain_padic_numbers/ https://www.reddit.com/r/elimath/comments/2rhavm/explain_pad...
- avalonalex 12y ago
  http://www.amazon.com/Naive-Theory-Undergraduate-Texts-Mathematics/dp/0387782141/ref=sr_1_13 http://www.amazon.com/Naive-Theory-Undergraduate-Texts-Mathe... is quite readable
- sukilot 12y ago
  Lie algebra is a group whose underlying set is smooth. For example, angles (rotations) on a circle (or units in the complex numbers) form a Lie algebra. You can add and subtract angles on a circle, and if you perturb one angle you perturb the result slightly: (theta1 + delta) + theta2 = theta1 + theta2 + delta. That's a trivial example, of course. Contrast with a symmetry group of a hexagon. There's no (non-degenerate) notion of a continuous transition from identity to rotate-by-pi.
  - aswanson 12y ago
    Ah! So its just a group that has a defined group of continuous transformations that can be applied to it?
    - dougabug 12y ago
      Actually, a Lie Group is a group which is simultaneously a manifold, such that the group multiplication corresponds to diffeomorphisms of the manifold onto itself. The Lie Algebra corresponding to that Lie Group is the tangent space of the identity. The elements of the Lie Algebra generate the given Lie Group via the exponential map.
      - cottonseed 12y ago
        What do you mean by "generate" here? The exponential map is not necessarily surjective.
    - alphaBetaGamma 12y ago
      Not really. Very roughly: It's a group with a an infinite number of elements which is smooth, in the sense that there is a notion that elements can be close or far from each other, and locally around each element the group looks like R^n (for some n). Of course the group operation must play nice with this notion of "close". I.e. if elements A and B are close, after multiplying both of them by X, AX and BX can't be too far: the group operation does not "tear" the structure apart. So in the example above of rotations on the plane, you can visualize the group as a circle where a point represents the rotation by that angle. Locally it looks like R, but it has a different global structure.
agumonkey 12y ago
I always loved the fact that matrices encode operation on matrices (see comment#1 http://betterexplained.com/articles/linear-algebra-guide/#comment-124145 http://betterexplained.com/articles/linear-algebra-guide/#co... ). Very meta-circular, Ycombinatorish. Would that mean Linear Algebra is homoiconic ?
- reikonomusha 12y ago
  Having group or ring structure isn't too surprising and isn't too intimately related to the idea of metacircularity or the Y-combinator. It's actually more interesting to really see that matrices encode linear transformations, and multiplication is function composition.
  - agumonkey 12y ago
    Yes, I can't recall where I first read this (someone's comment on HN most probably) but it was very enlightening.
niels_olson 12y ago
Perhaps I don't recall my linear algebra well (we called it engine math), but there's a hell of a lot more to it than just passing inputs through a series of operations. Linear algebra is more about the methods used to design very elegant models and transforms and understanding the limits of those models and transforms.
- Ar-Curunir 12y ago
  No, those are the applications of linear algebra. Linear algebra itself is just a study of vector spaces and linear operators and transforms and such.
  - niels_olson 12y ago
    For context, I don't think I ever knew the thing we were studying was called linear algebra, so it took me another 17 years to realize that is the common name for what we studied in that class. Now I'm wondering if maybe one of the intents of the small engineering-focused school's tightly integrated math-science-engineering curriculum was to avoid an introductory linear algebra class. We had to understand matrices and vector spaces earlier for other classes and didn't do what you call the "applications" of linear algebra until sophomore year.
enahs-sf 12y ago
It seems like college Linear Algebra expeiences fall into one of two buckets: either you took a "Linear Algebra / Differential Equations" class where you learned no theory and saw a bunch of practical examples and did problem sets that were routinely north of 10 pages, OR you took a highly theoretical course where everything was far too abstract for your first year of college level of comprehension. I took the first route and retained none of the information, despite doing well in the course. Then I took computer graphics and cried all the way home. This link seems like a happy middle ground.
- j2kun 12y ago
  It's strange how many of my CS friends were deathly afraid of linear algebra, yet still enjoying their graphics classes.
  - otoburb 12y ago
    What are your thoughts on those revisiting linear algebra to instead try their hand at learning geometrical algebra instead?
    - dfan 12y ago
      You can't really get around linear algebra. Geometric algebra is really neat, and you can benefit from taking a geometric algebra approach to linear algebra (Macdonald's book is pretty good for that), but it enhances linear algebra, it's not a substitute for it.
  - jlarocco 12y ago
    I think I'm in that camp. I got a D in linear algebra, my lowest grade in university and one of my lowest grades ever, but excelled in computer graphics, and even got permission to take a graduate level course in it for elective credit as an undergrad. My recollection is that the problem sets in the linear algebra class seemed really abstract and vague, and I couldn't reason through it very well because I couldn't visualize what we learned, given the information from the class. Computer graphics, however, let me visualize what we were covered, and suddenly everything made sense. I could see, literally, what the end result was supposed to be, and it just clicked. Now that I think about it, it's kind of a shame I had the linear algebra class first. The CG experience would have helped me out quite a bit.
dj-wonk 12y ago
I find it non-intuitive that many of the input-output diagrams flow from right to left. (We all know that causality only flows left-to-right.)
- SAI_Peregrinus 12y ago
  That's because matrix multiplication is noncommutative, and it's traditional in linear algebra to write the inputs matrix on the right of the operations matrix. Just like in normal algebra, you write 5x+7y instead of x5+y7. So the picture matches up with the notation.
IndianAstronaut 12y ago
A lot of programming puzzles companies hand out can be solved with linear algebra. One company I interviewed at gave me a pretty complex problem, solved it with linalg. Didn't get the jib since I didn't use the usual data structures, but it was still a fun experience.
- ChristianGeek 12y ago
  Your use of linear algebra over what was expected would have significantly increased your chance of getting an offer if I'd been interviewing.
skierscott 12y ago
Linear algebra is cool, no doubt. This blog post is certainly well written and explains how to think of matrices as functions. I have written a blog post[1] that describes matrices as functions. This blog post is titled "Correcting common mathematical misconceptions" and walks through N dimensions, linear functions and linear algebra. After explaining what linear functions are, I mentioned nonlinear functions and how they often don't have a closed form solution and why mathematicians find bounds. [1]:http://scottsievert.github.io/blog/2014/07/31/common-mathematical-misconceptions/ http://scottsievert.github.io/blog/2014/07/31/common-mathema...
quickquicker 12y ago
I've always been partial to this: http://blog.wolfire.com/2009/07/linear-algebra-for-game-developers-part-1/ http://blog.wolfire.com/2009/07/linear-algebra-for-game-deve...
- iamcreasy 12y ago
  They stopped posting these kinds of stuff. Sad.
interdrift 12y ago
I took linear algebra 2 semesters ago.It was awesome! Highly recommend learning it, I use it on a daily basis in my work in 2D and 3D!
allending 12y ago
If you like this, then go read the No bullshit guide to math & physics after this: http://minireference.com http://minireference.com. Love the trend of indie publishing that has been exploding in the past 2 to 3 years. The biggest issue is that the marketing & design is often mediocre. Content wise, completely awesome though. I lost interest in books from larger publishers years ago.
- wnevets 12y ago
  Is this book worth getting for someone who doesnt remember anything about math from high school and beyond?
  - allending 12y ago
    I think so – it covers the fundamentals. Numbers, Algebra, Functions. Not exactly in depth, but if you've done high school math you should quickly remember and be able to get to the fun stuff.
wodenokoto 12y ago
Has anyone bought their calculus course? is it worth it?
eli_gottlieb 12y ago
>We’re getting organized: inputs in vertical columns, operations in horizontal rows. OH! Bloody hell, so that's why we write it like that! >The eigenvector and eigenvalue represent the “axes” of the transformation. >Consider spinning a globe: every location faces a new direction, except the poles. >An “eigenvector” is an input that doesn’t change direction when it’s run through the matrix (it points “along the axis”). And although the direction doesn’t change, the size might. The eigenvalue is the amount the eigenvector is scaled up or down when going through the matrix. Very nice! I'd had an algebraic (ahaha) understanding of eigenvectors/values before, but hadn't a geometric intuition. Thank you. Now I can neatly imagine why the eigenvector is orthogonal/perpendicular to the "direction" of the transformation. Determinants, too. Excellent article.

371 points by nkurz 12y ago

aswanson 12y ago

I wish someone would better explain Lie algebra.

throwaway5752 12y ago
I'll would request the same for p-adic numbers, while they're at it!
- avalonalex 12y ago
  Obligatory Schemes and Syzygies
- sukilot 12y ago
  p-adic numbers are measure of necesary- resolution to describe. A pair of p-adics is very close when you can use a very coarse ruler (of tick size p^n, large n) to measure the exact difference (1 vs 10000000001). p-adic numbers are far away when you need a very fine grid (1.0 vs 10.000000000001) to measure the exact difference.
- sillysaurus3 12y ago
  I'd request the same for whatever mathematical concepts Quants use for stock market trading. Or at least a list of concepts useful for such an endeavor.
- Chinjut 12y ago
  p-adic numbers are actually quite simple; so simple, I've actually seen children invent it on their own!! They're just like ordinary decimal notation numbers, except instead of allowing these to extend infinitely to the right (as in 3.14159...), you allow them to extend infinitely to the left. So, in addition to familiar numbers like 45 or 67.8, you also have numbers like ...95141.3 or ...999 or what have you. You add and multiply these according to the same rules you're already familiar with. So, for example, if you add 1 to ...999, the last digit of the output is 0, and you get a carry. Making the second to last digit 0, with another carry. And so on and so on, making the result ...0000 over all. Thus, ...999 acts like -1. (If we were working in base two, this last example would be just like the "two's complement" you are perhaps familiar with from computer arithmetic!) In fact, most discussion of p-adic numbers isn't done in base ten. Instead, people typically focus on p-adics in a prime base (hence the p). Why? Because in a prime base, you will find that every nonzero p-adic number has a multiplicative inverse, which is very convenient (while in a composite base, you will find that sometimes nonzero numbers multiply to zero). But there's nothing actually stopping you from making use of the notion for non-prime base, should you be interested in doing so; the notion is still perfectly coherent. (That having been said, another reason mathematicians focus on p-adics in prime bases is that the Chinese Remainder Theorem essentially allows one to reduce the study of all other bases to this case.) There's a lot of beautiful further theory to explore here, but again, the basic idea is hopefully quite simple. Let me know if you have any questions and I'll be happy to try explaining further or more clearly.
  - zwegner 12y ago
    Interesting. Very similar to the way ints are represented in Python (since they have arbitrary length, negative numbers are represented with infinitely many 1's in the most significant bits).
- cottonseed 12y ago
  You might enjoy the r/elimath subreddit: https://www.reddit.com/r/elimath/ https://www.reddit.com/r/elimath/ Someone already asked about the p-adics: https://www.reddit.com/r/elimath/comments/2rhavm/explain_padic_numbers/ https://www.reddit.com/r/elimath/comments/2rhavm/explain_pad...
avalonalex 12y ago
http://www.amazon.com/Naive-Theory-Undergraduate-Texts-Mathematics/dp/0387782141/ref=sr_1_13 http://www.amazon.com/Naive-Theory-Undergraduate-Texts-Mathe... is quite readable
sukilot 12y ago
Lie algebra is a group whose underlying set is smooth. For example, angles (rotations) on a circle (or units in the complex numbers) form a Lie algebra. You can add and subtract angles on a circle, and if you perturb one angle you perturb the result slightly: (theta1 + delta) + theta2 = theta1 + theta2 + delta. That's a trivial example, of course. Contrast with a symmetry group of a hexagon. There's no (non-degenerate) notion of a continuous transition from identity to rotate-by-pi.
- aswanson 12y ago
  Ah! So its just a group that has a defined group of continuous transformations that can be applied to it?
  - dougabug 12y ago
    Actually, a Lie Group is a group which is simultaneously a manifold, such that the group multiplication corresponds to diffeomorphisms of the manifold onto itself. The Lie Algebra corresponding to that Lie Group is the tangent space of the identity. The elements of the Lie Algebra generate the given Lie Group via the exponential map.
    - cottonseed 12y ago
      What do you mean by "generate" here? The exponential map is not necessarily surjective.
  - alphaBetaGamma 12y ago
    Not really. Very roughly: It's a group with a an infinite number of elements which is smooth, in the sense that there is a notion that elements can be close or far from each other, and locally around each element the group looks like R^n (for some n). Of course the group operation must play nice with this notion of "close". I.e. if elements A and B are close, after multiplying both of them by X, AX and BX can't be too far: the group operation does not "tear" the structure apart. So in the example above of rotations on the plane, you can visualize the group as a circle where a point represents the rotation by that angle. Locally it looks like R, but it has a different global structure.

agumonkey 12y ago

I always loved the fact that matrices encode operation on matrices (see comment#1 http://betterexplained.com/articles/linear-algebra-guide/#comment-124145 http://betterexplained.com/articles/linear-algebra-guide/#co... ). Very meta-circular, Ycombinatorish. Would that mean Linear Algebra is homoiconic ?

reikonomusha 12y ago
Having group or ring structure isn't too surprising and isn't too intimately related to the idea of metacircularity or the Y-combinator. It's actually more interesting to really see that matrices encode linear transformations, and multiplication is function composition.
- agumonkey 12y ago
  Yes, I can't recall where I first read this (someone's comment on HN most probably) but it was very enlightening.

niels_olson 12y ago

Perhaps I don't recall my linear algebra well (we called it engine math), but there's a hell of a lot more to it than just passing inputs through a series of operations. Linear algebra is more about the methods used to design very elegant models and transforms and understanding the limits of those models and transforms.

Ar-Curunir 12y ago
No, those are the applications of linear algebra. Linear algebra itself is just a study of vector spaces and linear operators and transforms and such.
- niels_olson 12y ago
  For context, I don't think I ever knew the thing we were studying was called linear algebra, so it took me another 17 years to realize that is the common name for what we studied in that class. Now I'm wondering if maybe one of the intents of the small engineering-focused school's tightly integrated math-science-engineering curriculum was to avoid an introductory linear algebra class. We had to understand matrices and vector spaces earlier for other classes and didn't do what you call the "applications" of linear algebra until sophomore year.

enahs-sf 12y ago

It seems like college Linear Algebra expeiences fall into one of two buckets: either you took a "Linear Algebra / Differential Equations" class where you learned no theory and saw a bunch of practical examples and did problem sets that were routinely north of 10 pages, OR you took a highly theoretical course where everything was far too abstract for your first year of college level of comprehension. I took the first route and retained none of the information, despite doing well in the course. Then I took computer graphics and cried all the way home. This link seems like a happy middle ground.

j2kun 12y ago
It's strange how many of my CS friends were deathly afraid of linear algebra, yet still enjoying their graphics classes.
- otoburb 12y ago
  What are your thoughts on those revisiting linear algebra to instead try their hand at learning geometrical algebra instead?
  - dfan 12y ago
    You can't really get around linear algebra. Geometric algebra is really neat, and you can benefit from taking a geometric algebra approach to linear algebra (Macdonald's book is pretty good for that), but it enhances linear algebra, it's not a substitute for it.
- jlarocco 12y ago
  I think I'm in that camp. I got a D in linear algebra, my lowest grade in university and one of my lowest grades ever, but excelled in computer graphics, and even got permission to take a graduate level course in it for elective credit as an undergrad. My recollection is that the problem sets in the linear algebra class seemed really abstract and vague, and I couldn't reason through it very well because I couldn't visualize what we learned, given the information from the class. Computer graphics, however, let me visualize what we were covered, and suddenly everything made sense. I could see, literally, what the end result was supposed to be, and it just clicked. Now that I think about it, it's kind of a shame I had the linear algebra class first. The CG experience would have helped me out quite a bit.

dj-wonk 12y ago

I find it non-intuitive that many of the input-output diagrams flow from right to left. (We all know that causality only flows left-to-right.)

SAI_Peregrinus 12y ago
That's because matrix multiplication is noncommutative, and it's traditional in linear algebra to write the inputs matrix on the right of the operations matrix. Just like in normal algebra, you write 5x+7y instead of x5+y7. So the picture matches up with the notation.

IndianAstronaut 12y ago

A lot of programming puzzles companies hand out can be solved with linear algebra. One company I interviewed at gave me a pretty complex problem, solved it with linalg. Didn't get the jib since I didn't use the usual data structures, but it was still a fun experience.

ChristianGeek 12y ago
Your use of linear algebra over what was expected would have significantly increased your chance of getting an offer if I'd been interviewing.

skierscott 12y ago

Linear algebra is cool, no doubt. This blog post is certainly well written and explains how to think of matrices as functions. I have written a blog post[1] that describes matrices as functions. This blog post is titled "Correcting common mathematical misconceptions" and walks through N dimensions, linear functions and linear algebra. After explaining what linear functions are, I mentioned nonlinear functions and how they often don't have a closed form solution and why mathematicians find bounds. [1]:http://scottsievert.github.io/blog/2014/07/31/common-mathematical-misconceptions/ http://scottsievert.github.io/blog/2014/07/31/common-mathema...

quickquicker 12y ago

I've always been partial to this: http://blog.wolfire.com/2009/07/linear-algebra-for-game-developers-part-1/ http://blog.wolfire.com/2009/07/linear-algebra-for-game-deve...

iamcreasy 12y ago
They stopped posting these kinds of stuff. Sad.

interdrift 12y ago

I took linear algebra 2 semesters ago.It was awesome! Highly recommend learning it, I use it on a daily basis in my work in 2D and 3D!

allending 12y ago

If you like this, then go read the No bullshit guide to math & physics after this: http://minireference.com http://minireference.com. Love the trend of indie publishing that has been exploding in the past 2 to 3 years. The biggest issue is that the marketing & design is often mediocre. Content wise, completely awesome though. I lost interest in books from larger publishers years ago.

wnevets 12y ago
Is this book worth getting for someone who doesnt remember anything about math from high school and beyond?
- allending 12y ago
  I think so – it covers the fundamentals. Numbers, Algebra, Functions. Not exactly in depth, but if you've done high school math you should quickly remember and be able to get to the fun stuff.

wodenokoto 12y ago

Has anyone bought their calculus course? is it worth it?

eli_gottlieb 12y ago

>We’re getting organized: inputs in vertical columns, operations in horizontal rows. OH! Bloody hell, so that's why we write it like that! >The eigenvector and eigenvalue represent the “axes” of the transformation. >Consider spinning a globe: every location faces a new direction, except the poles. >An “eigenvector” is an input that doesn’t change direction when it’s run through the matrix (it points “along the axis”). And although the direction doesn’t change, the size might. The eigenvalue is the amount the eigenvector is scaled up or down when going through the matrix. Very nice! I'd had an algebraic (ahaha) understanding of eigenvectors/values before, but hadn't a geometric intuition. Thank you. Now I can neatly imagine why the eigenvector is orthogonal/perpendicular to the "direction" of the transformation. Determinants, too. Excellent article.