9 ms·

Linear algebra for programmers

237 points by coffeemug 3y ago

mvkel 3y ago
This reads like someone who thinks they absolutely nailed matrices and is using that as a proxy for linear algebra expertise. I wouldn't know for sure because it all whooshed about 60 feet over my head
marginalia_nu 3y ago
I strongly suggest anyone getting into Linear Algebra to have a project to work on. It makes everything so much easier when you get to play with the stuff. My hint for something to play with is that basic linear algebra applies very directly to graphics, rotation matrices and so on. If you know how to multiply a matrix with a vector, you basically know what you need to render basic line-art 3D graphics. May want to look into dot and cross products as well as vector projection, but it's fairly basic all of this.
- seanmcdirmid 3y ago
  I did this awhile ago, but found myself simply copying and tweaking graphics algorithms, rather than gaining any intrinsic understanding of what linear algebra is really doing. I guess I just didn’t have enough computer graphics background? Yes, I can use lighting equations to define a pixel shader, but I’m basically copying and translating the algorithm from a book.
  - marginalia_nu 3y ago
    No I mean do full software rendering, no shaders, no graphics card.
    - SotCodeLaureate 3y ago
      > No I mean do full software rendering, no shaders, no graphics card. But why? There doesn't seems to be a lot to learn about applied linear algebra (in the sense discussed here) by implementing a rasterizer. But there is plenty of LA above and below that (in the scene management, in the shaders).
      - marginalia_nu 3y ago
        Eh, it's basically a project consisting entirely of the parts of LA that the article is talking about.
  - delusional 3y ago
    I'm sorry if this comes across as curt, but can't you just decide to not do that? Decide to not look at the examples in the book, but write it from the explanation instead. You could even forego the book and just sit down with a piece of paper and do the work from first principles. You might not come up with the most efficient algorithm, but you'll have a foothold into understanding the one you can then go and look up.
    - seanmcdirmid 3y ago
      Sure that would be ideal. But I would need some 101 knowledge that I couldn’t really find in the books I had, or maybe I wasn’t patient enough in reading them, like somewhere there was a key realization of linear algebra that would have made doing everything from first principles possible, or perhaps I could just read Phong’s original paper.
- paulsmith 3y ago
  Deep learning! It's all "just" (more or less) high school calculus (partial derivatives, chain rule) and matrix multiplication.
  - thewataccount 3y ago
    I feel like I saw one once but lost it - Is there a githubrepo/tutorial for how linear algebra is used for a very small model just to demonstrate how that allows it to "learn"? I've got the calc, I just don't understand what the matrix multiplication "does"
    - tstrimple 3y ago
      I cannot recommend Andrew Ng's courses on Machine Learning enough. Something like this seems like it would cover everything you're looking for. https://www.coursera.org/learn/machine-learning https://www.coursera.org/learn/machine-learning I cannot speak to the author of the content of this github repo, but it appears they have completed the course and included all of the solutions here. It might let you jump right to what you're looking for. https://github.com/greyhatguy007/Machine-Learning-Specialization-Coursera https://github.com/greyhatguy007/Machine-Learning-Specializa...
      - saberience 3y ago
        What math pre-reqs does it need for someone who never made it to college level maths?
        tstrimple 3y ago
        Based on my experience as long as you have a good foundation in the basics of algebra you’ll be able to pick up the rest from the course.
    - paulsmith 3y ago
      Watch Karpathy's recent lectures. They're gold. Start here[1] with micrograd[2]. It doesn't use linear algebra/matrices to start, but the principles are the same. The matrix multiplication is how the weights of the connections between neurons and the input values are combined (to form an activation value that then may lead to that neuron "firing" or not, depending on whether it passes some threshold function). We use matrices to model the connections between neurons - each row is a connection, and each column is a weight corresponding to an input. [1] https://www.youtube.com/watch?v=VMj-3S1tku0 https://www.youtube.com/watch?v=VMj-3S1tku0 [2] https://github.com/karpathy/micrograd https://github.com/karpathy/micrograd
- deleted 3y ago
  [deleted]
nh23423fefe 3y ago
weird to talk about linear algebra and never invoke linearity. also, why say these 2 things? >If you forget how matrix-vector multiplication works, just remember that its definition flows out of the notation. >Another way to think of matrix-vector multiplication is by treating each row of a matrix as its own vector, and computing the dot products of these row vectors with the vector we’re multiplying by. How on earth does that work?! What does vector similarity have to do with linear equations, or with matrix-vector multiplication? But you just told me to plug into the definition? Which is a dot product. Pretty incoherent.
- ak_111 3y ago
  IMO it is much clearer to justify something like matrix multiplication via a simple real life example like markov chain computations, it might add a bit of complexity in understanding the application but motivates the definition much better.
- stOneskull 3y ago
  i think it's a rushed draft it hasn't gone through a spell-checker, with words in there like hight and strage
- adrian_b 3y ago
  True. Moreover, here as in most other writings about linear algebra that I have seen, there is the very bad habit of describing the more complex operations as being composed from dot products. On modern CPUs, dot products must be avoided, because like all reduction operations they consist of one chain of dependent operations, so their speed is limited by the latency of the fused multiply-add operations, instead of being limited by the much higher throughput of the FMA operations. When vectors are multiplied with vectors, there is no alternative for dot products, so the only way to accelerate them is to reorder the operations into a tree, to be able to overlap a part of them, which are independent. When arrays with more dimensions are multiplied, e.g. matrices with vectors or matrices with matrices, the multiplications correspond with nested for loops, i.e. 2 nested loops for matrix-vector multiplication and 3 nested loops for matrix-matrix multiplication. The nested loops can be reordered arbitrarily. In each case, one of the possible loop orders has in the innermost loop the computation of a dot product. This is the only order mentioned in the parent article and in most other linear algebra manuals. However, this order is exactly the worst possible computationally. For 2 or more nested loops, there is always another order where the innermost operation is a so-called AXPY operation (the BLAS function name). AXPY means the scalar A multiplied by the vector X Plus the vector Y, with the result stored in Y. AXPY operations are always better than dot products, because the FMA operations are independent, so they can be pipelined, but for 3 or more nested loops there are better orders, where the innermost loop needs much less load and store operations than for AXPY. For 3 or more nested loops, there is always an order where the innermost operation is a tensor product of 2 vectors. This is a more attractive operation than both AXPY and dot products. If the tensor product of 2 vectors with N elements is stored in registers, then its computation needs N+N loads from memory, but N*N FMA operations, so if there are enough registers so that N>2, there will be much more FMA than loads, allowing the full utilization of the execution units of a modern CPU. In conclusion, matrix-vector products must not be described as being composed of N dot products that are executed separately for each element of the result vector, but as being the sum of N AXPY operations, which are accumulated into the result vector. Similarly, a matrix-matrix product must not be described as being composed of N^2 dot products, one for each element of the result matrix, but as being the sum of N vector-vector tensor products, which are accumulated into the result matrix. Such descriptions would be much more useful in practice, where the definitions based on dot products are just a hindrance.
thewataccount 3y ago
Can anyone suggest a something that teaches Linear Algebra with a practical applications, especially for software engineering? I can sorta kinda get the theory, but every demonstration involves moving an arrow around which is.... not something I need to do frequently. So I'm not sure how I actually apply linear algebra to solve actual problems. I'm a software developer and I know it's useful I just don't get where to use it - and I'm struggling to actually understand the different operations, purpose of the dot product, etc. I have a decent base for basic stats and calc, both of which I can "conceptually apply" near daily for understanding how things work. 3Blue1Brown is helpful, but I just kinda go "yeah I guess that looks right" without knowing what to do with it. EDIT: Thank you!
- cratermoon 3y ago
  Much of the use of linear algebra in programming is for machine learning. To a first approximation, ML is statistics on huge datasets, and linear algebra makes it possible because matrix operations are massively, if not embarrassingly, parallel.
- latenightcoding 3y ago
  Fast ai had a computational linear algebra class iirc
- ak_111 3y ago
  Look into making a Doom-like game, tons of linear algebra there.
- tptacek 3y ago
  One place to start might be a tutorial on principal component analysis, which will take you through some of the intuitions for applying SVD. You can also go in the direction of cryptography; here's, for instance, a really excellent LLL tutorial that builds on Graham-Schmidt: https://kel.bz/post/lll/ https://kel.bz/post/lll/
  - anon____ 3y ago
    This is the PCA tutorial that worked for me: https://arxiv.org/abs/1404.1100 https://arxiv.org/abs/1404.1100.
Corsome 3y ago
> This is actually not so strange– you can think of many structures as functions. For example, you can think of a number 3 as a function. When you multiply it by things, it makes them three times bigger. I don't see how 3 can be a function from this example. "3*" (partially applied multiplication by 3) looks more like it. Matrices and vectors as functions? Yeah, if the argument is within bounds. That makes it just an indexing operation. (I guess one can view 3 as a one element vector but that sounds like a degenerate case) Or maybe I'm missing something...?
- denial 3y ago
  I take it as analogous to the association of a matrix to a linear transformation. This association is via multiplication.
- robot_no_421 3y ago
  3 is the following function: 3 == lambda x: 3*x But I think that the technical, mathematical way to think about it is: The monoid of linear functions L:R->R is isomorphic to the monoid (R, *) Meaning, the structure of 1x1 matrices under multiplication is exactly the same as the structure of real numbers under multiplication.
  - contravariant 3y ago
    Importantly matrix multiplication is the same as function composition of the linear functions, hence the analogy to functions that multiply by a factor. Seems trivial but among other things it implies associativity, which is not quite trivial for larger matrices.
- ajtulloch 3y ago
  I think the mathematical concept that you are looking for is that of the dual space. Essentially if you have a vector space V, you can construct a dual space V* where the elements of the dual space are functions taking elements of V to the underlying field F, and under certain conditions these spaces are isomorphic (the same) - so there is a 1:1 correspondence between elements of the vector space and the functions in the dual space.
robot_no_421 3y ago
"In hight school your math teacher may have started a treatment of linear algebra by making you solve a system of linear equations, at which point you very sensibly zoned out because you knew you’d go on to program computers and never have to solve a system of linear equations again (don’t worry, I won’t be talking much about them here)." No offense but I stopped reading there. Too many software developers have this weird superiority complex when it comes to math. When they struggle with math, I've seen devs criticize everything from naming conventions to curricula to it being "useless". A lot of them seem unwilling to acknowledge that math is sometimes... simply hard. If your attitude to math is "I don't need any of this useless stuff, so I'll zone out", then I kindly suggest you first at least try to learn Linear algebra for mathematicians first.
- deleted 3y ago
  [deleted]
- bbkane 3y ago
  Many devs who "zone out" because they won't "need math" still manage to do just fine? Software is a big field and there's room for folks with different interests and talents.
- notsurenymore 3y ago
  That’s how I was in high school, and immediately regretted it the minute I found interest in a domain where strong math ability was required.
- kuhewa 3y ago
  I don't understand why you take issue with that statement, it sounds descriptive as an attitude that may be common in the audience but perhaps you are interpreting it as the author endorsing that view? i.e. I imagine they might agree with you.
- JKCalhoun 3y ago
  Or write a flight simulator from scratch. I followed a book in the 90's to create a flight simulator from scratch. Besides learning Bresenham's line algorithm, I learned a lot of linear algebra. Probably this book: https://archive.org/details/build-your-own-flight-sim-in-c-dos-game-dev-michael-radtke-chris-lampton/ https://archive.org/details/build-your-own-flight-sim-in-c-d...
ivansavz 3y ago
If anyone wants to try things hands-on, I highly recommend the SymPy (in particular the online live shell https://live.sympy.org/ https://live.sympy.org/ ). The `Matrix` class can be used to create matrices (lists of lists of numbers) and vectors (lists of numbers). Here is a clickable link that demos the first example rotation: https://live.sympy.org/?evaluate=A%20%3D%20Matrix(%5B%5B0%2C%20%201%5D%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%5B-1%2C%200%5D%5D)%0A%23--%0AA%20*%20Matrix(%5B1%2C0%5D)%0A https://live.sympy.org/?evaluate=A%20%3D%20Matrix(%5B%5B0%2C... For more info about SymPy, see section "VI. Linear algebra" in the SymPy tutorial I wrote https://minireference.com/static/tutorials/sympy_tutorial.pdf https://minireference.com/static/tutorials/sympy_tutorial.pd... (also available as notebook https://github.com/minireference/sympytut_notebooks/blob/master/notebooks/Linear-algebra.ipynb https://github.com/minireference/sympytut_notebooks/blob/mas... )
- ivansavz 3y ago
  Oh and for even more linear algebra stuff, here is a 30 min condensed video tutorial that introduced most of topics in a standard LA course, also using SymPy to show demonstrations: https://www.youtube.com/watch?v=2G3PmEZI6n8&list=PLGmu4KtWiH6-F2JlLVvCTsMR91I4ix0_z&index=1 https://www.youtube.com/watch?v=2G3PmEZI6n8&list=PLGmu4KtWiH...
  - 3abiton 3y ago
    That brings back some memories
nyrikki 3y ago
If you haven't seen it yet, even if you are proficient in LA, 3Blue1Brown's playlist on YouTube is good to watch. It really helps build intuition beyond the typical teaching methods. It will really help connect the dots. https://youtube.com/playlist?list=PL0-GT3co4r2y2YErbmuJw2L5tW4Ew2O5B&si=PXkIbFMqqozrsYEu https://youtube.com/playlist?list=PL0-GT3co4r2y2YErbmuJw2L5t...
127 3y ago
For someone who actually needs to understand linear algebra to create novel complex programs, this is completely insufficient and possibly needlessly distracting. I suggest this one instead: https://www.youtube.com/playlist?list=PLE7DDD91010BC51F8 https://www.youtube.com/playlist?list=PLE7DDD91010BC51F8 (Gilbert Strang) Aggressively taking the least shortcuts possible is the fastest shortcut.
- seanhunter 3y ago
  Agree. I love the Gilbert Strang series and am working through it myself. Also, the book "Linear Algebra Done Right" by Sheldon Axler[1] is amazing. He has on his website a set of short videos and slides to accompany it[2]. It's incredibly comprehensive and takes an interesting strategy of teaching all of linear algebra in a very rigorous fashion without introducing determinants until the end. His reasoning is explained in a paper he published called "Down with Determinants!"[3]. [1] https://linear.axler.net/ https://linear.axler.net/ [2] https://linear.axler.net/LADRvideos.html https://linear.axler.net/LADRvideos.html [3] https://www.axler.net/DwD.html https://www.axler.net/DwD.html
  - tptacek 3y ago
    Axler and Strang seem like pretty different approaches to the same subject, and if your goal is just to do stuff with linear algebra, rather than deepening your understanding of it or really getting how it generalizes to functions and stuff, Strang's approach is likely to be more immediately profitable. I like 'em both. I helped my daughter through UIUC linear algebra last semester, and Strang definitely prepped me for all the computation, all the way through the end of the class, but I needed to go back to Axler for all the proof stuff.
    - seanhunter 3y ago
      I'm really enjoying them both for that precise reason.
- penguin_booze 3y ago
  I never found Strang's lectures out of ordinary nor particularly enlightening. But what managed to give me perspective was Pavel Grinfeld's playlists, starting from https://www.youtube.com/watch?v=Fnfh8jNqBlg&list=PLlXfTHzgMRUKXD88IdzS14F4NxAZudSmv https://www.youtube.com/watch?v=Fnfh8jNqBlg&list=PLlXfTHzgMR....
vouaobrasil 3y ago
As a mathematician, if there's one piece of advice I'd give people who need 1-2 math courses, it would be LEARN IT RIGHT. Pick up a rigorous textbook and read through it. Don't be lazy. I've spent decades reading many texts and papers. If you're smart, reading one real and thorough book will not take a lot of time. And, truly understandng the subject will pay you back in spades. Don't go for fluff. Linear algebra isn't some Facebook post. Truly go through the book line by line and really understand it down to the nuts and bolts. Suffer a little.
- rramadass 3y ago
  This is the advice for the ages! However i fear it is lost on the current generation which is always looking for that quick/short article/blog/video/etc. which will teach and make them understand the subject as painlessly as possible. Everything must be "Fun and Enjoyable" (i don't even know what that means anymore!). They have forgotten the maxim "There is no Royal Road to Mathematics"(or any other scientific field). The amount of people (even on HN!) who espouse disdain for Textbooks is astonishing. By definition, the process of learning involves diving into the unknown and hence there will be discomfort/difficulties/effort/time needed and thus is not going to be easy. But the student doesn't want to put in any effort at all; everything is the fault of the Teacher(can't teach properly)/Teaching method(i am a visual learner)/Too rigorous/Mathematical/I have ADHD/ADD/Autism/Whatever. The real tragedy is that used Books are now so easily/cheaply/universally available that there is no reason not to have one's own library of books on subjects of interest; it is "food" for the Mind.
dekervin 3y ago
We have a discord "HN Learn", where we collaborate to learn things, especially Maths. If you want some company to delve into linear algebra, feel free to join. [0] https://discord.gg/RxSjEMnW https://discord.gg/RxSjEMnW

237 points by coffeemug 3y ago

This reads like someone who thinks they absolutely nailed matrices and is using that as a proxy for linear algebra expertise. I wouldn't know for sure because it all whooshed about 60 feet over my head

marginalia_nu 3y ago

I strongly suggest anyone getting into Linear Algebra to have a project to work on. It makes everything so much easier when you get to play with the stuff. My hint for something to play with is that basic linear algebra applies very directly to graphics, rotation matrices and so on. If you know how to multiply a matrix with a vector, you basically know what you need to render basic line-art 3D graphics. May want to look into dot and cross products as well as vector projection, but it's fairly basic all of this.

seanmcdirmid 3y ago
I did this awhile ago, but found myself simply copying and tweaking graphics algorithms, rather than gaining any intrinsic understanding of what linear algebra is really doing. I guess I just didn’t have enough computer graphics background? Yes, I can use lighting equations to define a pixel shader, but I’m basically copying and translating the algorithm from a book.
- marginalia_nu 3y ago
  No I mean do full software rendering, no shaders, no graphics card.
  - SotCodeLaureate 3y ago
    > No I mean do full software rendering, no shaders, no graphics card. But why? There doesn't seems to be a lot to learn about applied linear algebra (in the sense discussed here) by implementing a rasterizer. But there is plenty of LA above and below that (in the scene management, in the shaders).
    - marginalia_nu 3y ago
      Eh, it's basically a project consisting entirely of the parts of LA that the article is talking about.
- delusional 3y ago
  I'm sorry if this comes across as curt, but can't you just decide to not do that? Decide to not look at the examples in the book, but write it from the explanation instead. You could even forego the book and just sit down with a piece of paper and do the work from first principles. You might not come up with the most efficient algorithm, but you'll have a foothold into understanding the one you can then go and look up.
  - seanmcdirmid 3y ago
    Sure that would be ideal. But I would need some 101 knowledge that I couldn’t really find in the books I had, or maybe I wasn’t patient enough in reading them, like somewhere there was a key realization of linear algebra that would have made doing everything from first principles possible, or perhaps I could just read Phong’s original paper.
paulsmith 3y ago
Deep learning! It's all "just" (more or less) high school calculus (partial derivatives, chain rule) and matrix multiplication.
- thewataccount 3y ago
  I feel like I saw one once but lost it - Is there a githubrepo/tutorial for how linear algebra is used for a very small model just to demonstrate how that allows it to "learn"? I've got the calc, I just don't understand what the matrix multiplication "does"
  - tstrimple 3y ago
    I cannot recommend Andrew Ng's courses on Machine Learning enough. Something like this seems like it would cover everything you're looking for. https://www.coursera.org/learn/machine-learning https://www.coursera.org/learn/machine-learning I cannot speak to the author of the content of this github repo, but it appears they have completed the course and included all of the solutions here. It might let you jump right to what you're looking for. https://github.com/greyhatguy007/Machine-Learning-Specialization-Coursera https://github.com/greyhatguy007/Machine-Learning-Specializa...
    - saberience 3y ago
      What math pre-reqs does it need for someone who never made it to college level maths?
      - tstrimple 3y ago
        Based on my experience as long as you have a good foundation in the basics of algebra you’ll be able to pick up the rest from the course.
  - paulsmith 3y ago
    Watch Karpathy's recent lectures. They're gold. Start here[1] with micrograd[2]. It doesn't use linear algebra/matrices to start, but the principles are the same. The matrix multiplication is how the weights of the connections between neurons and the input values are combined (to form an activation value that then may lead to that neuron "firing" or not, depending on whether it passes some threshold function). We use matrices to model the connections between neurons - each row is a connection, and each column is a weight corresponding to an input. [1] https://www.youtube.com/watch?v=VMj-3S1tku0 https://www.youtube.com/watch?v=VMj-3S1tku0 [2] https://github.com/karpathy/micrograd https://github.com/karpathy/micrograd
deleted 3y ago
[deleted]

nh23423fefe 3y ago

weird to talk about linear algebra and never invoke linearity. also, why say these 2 things? >If you forget how matrix-vector multiplication works, just remember that its definition flows out of the notation. >Another way to think of matrix-vector multiplication is by treating each row of a matrix as its own vector, and computing the dot products of these row vectors with the vector we’re multiplying by. How on earth does that work?! What does vector similarity have to do with linear equations, or with matrix-vector multiplication? But you just told me to plug into the definition? Which is a dot product. Pretty incoherent.

ak_111 3y ago
IMO it is much clearer to justify something like matrix multiplication via a simple real life example like markov chain computations, it might add a bit of complexity in understanding the application but motivates the definition much better.
stOneskull 3y ago
i think it's a rushed draft it hasn't gone through a spell-checker, with words in there like hight and strage
adrian_b 3y ago
True. Moreover, here as in most other writings about linear algebra that I have seen, there is the very bad habit of describing the more complex operations as being composed from dot products. On modern CPUs, dot products must be avoided, because like all reduction operations they consist of one chain of dependent operations, so their speed is limited by the latency of the fused multiply-add operations, instead of being limited by the much higher throughput of the FMA operations. When vectors are multiplied with vectors, there is no alternative for dot products, so the only way to accelerate them is to reorder the operations into a tree, to be able to overlap a part of them, which are independent. When arrays with more dimensions are multiplied, e.g. matrices with vectors or matrices with matrices, the multiplications correspond with nested for loops, i.e. 2 nested loops for matrix-vector multiplication and 3 nested loops for matrix-matrix multiplication. The nested loops can be reordered arbitrarily. In each case, one of the possible loop orders has in the innermost loop the computation of a dot product. This is the only order mentioned in the parent article and in most other linear algebra manuals. However, this order is exactly the worst possible computationally. For 2 or more nested loops, there is always another order where the innermost operation is a so-called AXPY operation (the BLAS function name). AXPY means the scalar A multiplied by the vector X Plus the vector Y, with the result stored in Y. AXPY operations are always better than dot products, because the FMA operations are independent, so they can be pipelined, but for 3 or more nested loops there are better orders, where the innermost loop needs much less load and store operations than for AXPY. For 3 or more nested loops, there is always an order where the innermost operation is a tensor product of 2 vectors. This is a more attractive operation than both AXPY and dot products. If the tensor product of 2 vectors with N elements is stored in registers, then its computation needs N+N loads from memory, but N*N FMA operations, so if there are enough registers so that N>2, there will be much more FMA than loads, allowing the full utilization of the execution units of a modern CPU. In conclusion, matrix-vector products must not be described as being composed of N dot products that are executed separately for each element of the result vector, but as being the sum of N AXPY operations, which are accumulated into the result vector. Similarly, a matrix-matrix product must not be described as being composed of N^2 dot products, one for each element of the result matrix, but as being the sum of N vector-vector tensor products, which are accumulated into the result matrix. Such descriptions would be much more useful in practice, where the definitions based on dot products are just a hindrance.

thewataccount 3y ago

Can anyone suggest a something that teaches Linear Algebra with a practical applications, especially for software engineering? I can sorta kinda get the theory, but every demonstration involves moving an arrow around which is.... not something I need to do frequently. So I'm not sure how I actually apply linear algebra to solve actual problems. I'm a software developer and I know it's useful I just don't get where to use it - and I'm struggling to actually understand the different operations, purpose of the dot product, etc. I have a decent base for basic stats and calc, both of which I can "conceptually apply" near daily for understanding how things work. 3Blue1Brown is helpful, but I just kinda go "yeah I guess that looks right" without knowing what to do with it. EDIT: Thank you!

cratermoon 3y ago
Much of the use of linear algebra in programming is for machine learning. To a first approximation, ML is statistics on huge datasets, and linear algebra makes it possible because matrix operations are massively, if not embarrassingly, parallel.
latenightcoding 3y ago
Fast ai had a computational linear algebra class iirc
ak_111 3y ago
Look into making a Doom-like game, tons of linear algebra there.
tptacek 3y ago
One place to start might be a tutorial on principal component analysis, which will take you through some of the intuitions for applying SVD. You can also go in the direction of cryptography; here's, for instance, a really excellent LLL tutorial that builds on Graham-Schmidt: https://kel.bz/post/lll/ https://kel.bz/post/lll/
- anon____ 3y ago
  This is the PCA tutorial that worked for me: https://arxiv.org/abs/1404.1100 https://arxiv.org/abs/1404.1100.

Corsome 3y ago

> This is actually not so strange– you can think of many structures as functions. For example, you can think of a number 3 as a function. When you multiply it by things, it makes them three times bigger. I don't see how 3 can be a function from this example. "3*" (partially applied multiplication by 3) looks more like it. Matrices and vectors as functions? Yeah, if the argument is within bounds. That makes it just an indexing operation. (I guess one can view 3 as a one element vector but that sounds like a degenerate case) Or maybe I'm missing something...?

denial 3y ago
I take it as analogous to the association of a matrix to a linear transformation. This association is via multiplication.
robot_no_421 3y ago
3 is the following function: 3 == lambda x: 3*x But I think that the technical, mathematical way to think about it is: The monoid of linear functions L:R->R is isomorphic to the monoid (R, *) Meaning, the structure of 1x1 matrices under multiplication is exactly the same as the structure of real numbers under multiplication.
- contravariant 3y ago
  Importantly matrix multiplication is the same as function composition of the linear functions, hence the analogy to functions that multiply by a factor. Seems trivial but among other things it implies associativity, which is not quite trivial for larger matrices.
ajtulloch 3y ago
I think the mathematical concept that you are looking for is that of the dual space. Essentially if you have a vector space V, you can construct a dual space V* where the elements of the dual space are functions taking elements of V to the underlying field F, and under certain conditions these spaces are isomorphic (the same) - so there is a 1:1 correspondence between elements of the vector space and the functions in the dual space.

robot_no_421 3y ago

"In hight school your math teacher may have started a treatment of linear algebra by making you solve a system of linear equations, at which point you very sensibly zoned out because you knew you’d go on to program computers and never have to solve a system of linear equations again (don’t worry, I won’t be talking much about them here)." No offense but I stopped reading there. Too many software developers have this weird superiority complex when it comes to math. When they struggle with math, I've seen devs criticize everything from naming conventions to curricula to it being "useless". A lot of them seem unwilling to acknowledge that math is sometimes... simply hard. If your attitude to math is "I don't need any of this useless stuff, so I'll zone out", then I kindly suggest you first at least try to learn Linear algebra for mathematicians first.

deleted 3y ago
[deleted]
bbkane 3y ago
Many devs who "zone out" because they won't "need math" still manage to do just fine? Software is a big field and there's room for folks with different interests and talents.
notsurenymore 3y ago
That’s how I was in high school, and immediately regretted it the minute I found interest in a domain where strong math ability was required.
kuhewa 3y ago
I don't understand why you take issue with that statement, it sounds descriptive as an attitude that may be common in the audience but perhaps you are interpreting it as the author endorsing that view? i.e. I imagine they might agree with you.
JKCalhoun 3y ago
Or write a flight simulator from scratch. I followed a book in the 90's to create a flight simulator from scratch. Besides learning Bresenham's line algorithm, I learned a lot of linear algebra. Probably this book: https://archive.org/details/build-your-own-flight-sim-in-c-dos-game-dev-michael-radtke-chris-lampton/ https://archive.org/details/build-your-own-flight-sim-in-c-d...

ivansavz 3y ago

If anyone wants to try things hands-on, I highly recommend the SymPy (in particular the online live shell https://live.sympy.org/ https://live.sympy.org/ ). The `Matrix` class can be used to create matrices (lists of lists of numbers) and vectors (lists of numbers). Here is a clickable link that demos the first example rotation: https://live.sympy.org/?evaluate=A%20%3D%20Matrix(%5B%5B0%2C%20%201%5D%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%5B-1%2C%200%5D%5D)%0A%23--%0AA%20*%20Matrix(%5B1%2C0%5D)%0A https://live.sympy.org/?evaluate=A%20%3D%20Matrix(%5B%5B0%2C... For more info about SymPy, see section "VI. Linear algebra" in the SymPy tutorial I wrote https://minireference.com/static/tutorials/sympy_tutorial.pdf https://minireference.com/static/tutorials/sympy_tutorial.pd... (also available as notebook https://github.com/minireference/sympytut_notebooks/blob/master/notebooks/Linear-algebra.ipynb https://github.com/minireference/sympytut_notebooks/blob/mas... )

ivansavz 3y ago
Oh and for even more linear algebra stuff, here is a 30 min condensed video tutorial that introduced most of topics in a standard LA course, also using SymPy to show demonstrations: https://www.youtube.com/watch?v=2G3PmEZI6n8&list=PLGmu4KtWiH6-F2JlLVvCTsMR91I4ix0_z&index=1 https://www.youtube.com/watch?v=2G3PmEZI6n8&list=PLGmu4KtWiH...
- 3abiton 3y ago
  That brings back some memories

nyrikki 3y ago

If you haven't seen it yet, even if you are proficient in LA, 3Blue1Brown's playlist on YouTube is good to watch. It really helps build intuition beyond the typical teaching methods. It will really help connect the dots. https://youtube.com/playlist?list=PL0-GT3co4r2y2YErbmuJw2L5tW4Ew2O5B&si=PXkIbFMqqozrsYEu https://youtube.com/playlist?list=PL0-GT3co4r2y2YErbmuJw2L5t...

127 3y ago

For someone who actually needs to understand linear algebra to create novel complex programs, this is completely insufficient and possibly needlessly distracting. I suggest this one instead: https://www.youtube.com/playlist?list=PLE7DDD91010BC51F8 https://www.youtube.com/playlist?list=PLE7DDD91010BC51F8 (Gilbert Strang) Aggressively taking the least shortcuts possible is the fastest shortcut.

seanhunter 3y ago
Agree. I love the Gilbert Strang series and am working through it myself. Also, the book "Linear Algebra Done Right" by Sheldon Axler[1] is amazing. He has on his website a set of short videos and slides to accompany it[2]. It's incredibly comprehensive and takes an interesting strategy of teaching all of linear algebra in a very rigorous fashion without introducing determinants until the end. His reasoning is explained in a paper he published called "Down with Determinants!"[3]. [1] https://linear.axler.net/ https://linear.axler.net/ [2] https://linear.axler.net/LADRvideos.html https://linear.axler.net/LADRvideos.html [3] https://www.axler.net/DwD.html https://www.axler.net/DwD.html
- tptacek 3y ago
  Axler and Strang seem like pretty different approaches to the same subject, and if your goal is just to do stuff with linear algebra, rather than deepening your understanding of it or really getting how it generalizes to functions and stuff, Strang's approach is likely to be more immediately profitable. I like 'em both. I helped my daughter through UIUC linear algebra last semester, and Strang definitely prepped me for all the computation, all the way through the end of the class, but I needed to go back to Axler for all the proof stuff.
  - seanhunter 3y ago
    I'm really enjoying them both for that precise reason.
penguin_booze 3y ago
I never found Strang's lectures out of ordinary nor particularly enlightening. But what managed to give me perspective was Pavel Grinfeld's playlists, starting from https://www.youtube.com/watch?v=Fnfh8jNqBlg&list=PLlXfTHzgMRUKXD88IdzS14F4NxAZudSmv https://www.youtube.com/watch?v=Fnfh8jNqBlg&list=PLlXfTHzgMR....

vouaobrasil 3y ago

As a mathematician, if there's one piece of advice I'd give people who need 1-2 math courses, it would be LEARN IT RIGHT. Pick up a rigorous textbook and read through it. Don't be lazy. I've spent decades reading many texts and papers. If you're smart, reading one real and thorough book will not take a lot of time. And, truly understandng the subject will pay you back in spades. Don't go for fluff. Linear algebra isn't some Facebook post. Truly go through the book line by line and really understand it down to the nuts and bolts. Suffer a little.

rramadass 3y ago
This is the advice for the ages! However i fear it is lost on the current generation which is always looking for that quick/short article/blog/video/etc. which will teach and make them understand the subject as painlessly as possible. Everything must be "Fun and Enjoyable" (i don't even know what that means anymore!). They have forgotten the maxim "There is no Royal Road to Mathematics"(or any other scientific field). The amount of people (even on HN!) who espouse disdain for Textbooks is astonishing. By definition, the process of learning involves diving into the unknown and hence there will be discomfort/difficulties/effort/time needed and thus is not going to be easy. But the student doesn't want to put in any effort at all; everything is the fault of the Teacher(can't teach properly)/Teaching method(i am a visual learner)/Too rigorous/Mathematical/I have ADHD/ADD/Autism/Whatever. The real tragedy is that used Books are now so easily/cheaply/universally available that there is no reason not to have one's own library of books on subjects of interest; it is "food" for the Mind.

dekervin 3y ago

We have a discord "HN Learn", where we collaborate to learn things, especially Maths. If you want some company to delve into linear algebra, feel free to join. [0] https://discord.gg/RxSjEMnW https://discord.gg/RxSjEMnW