7 ms·

Can anyone suggest a something that teaches Linear Algebra with a practical applications, especially for software engineering? I can sorta kinda get the theory

Can anyone suggest a something that teaches Linear Algebra with a practical applications, especially for software engineering? I can sorta kinda get the theory, but every demonstration involves moving an arrow around which is.... not something I need to do frequently. So I'm not sure how I actually apply linear algebra to solve actual problems. I'm a software developer and I know it's useful I just don't get where to use it - and I'm struggling to actually understand the different operations, purpose of the dot product, etc. I have a decent base for basic stats and calc, both of which I can "conceptually apply" near daily for understanding how things work. 3Blue1Brown is helpful, but I just kinda go "yeah I guess that looks right" without knowing what to do with it. EDIT: Thank you!

cratermoon 3y ago
Much of the use of linear algebra in programming is for machine learning. To a first approximation, ML is statistics on huge datasets, and linear algebra makes it possible because matrix operations are massively, if not embarrassingly, parallel.
latenightcoding 3y ago
Fast ai had a computational linear algebra class iirc
ak_111 3y ago
Look into making a Doom-like game, tons of linear algebra there.
tptacek 3y ago
One place to start might be a tutorial on principal component analysis, which will take you through some of the intuitions for applying SVD. You can also go in the direction of cryptography; here's, for instance, a really excellent LLL tutorial that builds on Graham-Schmidt: https://kel.bz/post/lll/ https://kel.bz/post/lll/
- anon____ 3y ago
  This is the PCA tutorial that worked for me: https://arxiv.org/abs/1404.1100 https://arxiv.org/abs/1404.1100.
dlevine 3y ago
Imperial College has a pretty good course on Linear Algebra for Machine Learning on Coursera. It's part of their Mathematics for Machine Learning Specialization. Deep Learning also has a course that appears to be similar. The specialization as a whole starts simple, and works its way up to PCA. https://www.coursera.org/specializations/mathematics-for-machine-learning-and-data-science https://www.coursera.org/specializations/mathematics-for-mac...
ndriscoll 3y ago
I don't know a recommendation to make; probably it makes more sense to find a field your interested in (pretty much any STEM field will do), and learn the more math heavy version of that. Inevitably linear algebra will come up, giving some motivation for the pure theory. As for why it comes up so much, it concerns itself with solving systems that look like y = Ax + b, where the Ax term works similarly to multiplication in 1-D. The point is these simple equations are ones we can actually understand! Everything else is too hard. But there's a trick we have for everything else: if you have some y = f(x) where f is super complicated, you can differentiate. The derivative of f at a point x_0 is the best linear approximation to f. i.e. f'(x_0) is the best matrix A such that y ~= Ax + b near x_0. Now your problem is linear and you can understand it (locally)! Then you can integrate your local solutions into a global one. The purpose of dot products is that they let you talk about things like angles, lengths, and projections. The point is you learn how it works for arrows and shadows and stuff, and figure out some equations that hopefully make intuitive sense in 2- and 3-D, and then it turns out those equations work in higher (even infinite) dimensions too. Projections are useful because they let you break vectors down and build them back up, and hopefully the broken down version is easier to understand. Understanding projections in high or infinite dimensions gives some intuition for things like the Fourier transform, where you project a function onto simpler waves, maybe study how a system reacts to those waves, and then use that description to build back how up the system reacts to your original function. Angles give one way to measure closeness. If you have some machine learning model that figures out a way to map text into a 50,000 dimensional space, you might be able to do it in a way where two sentences are intuitively similar if they are mostly pointing off in the same directions, so if the angle between them is small. So tl;dr the idea is you learn some geometry with arrows and all that, you figure out some equations from that geometry, and then you realize that those equations and that geometric intuition work anytime you have a linear (i.e. f(ax+b) = af(x) + f(b)) system. Calculus gives you ways to turn non-linear problems into linear ones, so you will find examples of linear systems everywhere.
farrelle25 3y ago
I found a lot of practical CS applications in this one: "Coding The Matrix: Linear Algebra Through Computer Science Applications" https://codingthematrix.com/ https://codingthematrix.com/ It used to be on Coursera too, has video lectures and a book to go with it.
- gensym 3y ago
  I endorse this as well. I worked through the course back when it was on Coursera, and thoroughly enjoyed it. I've also found what I learned applicable many times since - most recently when experimenting with some computer vision algorithms.
nextos 3y ago
Boyd & Vandenberghe's new book: https://web.stanford.edu/~boyd/vmls https://web.stanford.edu/~boyd/vmls It has companion code in Julia and Python, and addresses many important applications, such as convex optimization where the authors' previous book is really famous.
JKCalhoun 3y ago
I just posted this in another comment: https://archive.org/details/build-your-own-flight-sim-in-c-dos-game-dev-michael-radtke-chris-lampton/ https://archive.org/details/build-your-own-flight-sim-in-c-d... It turns out dot products, matrix multiplication, etc. are fairly common in projecting a 3D world onto a 2D display (flight simulators, for example).
sevensor 3y ago
Find and work through some old OpenGL tutorials from the fixed-function era. It's long obsolete, but there's a nice immediate connection between your matrix algebra and what you see on the screen. Putting chunky triangles on the screen is fun! Edit: I should mention that this is a good way to learn to appreciate what modern shaders are doing. You could jump into those as well, but I think it's a little harder to get your first triangle on the screen. All this assumes you're comfortable writing C, of course.
j2kun 3y ago
Two chapters of my book https://pimbook.org https://pimbook.org focus on linear algebra, with the first chapter being for SVD (a workhorse of data analysis) and the second demonstrating physics modeling using eigenvectors and eigenvalues. But if you really want practical applications, you probably want to read a book on numerical analysis, which is where linear algebra really finds its glory.
- tptacek 3y ago
  Love this book, by the way.
rramadass 3y ago
You might want to watch this short video a couple of times - https://www.youtube.com/watch?v=rowWM-MijXU https://www.youtube.com/watch?v=rowWM-MijXU A simple way to think about Linear Algebra is as a set of rules i.e. functions (aka Matrices) applied to input points (aka vectors) to give output points in a vector (i.e. multidimensional) space. Thus multi-valued inputs give multi-valued outputs. The key is to keep both the algebraic manipulations and what they may mean in the geometric interpretation simultaneously in mind. The basic example of solving a system of linear equations in two variables will cement this idea when you map it to the XY coordinate system. You might find the book Practical Linear Algebra : A Geometry Toolbox by Farin and Hansford useful for further study.
gmiller123456 3y ago
In addition to what has already been recommended, I would recommend going to a used book site and typing in "applied linear algebra" or "linear algebra applications". You'll see a ton of books, many under $10 shipped. Usually old textbooks about 5-10 years old, but the math doesn't go out of date.
owlbite 3y ago
In my experience it largely depends on the field within Software Engineering, and where it blurs into subfields of Applied Math / Engineering. Machine Learning needs some things (mainly "easy" things like matrix multiplication and convolution with the occasional truncated SVD), Modeling of Physical Systems needs other things (PDE solvers), Computer graphics different stuff (mainly focusing on small rotation/transformation matrices applied to many many points), Nonlinear Optimization yet another subset (solution of large sparse systems), and that's before you get to signal processing and statistics. The main commonality is once you have the basic terminology down and understand that if you're explicitly inverting a matrix you're probably doing it wrong, you should just use the existing highly tuned libraries for your use area (at least until you decide it would be cool to try and beat them). Once you need to go beyond that you're more into the realms of matrix analysis and structure exploitation, and firmly have at least one foot in the math camp.

cratermoon 3y ago

Much of the use of linear algebra in programming is for machine learning. To a first approximation, ML is statistics on huge datasets, and linear algebra makes it possible because matrix operations are massively, if not embarrassingly, parallel.

latenightcoding 3y ago

Fast ai had a computational linear algebra class iirc

ak_111 3y ago

Look into making a Doom-like game, tons of linear algebra there.

tptacek 3y ago

One place to start might be a tutorial on principal component analysis, which will take you through some of the intuitions for applying SVD. You can also go in the direction of cryptography; here's, for instance, a really excellent LLL tutorial that builds on Graham-Schmidt: https://kel.bz/post/lll/ https://kel.bz/post/lll/

anon____ 3y ago
This is the PCA tutorial that worked for me: https://arxiv.org/abs/1404.1100 https://arxiv.org/abs/1404.1100.

dlevine 3y ago

Imperial College has a pretty good course on Linear Algebra for Machine Learning on Coursera. It's part of their Mathematics for Machine Learning Specialization. Deep Learning also has a course that appears to be similar. The specialization as a whole starts simple, and works its way up to PCA. https://www.coursera.org/specializations/mathematics-for-machine-learning-and-data-science https://www.coursera.org/specializations/mathematics-for-mac...

ndriscoll 3y ago

I don't know a recommendation to make; probably it makes more sense to find a field your interested in (pretty much any STEM field will do), and learn the more math heavy version of that. Inevitably linear algebra will come up, giving some motivation for the pure theory. As for why it comes up so much, it concerns itself with solving systems that look like y = Ax + b, where the Ax term works similarly to multiplication in 1-D. The point is these simple equations are ones we can actually understand! Everything else is too hard. But there's a trick we have for everything else: if you have some y = f(x) where f is super complicated, you can differentiate. The derivative of f at a point x_0 is the best linear approximation to f. i.e. f'(x_0) is the best matrix A such that y ~= Ax + b near x_0. Now your problem is linear and you can understand it (locally)! Then you can integrate your local solutions into a global one. The purpose of dot products is that they let you talk about things like angles, lengths, and projections. The point is you learn how it works for arrows and shadows and stuff, and figure out some equations that hopefully make intuitive sense in 2- and 3-D, and then it turns out those equations work in higher (even infinite) dimensions too. Projections are useful because they let you break vectors down and build them back up, and hopefully the broken down version is easier to understand. Understanding projections in high or infinite dimensions gives some intuition for things like the Fourier transform, where you project a function onto simpler waves, maybe study how a system reacts to those waves, and then use that description to build back how up the system reacts to your original function. Angles give one way to measure closeness. If you have some machine learning model that figures out a way to map text into a 50,000 dimensional space, you might be able to do it in a way where two sentences are intuitively similar if they are mostly pointing off in the same directions, so if the angle between them is small. So tl;dr the idea is you learn some geometry with arrows and all that, you figure out some equations from that geometry, and then you realize that those equations and that geometric intuition work anytime you have a linear (i.e. f(ax+b) = af(x) + f(b)) system. Calculus gives you ways to turn non-linear problems into linear ones, so you will find examples of linear systems everywhere.

farrelle25 3y ago

I found a lot of practical CS applications in this one: "Coding The Matrix: Linear Algebra Through Computer Science Applications" https://codingthematrix.com/ https://codingthematrix.com/ It used to be on Coursera too, has video lectures and a book to go with it.

gensym 3y ago
I endorse this as well. I worked through the course back when it was on Coursera, and thoroughly enjoyed it. I've also found what I learned applicable many times since - most recently when experimenting with some computer vision algorithms.

nextos 3y ago

Boyd & Vandenberghe's new book: https://web.stanford.edu/~boyd/vmls https://web.stanford.edu/~boyd/vmls It has companion code in Julia and Python, and addresses many important applications, such as convex optimization where the authors' previous book is really famous.

JKCalhoun 3y ago

I just posted this in another comment: https://archive.org/details/build-your-own-flight-sim-in-c-dos-game-dev-michael-radtke-chris-lampton/ https://archive.org/details/build-your-own-flight-sim-in-c-d... It turns out dot products, matrix multiplication, etc. are fairly common in projecting a 3D world onto a 2D display (flight simulators, for example).

sevensor 3y ago

Find and work through some old OpenGL tutorials from the fixed-function era. It's long obsolete, but there's a nice immediate connection between your matrix algebra and what you see on the screen. Putting chunky triangles on the screen is fun! Edit: I should mention that this is a good way to learn to appreciate what modern shaders are doing. You could jump into those as well, but I think it's a little harder to get your first triangle on the screen. All this assumes you're comfortable writing C, of course.

j2kun 3y ago

Two chapters of my book https://pimbook.org https://pimbook.org focus on linear algebra, with the first chapter being for SVD (a workhorse of data analysis) and the second demonstrating physics modeling using eigenvectors and eigenvalues. But if you really want practical applications, you probably want to read a book on numerical analysis, which is where linear algebra really finds its glory.

tptacek 3y ago
Love this book, by the way.

rramadass 3y ago

You might want to watch this short video a couple of times - https://www.youtube.com/watch?v=rowWM-MijXU https://www.youtube.com/watch?v=rowWM-MijXU A simple way to think about Linear Algebra is as a set of rules i.e. functions (aka Matrices) applied to input points (aka vectors) to give output points in a vector (i.e. multidimensional) space. Thus multi-valued inputs give multi-valued outputs. The key is to keep both the algebraic manipulations and what they may mean in the geometric interpretation simultaneously in mind. The basic example of solving a system of linear equations in two variables will cement this idea when you map it to the XY coordinate system. You might find the book Practical Linear Algebra : A Geometry Toolbox by Farin and Hansford useful for further study.

gmiller123456 3y ago

In addition to what has already been recommended, I would recommend going to a used book site and typing in "applied linear algebra" or "linear algebra applications". You'll see a ton of books, many under $10 shipped. Usually old textbooks about 5-10 years old, but the math doesn't go out of date.

owlbite 3y ago

In my experience it largely depends on the field within Software Engineering, and where it blurs into subfields of Applied Math / Engineering. Machine Learning needs some things (mainly "easy" things like matrix multiplication and convolution with the occasional truncated SVD), Modeling of Physical Systems needs other things (PDE solvers), Computer graphics different stuff (mainly focusing on small rotation/transformation matrices applied to many many points), Nonlinear Optimization yet another subset (solution of large sparse systems), and that's before you get to signal processing and statistics. The main commonality is once you have the basic terminology down and understand that if you're explicitly inverting a matrix you're probably doing it wrong, you should just use the existing highly tuned libraries for your use area (at least until you decide it would be cool to try and beat them). Once you need to go beyond that you're more into the realms of matrix analysis and structure exploitation, and firmly have at least one foot in the math camp.