4 ms·

What are people using Linear Algebra for in Data Science? Aside from the stock representing words as N-dimensional vectors I mean? I ask because I do this kind

What are people using Linear Algebra for in Data Science? Aside from the stock representing words as N-dimensional vectors I mean? I ask because I do this kind of work as my J.O.B. and every skill he refers to I totally see the necessity of except this one.

treycausey 12y ago
This has been the most common question I've gotten. I'd say partially it's my bias as someone who works in recommender systems a lot. Many/most common statistical problems have a compact matrix representation (e.g., systems of linear equations). Finally, everyday matrix decomposition tasks like SVD.
- darkxanthos 12y ago
  Thanks for the response. That gives me something new to learn/look into.
mjw 12y ago
In pretty much any statistical model with more than a handful of parameters (and almost all machine learning models do require more than a handful of parameters!), those parameters are represented as vectors or matrices. Linear algebra (and multivariate calculus) then become very important for reasoning about those parameters, fitting the models, making predictions and so on. The multivariate Gaussian distribution is a great example of this. It's probably the most fundamental and important distribution in statistics, and working with these distributions is pretty much pure linear algebra -- quadratic forms over vectors of parameters, eigendecompositions of covariance matrices etc. Even for non-statistically-motivated data mining: any time you're optimising over a lot of parameters, it's likely that linear algebra (and, as before, multivariate calculus) will help. Linear algebra is as important to calculus over multiple variables, as plain old high-scool algebra is to plain old univariate calculus.
petulla 12y ago
Linear algebra is necessary for understanding linear regression and most clustering/prediction models on a mathematical level. The truth is, though, you can do data science without understanding the underlying math insofar as you do understand your objectives and the meaning of the conclusions you draw.
srean 12y ago
I am actually _very_ puzzled by by this comment because its the polar opposite of a point of view that I would have expected. In fact I can think of very few datamining and machine learning algorithms where linear algebra does not play a role. Representing features of a datapoint as a vector, pervades and populates every pore of this field. Without an understanding of linear algebra you wouldn't have support vector machines, no kernel methods, no neural networks, no perceptrons, no gradient descent methods, no Newton / Quasi-Newton methods, no multi-dimensional (or as they say in statistics, multivariate) Gaussian random variables, no matrix factorization, no Pagerank, no Markov chains, this list can go on and on. Take the simplest of data science problems: you have one variable x and another variable y and you want to predict the value of y given x. Usually x is not a single scalar but n scalars (called a feature vector). Simplest thing you can do here is least squares and that is as linear algebraic as you can get. There many fancy ways of dealing with this problem but almost always it is reduced to solving a related linear system. The bottom line is this: we understand very few things. Thankfully linear algebra is one of the few things that we do understand, so almost every analytical problem is reduced to this case (if, but locally) and then solved. I would be very curious to know how you have been able to avoid linear algebra. It will give me a new and valuable perspective, because apart from "click button, didnt work? ok click the next button" data analysis I find it hard how one can do much data analysis without it. So please break my bubble, I will be thankful for it. Canned packages often do not work out of the box. The knowledge of linear comes very handy in analyzing and debugging why is the model not working" "oh I see this matrix is near singular, thats why my estimates are off the park", or "oh these two variables are very correlated, that is why gradient descent is having so much trouble converging fast", "ah I see why I am getting NaN here" etc etc. EDIT: darkxanthos, appreciate your comment. I would say it is a bit like driving. Knowing the internal mechanics is neither necessary nor sufficient, and hardly correlated with good driving skills when things are going well. But sometimes when things are not going as expected, it helps in debugging. Let me try and pique your interest: Note that the decision boundary of naive Bayes is actually a linear function of the log conditional probabilities considered all independent, with LA you can now also consider the case that they have dependence. Consider updating multi-armed bandit problems, the updates are variants of gradient descent, and its nature is indeed characterized by the eigenvalues of Hessian of the thing you want to optimize. Consider K-means clustering, one way to get very close to its global optimum is to solve the same cost function using linear algebraic updates (called spectral graph partitioning). By trig I think you have the dot-product of two vectors in mind, the related analysis actually does not rely much on trigonometric properties but heavily on the linear algebraic properties, in fact this what allows one to escalate affairs from simple linear feature vectors to extremely non-linear ones because even though they are nonlinear in the data space in some other space they are linear so people do the math in that space (called the kernel trick although I find that term quite silly) ..This thing, linear algebra, lurks everywhere, I tell you :)
gtani 12y ago
https://news.ycombinator.com/item?id=4635274 https://news.ycombinator.com/item?id=4635274 http://aix1.uottawa.ca/~jkhoury/app.htm http://aix1.uottawa.ca/~jkhoury/app.htm The linear algebra text by Anton (9th and 10th eds) has a huge section on applications of LA. Also this book does same for ODE's http://www.amazon.com/Topics-Mathematical-Modeling-K-Tung/dp/B00BV2NGZW http://www.amazon.com/Topics-Mathematical-Modeling-K-Tung/dp...

treycausey 12y ago

This has been the most common question I've gotten. I'd say partially it's my bias as someone who works in recommender systems a lot. Many/most common statistical problems have a compact matrix representation (e.g., systems of linear equations). Finally, everyday matrix decomposition tasks like SVD.

darkxanthos 12y ago
Thanks for the response. That gives me something new to learn/look into.

mjw 12y ago

In pretty much any statistical model with more than a handful of parameters (and almost all machine learning models do require more than a handful of parameters!), those parameters are represented as vectors or matrices. Linear algebra (and multivariate calculus) then become very important for reasoning about those parameters, fitting the models, making predictions and so on. The multivariate Gaussian distribution is a great example of this. It's probably the most fundamental and important distribution in statistics, and working with these distributions is pretty much pure linear algebra -- quadratic forms over vectors of parameters, eigendecompositions of covariance matrices etc. Even for non-statistically-motivated data mining: any time you're optimising over a lot of parameters, it's likely that linear algebra (and, as before, multivariate calculus) will help. Linear algebra is as important to calculus over multiple variables, as plain old high-scool algebra is to plain old univariate calculus.

petulla 12y ago

Linear algebra is necessary for understanding linear regression and most clustering/prediction models on a mathematical level. The truth is, though, you can do data science without understanding the underlying math insofar as you do understand your objectives and the meaning of the conclusions you draw.

srean 12y ago

I am actually _very_ puzzled by by this comment because its the polar opposite of a point of view that I would have expected. In fact I can think of very few datamining and machine learning algorithms where linear algebra does not play a role. Representing features of a datapoint as a vector, pervades and populates every pore of this field. Without an understanding of linear algebra you wouldn't have support vector machines, no kernel methods, no neural networks, no perceptrons, no gradient descent methods, no Newton / Quasi-Newton methods, no multi-dimensional (or as they say in statistics, multivariate) Gaussian random variables, no matrix factorization, no Pagerank, no Markov chains, this list can go on and on. Take the simplest of data science problems: you have one variable x and another variable y and you want to predict the value of y given x. Usually x is not a single scalar but n scalars (called a feature vector). Simplest thing you can do here is least squares and that is as linear algebraic as you can get. There many fancy ways of dealing with this problem but almost always it is reduced to solving a related linear system. The bottom line is this: we understand very few things. Thankfully linear algebra is one of the few things that we do understand, so almost every analytical problem is reduced to this case (if, but locally) and then solved. I would be very curious to know how you have been able to avoid linear algebra. It will give me a new and valuable perspective, because apart from "click button, didnt work? ok click the next button" data analysis I find it hard how one can do much data analysis without it. So please break my bubble, I will be thankful for it. Canned packages often do not work out of the box. The knowledge of linear comes very handy in analyzing and debugging why is the model not working" "oh I see this matrix is near singular, thats why my estimates are off the park", or "oh these two variables are very correlated, that is why gradient descent is having so much trouble converging fast", "ah I see why I am getting NaN here" etc etc. EDIT: darkxanthos, appreciate your comment. I would say it is a bit like driving. Knowing the internal mechanics is neither necessary nor sufficient, and hardly correlated with good driving skills when things are going well. But sometimes when things are not going as expected, it helps in debugging. Let me try and pique your interest: Note that the decision boundary of naive Bayes is actually a linear function of the log conditional probabilities considered all independent, with LA you can now also consider the case that they have dependence. Consider updating multi-armed bandit problems, the updates are variants of gradient descent, and its nature is indeed characterized by the eigenvalues of Hessian of the thing you want to optimize. Consider K-means clustering, one way to get very close to its global optimum is to solve the same cost function using linear algebraic updates (called spectral graph partitioning). By trig I think you have the dot-product of two vectors in mind, the related analysis actually does not rely much on trigonometric properties but heavily on the linear algebraic properties, in fact this what allows one to escalate affairs from simple linear feature vectors to extremely non-linear ones because even though they are nonlinear in the data space in some other space they are linear so people do the math in that space (called the kernel trick although I find that term quite silly) ..This thing, linear algebra, lurks everywhere, I tell you :)

gtani 12y ago

https://news.ycombinator.com/item?id=4635274 https://news.ycombinator.com/item?id=4635274 http://aix1.uottawa.ca/~jkhoury/app.htm http://aix1.uottawa.ca/~jkhoury/app.htm The linear algebra text by Anton (9th and 10th eds) has a huge section on applications of LA. Also this book does same for ODE's http://www.amazon.com/Topics-Mathematical-Modeling-K-Tung/dp/B00BV2NGZW http://www.amazon.com/Topics-Mathematical-Modeling-K-Tung/dp...