4 ms·

The author is basically using a linear algebra tool for creating orthogonal basis vectors of a matrix of stock prices. (The PCA is like eigenvector decompositio

by lbarrow 14y ago

The author is basically using a linear algebra tool for creating orthogonal basis vectors of a matrix of stock prices. (The PCA is like eigenvector decomposition, but it works on rectangular matrices too. In fact, unlike many operations, it's very fast on unbalanced rectangular matrices!) Since these vectors are, by definition, uncorrelated, they can be very useful in building CAPM-balanced stock portfolios. Using the PCA is great in this situation, but people often run into traps when using these sorts of spectral-decomposition methods on real world data. The most obvious is that they try to interpret what the vectors "represent". Sometimes this is reasonable -- if you did a similar experiment on the stock price of energy companies, the strongest vector probably really would be closely correlated with the price of oil. But aside from unusual situations like that, interpreting the "meaning" of spectral vectors is a fool's errand.

mturmon 14y ago
It's often true that you can figure out what the first handful (say, 3 to 6) PCA components mean, in a large problem. The first is usually the mean of the quantities. It is typical in practice to compute PCA by using the SVD of the data itself; if you subtract the mean first, then of course it will not appear as the first PCA component. In matlab, this is literally a one-liner using the svd of the original data -- not even forming a covariance matrix. No, the people who do this don't care if you know what the Karhunen-Loeve decomposition is, they just use the one-liner: [U,D] = svd(X) Anyway, after the mean, then you get the varying components. It's smart to plot these somehow, to interpret meaning. The post should have plotted the time history of the 6 stocks together with the time history of each PC, then some pattern might have suggested itself. The first PC could be as simple as "GOOG,AMZN,AAPL,AKAM going up, MSFT steady, and FB going down", given the stocks mentioned and their weighting. The classic examples are (mentioned elsewhere on the thead) the eigenfaces example (on Wikipedia), where PCA was used for faces, and various features like eyes, foreheads, and mouths are emphasized, plus "second-order" features like edges around the eyes, noses, and mouths. If you try it yourself, what you find is that adding more of these second-order features to a face (literally, adding, as in: new_image = old_image + alpha * second_order_feature where alpha is a small scalar) will shift the nose left or right, or make the mouth bigger. People have done the same thing with natural images, and out pop things like 2d wavelets (the Gabor filters, http://en.wikipedia.org/wiki/Gabor_filter http://en.wikipedia.org/wiki/Gabor_filter). It's somewhat magical, because you went in with no information, and out pops this structure, which also characterizes (surprise!) the human visual cortex. Other classic examples are in atmosphere/weather analysis, where ENSO ("el nino") will pop out of analysis of temperature and pressure fields in the Pacific ocean.
- psb217 14y ago
  FYI, Gabor-like filters pop out from doing ICA (i.e. Independent Components Analysis), not PCA. While PCA looks for orthogonal vectors onto which the data's projection is normally-distributed (among other properties), ICA, roughly speaking, looks for a set of orthogonal vectors onto which the data's projection has maximal kurtosis (among other properties). It is the kurtosis-maximization of ICA that tends to produce filters mimicking those found in (early layers of) visual cortex. Hence, the production of such filters by techniques like "sparse coding" and "sparse autoencoders", which explicitly pursue highly-kurtotic representations of the training data. PCA, on the other hand, tends to produce checkerboard (i.e. 2d sinusoidal) filters of various frequencies when trained on "natural image patches". See: "The 'independent components' of natural scenes are edge filters" by Bell and Sejnowski, 1997.
  - mturmon 14y ago
    Thanks for the reminder. I was thinking of this 1991 paper, which I ran into a long time ago: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.41.192 http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.41.1... They used a (linear) "neural network" with gradient descent training that implemented PCA (kind of an iterative graham-schmidt process), and got Gabor-like filters. I think a lot of people have done similar experiments, with varying results.
    - psb217 14y ago
      I hadn't seen that paper before; thanks for the reference. I read through it and saw that they were reweighting the sampled image patches with a Gaussian mask prior to learning, which explains how they got Gabor-like filters. The masking effectively forced the learned filters to have localized receptive fields, while locality/nonlocality is generally one of the (visually) clearer differences between filters learned with ICA/PCA. In other words, the Gaussian-modulated part of Gaussian-modulated sinusoids was built into their learning process, rather than appearing as an emergent property. I also chuckled a bit when they described how computing eigenvectors for 4096x4096 matrices was "beyond reasonable computation".
hobbyist 14y ago
Wow. thts cool really cool. PCA of energy companies being correlated with oil.. intriguing and interesting

by lbarrow 14y ago

mturmon 14y ago

It's often true that you can figure out what the first handful (say, 3 to 6) PCA components mean, in a large problem. The first is usually the mean of the quantities. It is typical in practice to compute PCA by using the SVD of the data itself; if you subtract the mean first, then of course it will not appear as the first PCA component. In matlab, this is literally a one-liner using the svd of the original data -- not even forming a covariance matrix. No, the people who do this don't care if you know what the Karhunen-Loeve decomposition is, they just use the one-liner: [U,D] = svd(X) Anyway, after the mean, then you get the varying components. It's smart to plot these somehow, to interpret meaning. The post should have plotted the time history of the 6 stocks together with the time history of each PC, then some pattern might have suggested itself. The first PC could be as simple as "GOOG,AMZN,AAPL,AKAM going up, MSFT steady, and FB going down", given the stocks mentioned and their weighting. The classic examples are (mentioned elsewhere on the thead) the eigenfaces example (on Wikipedia), where PCA was used for faces, and various features like eyes, foreheads, and mouths are emphasized, plus "second-order" features like edges around the eyes, noses, and mouths. If you try it yourself, what you find is that adding more of these second-order features to a face (literally, adding, as in: new_image = old_image + alpha * second_order_feature where alpha is a small scalar) will shift the nose left or right, or make the mouth bigger. People have done the same thing with natural images, and out pop things like 2d wavelets (the Gabor filters, http://en.wikipedia.org/wiki/Gabor_filter http://en.wikipedia.org/wiki/Gabor_filter). It's somewhat magical, because you went in with no information, and out pops this structure, which also characterizes (surprise!) the human visual cortex. Other classic examples are in atmosphere/weather analysis, where ENSO ("el nino") will pop out of analysis of temperature and pressure fields in the Pacific ocean.

psb217 14y ago
FYI, Gabor-like filters pop out from doing ICA (i.e. Independent Components Analysis), not PCA. While PCA looks for orthogonal vectors onto which the data's projection is normally-distributed (among other properties), ICA, roughly speaking, looks for a set of orthogonal vectors onto which the data's projection has maximal kurtosis (among other properties). It is the kurtosis-maximization of ICA that tends to produce filters mimicking those found in (early layers of) visual cortex. Hence, the production of such filters by techniques like "sparse coding" and "sparse autoencoders", which explicitly pursue highly-kurtotic representations of the training data. PCA, on the other hand, tends to produce checkerboard (i.e. 2d sinusoidal) filters of various frequencies when trained on "natural image patches". See: "The 'independent components' of natural scenes are edge filters" by Bell and Sejnowski, 1997.
- mturmon 14y ago
  Thanks for the reminder. I was thinking of this 1991 paper, which I ran into a long time ago: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.41.192 http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.41.1... They used a (linear) "neural network" with gradient descent training that implemented PCA (kind of an iterative graham-schmidt process), and got Gabor-like filters. I think a lot of people have done similar experiments, with varying results.
  - psb217 14y ago
    I hadn't seen that paper before; thanks for the reference. I read through it and saw that they were reweighting the sampled image patches with a Gaussian mask prior to learning, which explains how they got Gabor-like filters. The masking effectively forced the learned filters to have localized receptive fields, while locality/nonlocality is generally one of the (visually) clearer differences between filters learned with ICA/PCA. In other words, the Gaussian-modulated part of Gaussian-modulated sinusoids was built into their learning process, rather than appearing as an emergent property. I also chuckled a bit when they described how computing eigenvectors for 4096x4096 matrices was "beyond reasonable computation".

hobbyist 14y ago

Wow. thts cool really cool. PCA of energy companies being correlated with oil.. intriguing and interesting