5 ms·

You can keep adding data points to your spline/polynomial interpolation, but that doesn't mean the model will be any better. In fact, for the purposes of predic

You can keep adding data points to your spline/polynomial interpolation, but that doesn't mean the model will be any better. In fact, for the purposes of predictive power, it will probably be worse. I'm not extremely well educated on the subject, so somebody correct me, but these predictive models have always struck me as just complicated splines. Add a thousand data points, and the "predictions" for old datapoints look really good. The problem isn't with the math, it's with an inappropriate application of the math based either on fundamental misunderstanding of extrapolation or on a fundamental failing of human psychology in not realizing that the model was built on the data, and not the other way around.

mikecane 16y ago
Long-Term Capital Management always springs to my mind. http://en.wikipedia.org/wiki/LTCM http://en.wikipedia.org/wiki/LTCM
kiba 16y ago
Taleb's empiricism implies resisting generalization from data and limiting the derivation of general rules from particular observations as one can be missing hidden properties. Thus he believes that scientists, economists, historians, policy makers, businessmen, and financiers are victims of an illusion of pattern. They overestimate the value of rational explanations of past data, and underestimate the prevalence of unexplainable randomness in those data. -- Wikipedia on Nassim Nicholas Taleb People project patterns onto data when in fact, they don't exists.
- mikecane 16y ago
  "Fooled by Randomness" is a great book. But I sometimes feel about it the same way I feel about people who say lotteries are a waste of time: some people do win. And sometimes, despite everything, a transient pattern does repeat. How frustrating! Edited to add this, which is sometimes also brought up: http://en.wikipedia.org/wiki/Illusion_of_control http://en.wikipedia.org/wiki/Illusion_of_control
  - Tichy 16y ago
    Yeah, now Taleb is betting on the big random event, but what if the unexpected thing is that the unexpected thing never happens?
- HSO 16y ago
  http://en.wikipedia.org/wiki/Apophenia http://en.wikipedia.org/wiki/Apophenia
- rbanffy 16y ago
  > People project patterns onto data when in fact, they don't exists. So, the patterns we see in the Ulam spiral can be completely random? We observe patterns. Human brains are really good in finding them. The problem is finding patterns in past data may or may not provide useful insights into future data. With the resources being dedicated to this problem, I find it highly unlikely I could be able to offer any helpful advice that would prove more helpful than what already exists. It would be hard enough to predict the market if the only factors influencing it were signals coming out of the companies we trade. Predicting what other algos may me doing in the space we are trying to examine and the influence their trades exert in its future is... Well... I won't try.
alextp 16y ago
Yes, that is the main pitfall with machine learning in general. Usually you do two things to mitigate it (but they don't fix it entirely): regularization and a test set. Regularization is some means of complexity control. You switch form trying to find the model that has minimal error on your data to finding the model that minimizes "error(model, data) + C*complexity(model)". There are many ways to measure the complexity, and usually the value of C is pretty unimportante (it is empirically found that it only affects the solutions when you change it in orders of magnitude). Keeping a test set is always testing your predictions on held-out data that you didn't use to train the model and that is hopefully independent from your training data. Still, if you come up with a huge number of models and test them on the same test data you're statistically bound to find a few that seem to work very well, until you try it in different data. My feeling with most of these market-beating schemes is that just the price history of stocks shouldn't be that predictive of future prices (but aggregating real-world information should). After all, usually what creates spikes and valleys in the prices are some relevant news or decisions.
- Robin_Message 16y ago
  Isn't it better to adopt a Baynesian approach and directly model the complexity of the model rather than some arbitrary Ccomplexity? To wit, P(model|data)=P(data|model)P(model) / P(data), which is to say that you calculate the probability a model is true given the data you see based on the probability you see the data you see given a certain model, multiplied by the probability of the model you are using. I guess the problem in all this is we want* the model to be very complex, but we can't tell that complexity from overfitting (and it may be that no model exists so the only thing you can do is overfit.)
  - alextp 16y ago
    Yes, but the formulation I showed you is equivalent to a bayesian prior. For example, if you want to learn a weight vector w that gives high likelihood to the data and has a gaussian prior with 0 mean and Cidentity covariance, the MAP answer is "minimize -log(likelihood) + C||w||", where ||w|| is the square norm of w. Equivalently, if the prior is a laplacian you just change the norm from the l2 to the l1 norm. Being bayesian gives you an extra capability that is model averaging, and this does usually improve the behavior at a high computational cost. I really like bayesian models, and right now I'm experimenting with one that should do unsupervised sentiment analysis without a priori knowledge of word polarity or things like that (yes, I'm a phd student in machine learning).
roel_v 16y ago
There are, roughly, two schools in modeling (I'm not in the finance modeling field but I see no reason to believe it's different there): extrapolation and process modeling. Extrapolation becomes better with more data and more accurate descriptions of the curve. Process modeling becomes better with (duh) understanding the processes better. Of course in practice all models incorporate aspects of both. My point is that just adding data isn't the only way to improve a model, and in my opinion, it's even only relevant after you've achieved a sufficient level of accuracy in your process description. In my limited understanding of financial modeling, quants focus a lot on the mathematical aspects.

mikecane 16y ago

Long-Term Capital Management always springs to my mind. http://en.wikipedia.org/wiki/LTCM http://en.wikipedia.org/wiki/LTCM

kiba 16y ago

Taleb's empiricism implies resisting generalization from data and limiting the derivation of general rules from particular observations as one can be missing hidden properties. Thus he believes that scientists, economists, historians, policy makers, businessmen, and financiers are victims of an illusion of pattern. They overestimate the value of rational explanations of past data, and underestimate the prevalence of unexplainable randomness in those data. -- Wikipedia on Nassim Nicholas Taleb People project patterns onto data when in fact, they don't exists.

mikecane 16y ago
"Fooled by Randomness" is a great book. But I sometimes feel about it the same way I feel about people who say lotteries are a waste of time: some people do win. And sometimes, despite everything, a transient pattern does repeat. How frustrating! Edited to add this, which is sometimes also brought up: http://en.wikipedia.org/wiki/Illusion_of_control http://en.wikipedia.org/wiki/Illusion_of_control
- Tichy 16y ago
  Yeah, now Taleb is betting on the big random event, but what if the unexpected thing is that the unexpected thing never happens?
HSO 16y ago
http://en.wikipedia.org/wiki/Apophenia http://en.wikipedia.org/wiki/Apophenia
rbanffy 16y ago
> People project patterns onto data when in fact, they don't exists. So, the patterns we see in the Ulam spiral can be completely random? We observe patterns. Human brains are really good in finding them. The problem is finding patterns in past data may or may not provide useful insights into future data. With the resources being dedicated to this problem, I find it highly unlikely I could be able to offer any helpful advice that would prove more helpful than what already exists. It would be hard enough to predict the market if the only factors influencing it were signals coming out of the companies we trade. Predicting what other algos may me doing in the space we are trying to examine and the influence their trades exert in its future is... Well... I won't try.

alextp 16y ago

Yes, that is the main pitfall with machine learning in general. Usually you do two things to mitigate it (but they don't fix it entirely): regularization and a test set. Regularization is some means of complexity control. You switch form trying to find the model that has minimal error on your data to finding the model that minimizes "error(model, data) + C*complexity(model)". There are many ways to measure the complexity, and usually the value of C is pretty unimportante (it is empirically found that it only affects the solutions when you change it in orders of magnitude). Keeping a test set is always testing your predictions on held-out data that you didn't use to train the model and that is hopefully independent from your training data. Still, if you come up with a huge number of models and test them on the same test data you're statistically bound to find a few that seem to work very well, until you try it in different data. My feeling with most of these market-beating schemes is that just the price history of stocks shouldn't be that predictive of future prices (but aggregating real-world information should). After all, usually what creates spikes and valleys in the prices are some relevant news or decisions.

Robin_Message 16y ago
Isn't it better to adopt a Baynesian approach and directly model the complexity of the model rather than some arbitrary Ccomplexity? To wit, P(model|data)=P(data|model)P(model) / P(data), which is to say that you calculate the probability a model is true given the data you see based on the probability you see the data you see given a certain model, multiplied by the probability of the model you are using. I guess the problem in all this is we want* the model to be very complex, but we can't tell that complexity from overfitting (and it may be that no model exists so the only thing you can do is overfit.)
- alextp 16y ago
  Yes, but the formulation I showed you is equivalent to a bayesian prior. For example, if you want to learn a weight vector w that gives high likelihood to the data and has a gaussian prior with 0 mean and Cidentity covariance, the MAP answer is "minimize -log(likelihood) + C||w||", where ||w|| is the square norm of w. Equivalently, if the prior is a laplacian you just change the norm from the l2 to the l1 norm. Being bayesian gives you an extra capability that is model averaging, and this does usually improve the behavior at a high computational cost. I really like bayesian models, and right now I'm experimenting with one that should do unsupervised sentiment analysis without a priori knowledge of word polarity or things like that (yes, I'm a phd student in machine learning).

roel_v 16y ago

There are, roughly, two schools in modeling (I'm not in the finance modeling field but I see no reason to believe it's different there): extrapolation and process modeling. Extrapolation becomes better with more data and more accurate descriptions of the curve. Process modeling becomes better with (duh) understanding the processes better. Of course in practice all models incorporate aspects of both. My point is that just adding data isn't the only way to improve a model, and in my opinion, it's even only relevant after you've achieved a sufficient level of accuracy in your process description. In my limited understanding of financial modeling, quants focus a lot on the mathematical aspects.