3 ms·

This is not really finance research so do not confuse the two. It is CS/ML with a financial application; two different things. Return predictability low in real

by zmk_ 8y ago

This is not really finance research so do not confuse the two. It is CS/ML with a financial application; two different things. Return predictability low in real data, even if you control for time- and firm-fixed effects.

a008t 8y ago
When applying CS/ML to finance, methodology is absolutely crucial. Yet most work - even work done by quants in major sell-side banks - pays little attention to methodology; it is often naively assumed that the methodology applied to classifying cat images can be directly transferred into the domain of finance. Here is an article that explains the problem well: http://zacharydavid.com/2017/08/06/fitting-to-noise-or-nothing-at-all-machine-learning-in-markets/ http://zacharydavid.com/2017/08/06/fitting-to-noise-or-nothi...
- zmk_ 8y ago
  But that's because it is being applied by people who come from classifying cat images and have little domain knowledge.
- nonbel 8y ago
  >"When applying CS/ML to finance, methodology is absolutely crucial." When wouldnt it be?
  - a008t 8y ago
    In other areas, we can often assume that each example in our dataset is an independent sample from some underlying distribution. We can also assume that the distribution stays constant. Furthermore, we have fairly high-accuracy ground truth labels. Also, we know that everything we need to classify a sample is in the data we are given. We may also have more data to play with than we really need. Take handwritten digit recognition task like MNIST, for example - the way we write the number '5' does not change much, and the distribution of the different ways people write that number stays pretty much the same over time. The labels we are given are very accurate and everything we need to classify the image is in the data. All these assumptions mean that picking the right methodology is fairly easy - anyone can read a short tutorial and get it right. None of the assumptions I listed hold in the world of finance. There is no standard methodology that everyone can agree should be followed. You really have to be very, very careful in order to produce useful results.
    - nonbel 8y ago
      It sounds like you just need to use a different methodology, not that methodology is unimportant for whatever ml 101 examples that became popular like MNIST, etc.
      - a008t 8y ago
        Yes, but I also think it's a lot easier to get it wrong in finance without even realizing it than in many other domains. Relatively more difficult to screw up the methodology on MNIST. Many mistakenly assume that the methodology from MNIST transfers directly to finance, which makes the problem worse. ALl of that means that when talking about applying ML to finance, discussing the methodology in detail is a must. If it is not talked about or only briefly mentioned, from my experience that usually means that the methodology used is rubbish. Whereas you don't need to focus as much on the methodology when talking about MNIST-like problems - one can usually assume a reasonable one is used.
dx034 8y ago
If you use financial performance to measure the success of a model, methodology is critical. It's easy to have models that generate outperformance on paper, in reality it turns out that nearly all of those had some flaw in them.
- nonbel 8y ago
  >"If you use financial performance to measure the success of a model, methodology is critical." When wouldn't it be? And why does this sound exactly like the other post I just read?
  - dx034 8y ago
    If you compare to other results, small problems with timing usually don't impact results as severely. You don't get completely different results if you forget to move one time series by just a few hours. As for the second part, I don't know. Probably because we had the same thought? Not sure which post you refer to.

by zmk_ 8y ago

a008t 8y ago

When applying CS/ML to finance, methodology is absolutely crucial. Yet most work - even work done by quants in major sell-side banks - pays little attention to methodology; it is often naively assumed that the methodology applied to classifying cat images can be directly transferred into the domain of finance. Here is an article that explains the problem well: http://zacharydavid.com/2017/08/06/fitting-to-noise-or-nothing-at-all-machine-learning-in-markets/ http://zacharydavid.com/2017/08/06/fitting-to-noise-or-nothi...

zmk_ 8y ago
But that's because it is being applied by people who come from classifying cat images and have little domain knowledge.
nonbel 8y ago
>"When applying CS/ML to finance, methodology is absolutely crucial." When wouldnt it be?
- a008t 8y ago
  In other areas, we can often assume that each example in our dataset is an independent sample from some underlying distribution. We can also assume that the distribution stays constant. Furthermore, we have fairly high-accuracy ground truth labels. Also, we know that everything we need to classify a sample is in the data we are given. We may also have more data to play with than we really need. Take handwritten digit recognition task like MNIST, for example - the way we write the number '5' does not change much, and the distribution of the different ways people write that number stays pretty much the same over time. The labels we are given are very accurate and everything we need to classify the image is in the data. All these assumptions mean that picking the right methodology is fairly easy - anyone can read a short tutorial and get it right. None of the assumptions I listed hold in the world of finance. There is no standard methodology that everyone can agree should be followed. You really have to be very, very careful in order to produce useful results.
  - nonbel 8y ago
    It sounds like you just need to use a different methodology, not that methodology is unimportant for whatever ml 101 examples that became popular like MNIST, etc.
    - a008t 8y ago
      Yes, but I also think it's a lot easier to get it wrong in finance without even realizing it than in many other domains. Relatively more difficult to screw up the methodology on MNIST. Many mistakenly assume that the methodology from MNIST transfers directly to finance, which makes the problem worse. ALl of that means that when talking about applying ML to finance, discussing the methodology in detail is a must. If it is not talked about or only briefly mentioned, from my experience that usually means that the methodology used is rubbish. Whereas you don't need to focus as much on the methodology when talking about MNIST-like problems - one can usually assume a reasonable one is used.

dx034 8y ago

If you use financial performance to measure the success of a model, methodology is critical. It's easy to have models that generate outperformance on paper, in reality it turns out that nearly all of those had some flaw in them.

nonbel 8y ago
>"If you use financial performance to measure the success of a model, methodology is critical." When wouldn't it be? And why does this sound exactly like the other post I just read?
- dx034 8y ago
  If you compare to other results, small problems with timing usually don't impact results as severely. You don't get completely different results if you forget to move one time series by just a few hours. As for the second part, I don't know. Probably because we had the same thought? Not sure which post you refer to.