27 ms·

Fitting to Noise or Nothing at All: Machine Learning in Markets

295 points by bilifuduo 9y ago

zacharydavid 9y ago
Sorry guys. Traffic killed the site. Booting up a new server
- rubatuga 9y ago
  Do you have a cached copy?
  - zacharydavid 9y ago
    En route
- zacharydavid 9y ago
  it's up, propagation should take a minute
zacharydavid 9y ago
Special thanks to Nickolas Younker (at LiquidWeb) for saving my behind and getting this all set up.
- arjie 9y ago
  Doesn't load for me, mate. Also, if you put your email in your profile, I can send you an email instead of posting a comment here.
  - arthurcolle 9y ago
    Same here. Unfortunate...
    - zacharydavid 9y ago
      Sorry you two are still having issues. I had to do a new incognito window to get to it
- nyounker 9y ago
  I was happy to help. Migrations are very complicated especially when in the face of massive traffic to move from one server to another and keep it live. I apologize that the dns transition wasnt seemless, but I managed a full migration in under 30 minutes. I'm glad I was able to help.
- nyounker 9y ago
  I was happy to help. Migrations are very complicated especially when in the face of massive traffic to move from one server to another and keep it live. I apologize that the dns transition wasnt seemless, but I managed a full migration in under 30 minutes. I'm glad I was able to help.
- nyounker 9y ago
  I was happy to help. Migrations are very complicated especially when in the face of massive traffic to move from one server to another and keep it live. I apologize that the dns transition wasnt seemless, but I managed a full migration in under 30 minutes. I'm glad I was able to help.
dkural 9y ago
Most academic CS literature is complete BS. The vast majority of papers fit into a simple formula of "We apply method X to problem Y, and outperform other approaches using approaches similar to method X". Meanwhile, no one uses method X or any of its cousins, because in the real world the problem is solved very differently with a combination of both principled algorithms and heuristics derived from real-world datasets. The paper also fails to give any theoretical reason or mathematical insight as to why their version of X is better. Thus, it doesn't actually solve a real world problem OR advance scientific understanding.
- jfoutz 9y ago
  Could an enterprising person find such papers, cleverly select a dataset such that X is way worse, and finally refute the original paper? This seems like something an amateur could do. Wouldn't that tear down a bunch of this junk?
  - shimon 9y ago
    Sure. But for approximately the same level of effort, you could generate a new paper in the same template. It would likely be easier to get that "novel result" published than the refutation.
    - beagle3 9y ago
      And that's generally the truth in every scientific field. Right now you have a "reproduction renaissance" in psychology and cancer, where you can get published, and get acknowledged, if you tear down a high level paper. But in general, unless you aim at refuting the highest level, it is hard to get published and advances your career nowhere, so people have no incentive to do it.
      - nonbel 9y ago
        A non-replication does not "tear down" a paper... It is not any kind of "attack", your attitude is one of the sources of the problem. Edit: When I did medical research I wished people would try to replicate my findings so I could see how consistent the results were and how well my model fit future data... In fact it was really annoying that it was unlikely anyone would ever try to do a direct replication.
murbard2 9y ago
I got into quant finance 12 years ago with the mistaken idea that I was going to successfully use all these cool machine learning techniques (genetic programming! SVMs! neural networks!) to run great statistical arbitrage books. Most machine learning techniques focus on problems where the signal is very strong, but the structure is very complex. For instance, take the problem of recognizing whether a picture is a picture of a bird. A human will do well on this task, which shows that there is very little intrinsic noise. However, the correlation of any given pixel with the class of the image is essentially 0. The "noise" is in discovering the unknown relationship between pixels and class, not in the actual output. Noise dominates everything you will find in statistical arbitrage. R^2 of 1% are something to write home about. With this amount of noise, it's generally hard to do much better than a linear regression. Any model complexity has to come from integrating over latent parameters or manual feature engineering, the rest will overfit. I think Geoffrey Hinton said that statistics and machine learning are really the same thing, but since we have two different names for it, we might as well call machine learning everything that focuses on dealing with problems with a complex structure and low noise, and statistics everything that focuses on dealing with problems with a large amount of noise. I like this distinction, and I did end up picking up a lot of statistics working in this field. I'll regularly get emails from friends who tried some machine learning technique on some dataset and found promising results. As the article points out, these generally don't hold up. Accounting for every source of bias in a backtest is an art. The most common mistake is to assume that you can observe the relative price of two stocks at the close, and trade at that price. Many pairs trading strategies appear to work if you make this assumption (which tends to be the case if all you have are daily bars), but they really do not. Others include: assuming transaction costs will be the same on average (they won't, your strategy likely detects opportunities at time were the spread is very large and prices are bad), assuming index memberships don't change (they do and that creates selection bias), assuming you can short anything (stocks can be hard to short or have high borrowing costs), etc. In general, statistical arbitrage isn't machine learning bound(1), and it is not a data mining endeavor. Understanding the latent market dynamics you are trying to capitalize on, finding new data feeds that provide valuable information, carefully building out a model to test your hypothesis, deriving a sound trading strategy from that model is how it works. (1: this isn't always true. For instance, analyzing news with NLP, or using computer vision to estimate crop outputs from satellite imagery can make use of machine learning techniques to yield useful, tradeable signals. My comment mostly focuses on machine learning applied to price information. )
dogruck 9y ago
Would be nice to see a standard academic platform for backtesting. Then, the paper could say "we submitted our implementation of this strategy to Backtest (which includes transaction costs and slippage)."
- sgt101 9y ago
  I think the pymc3 people have made something available
chvid 9y ago
Does anyone know of any paper that describes a reproducible method of generating above normal returns in the mature western financial markets? Nope. Me neither.
- proofofstake 9y ago
  Do you consider cryptocurrency mature?
  - chvid 9y ago
    Nope. But even for those types of market I doubt that there exists an academic paper describing a reproducible trading strategy/method with above normal returns.
    - proofofstake 9y ago
      https://pdfs.semanticscholar.org/db15/1836543d8a70db1dabef3dee43637a7cd29f.pdf https://pdfs.semanticscholar.org/db15/1836543d8a70db1dabef3d... "Bayesian regression and Bitcoin" > The strategy is able to nearly double the investment in less than 60 day period when run against real data trace. http://cs229.stanford.edu/proj2015/029_report.pdf http://cs229.stanford.edu/proj2015/029_report.pdf "Algorithmic Trading of Cryptocurrency Based on Twitter Sentiment Analysis" http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0177630 http://journals.plos.org/plosone/article?id=10.1371/journal.... "When Bitcoin encounters information in an online forum: Using text mining to analyse user opinions and predict value fluctuation" https://pdfs.semanticscholar.org/e065/3631b4a476abf5276a264f6bbff40b132061.pdf https://pdfs.semanticscholar.org/e065/3631b4a476abf5276a264f... "Automated Bitcoin Trading via Machine Learning Algorithms"
      - chvid 9y ago
        "Based on this price prediction method, we devise a simple strategy for trading Bitcoin. The strategy is able to nearly double the investment in less than 60 day period when run against real data trace." From the paper "Bayesian regression and Bitcoin". I think are more relevant question is: This is clearly not the case in the real world - so why does it appear to be like that? But thanks for the links; I will enjoy reading through them.
lettergram 9y ago
I can't up vote this enough.. I've spend an unreasonable amount of time reading research related to finance for my web app Piglet: https://projectpiglet.com/ https://projectpiglet.com/ Long story short, all "research" is pretty much B.S. I used to assume this is because people want to make money, so they keep the good analysis secret. However, after working in the industry a few years; it's mostly because they just don't know how to apply the algorithms or if it's even possible. I think my favorite example is the seminal paper on using twitter sentiment to predict stock movement[1]. They don't use a large enough data set, and more importantly they use granger causality to identify "casualty"[2] between sentiment and stock value. They then claim they found a specific range which has a p-value indicating they are correlated... Of course you'll find a correlation when you look at two normalized signals and try to match them up. Now, if they had not use the DJIA (Dow Jones Industrial Average) and instead used 500 individual stocks, and found the sentiment on twitter correlated with stock value(s) 90% of the time between 5 and 10 days. I'd argue they probably have something. However, because their method is literally a BFS on only two signals in an attempt to find a correlation, they must correct for the p-value. i.e. "look and you shall find"[3] This is just one of the hundred issues I've found, but really sheds light on how bad that industry is. [1] https://scholar.google.com/scholar?hl=en&q=twitter+stock+sentiment+analysis&btnG=&as_sdt=1%2C47&as_sdtp=&oq=twitter https://scholar.google.com/scholar?hl=en&q=twitter+stock+sen... [2] Granger Causality offsets two signals in an attempt to find a correlation between them at an offset between two times. AKA find "causality" by finding correlation at an offset in time [3] https://stats.stackexchange.com/questions/5750/look-and-you-shall-find-a-correlation https://stats.stackexchange.com/questions/5750/look-and-you-...
- mayank 9y ago
  > Long story short, all "research" is pretty much B.S. Easy there! When I start feeling cynical about things like almost every sentiment analysis paper, I go back and read some of my favorite papers, especially ones from the 70s and 80s (eg, [1]) Another useful heuristic is to read the best paper award winners from conferences in a particular area. It's not a perfect metric, but the signal/noise ratio is better. [1] the flajolet-martin algorithm: http://algo.inria.fr/flajolet/Publications/FlMa85.pdf http://algo.inria.fr/flajolet/Publications/FlMa85.pdf

295 points by bilifuduo 9y ago

zacharydavid 9y ago

Sorry guys. Traffic killed the site. Booting up a new server

rubatuga 9y ago
Do you have a cached copy?
- zacharydavid 9y ago
  En route
zacharydavid 9y ago
it's up, propagation should take a minute

zacharydavid 9y ago

Special thanks to Nickolas Younker (at LiquidWeb) for saving my behind and getting this all set up.

arjie 9y ago
Doesn't load for me, mate. Also, if you put your email in your profile, I can send you an email instead of posting a comment here.
- arthurcolle 9y ago
  Same here. Unfortunate...
  - zacharydavid 9y ago
    Sorry you two are still having issues. I had to do a new incognito window to get to it
nyounker 9y ago
I was happy to help. Migrations are very complicated especially when in the face of massive traffic to move from one server to another and keep it live. I apologize that the dns transition wasnt seemless, but I managed a full migration in under 30 minutes. I'm glad I was able to help.
nyounker 9y ago
I was happy to help. Migrations are very complicated especially when in the face of massive traffic to move from one server to another and keep it live. I apologize that the dns transition wasnt seemless, but I managed a full migration in under 30 minutes. I'm glad I was able to help.
nyounker 9y ago
I was happy to help. Migrations are very complicated especially when in the face of massive traffic to move from one server to another and keep it live. I apologize that the dns transition wasnt seemless, but I managed a full migration in under 30 minutes. I'm glad I was able to help.

dkural 9y ago

Most academic CS literature is complete BS. The vast majority of papers fit into a simple formula of "We apply method X to problem Y, and outperform other approaches using approaches similar to method X". Meanwhile, no one uses method X or any of its cousins, because in the real world the problem is solved very differently with a combination of both principled algorithms and heuristics derived from real-world datasets. The paper also fails to give any theoretical reason or mathematical insight as to why their version of X is better. Thus, it doesn't actually solve a real world problem OR advance scientific understanding.

jfoutz 9y ago
Could an enterprising person find such papers, cleverly select a dataset such that X is way worse, and finally refute the original paper? This seems like something an amateur could do. Wouldn't that tear down a bunch of this junk?
- shimon 9y ago
  Sure. But for approximately the same level of effort, you could generate a new paper in the same template. It would likely be easier to get that "novel result" published than the refutation.
  - beagle3 9y ago
    And that's generally the truth in every scientific field. Right now you have a "reproduction renaissance" in psychology and cancer, where you can get published, and get acknowledged, if you tear down a high level paper. But in general, unless you aim at refuting the highest level, it is hard to get published and advances your career nowhere, so people have no incentive to do it.
    - nonbel 9y ago
      A non-replication does not "tear down" a paper... It is not any kind of "attack", your attitude is one of the sources of the problem. Edit: When I did medical research I wished people would try to replicate my findings so I could see how consistent the results were and how well my model fit future data... In fact it was really annoying that it was unlikely anyone would ever try to do a direct replication.

murbard2 9y ago

I got into quant finance 12 years ago with the mistaken idea that I was going to successfully use all these cool machine learning techniques (genetic programming! SVMs! neural networks!) to run great statistical arbitrage books. Most machine learning techniques focus on problems where the signal is very strong, but the structure is very complex. For instance, take the problem of recognizing whether a picture is a picture of a bird. A human will do well on this task, which shows that there is very little intrinsic noise. However, the correlation of any given pixel with the class of the image is essentially 0. The "noise" is in discovering the unknown relationship between pixels and class, not in the actual output. Noise dominates everything you will find in statistical arbitrage. R^2 of 1% are something to write home about. With this amount of noise, it's generally hard to do much better than a linear regression. Any model complexity has to come from integrating over latent parameters or manual feature engineering, the rest will overfit. I think Geoffrey Hinton said that statistics and machine learning are really the same thing, but since we have two different names for it, we might as well call machine learning everything that focuses on dealing with problems with a complex structure and low noise, and statistics everything that focuses on dealing with problems with a large amount of noise. I like this distinction, and I did end up picking up a lot of statistics working in this field. I'll regularly get emails from friends who tried some machine learning technique on some dataset and found promising results. As the article points out, these generally don't hold up. Accounting for every source of bias in a backtest is an art. The most common mistake is to assume that you can observe the relative price of two stocks at the close, and trade at that price. Many pairs trading strategies appear to work if you make this assumption (which tends to be the case if all you have are daily bars), but they really do not. Others include: assuming transaction costs will be the same on average (they won't, your strategy likely detects opportunities at time were the spread is very large and prices are bad), assuming index memberships don't change (they do and that creates selection bias), assuming you can short anything (stocks can be hard to short or have high borrowing costs), etc. In general, statistical arbitrage isn't machine learning bound(1), and it is not a data mining endeavor. Understanding the latent market dynamics you are trying to capitalize on, finding new data feeds that provide valuable information, carefully building out a model to test your hypothesis, deriving a sound trading strategy from that model is how it works. (1: this isn't always true. For instance, analyzing news with NLP, or using computer vision to estimate crop outputs from satellite imagery can make use of machine learning techniques to yield useful, tradeable signals. My comment mostly focuses on machine learning applied to price information. )

dogruck 9y ago

Would be nice to see a standard academic platform for backtesting. Then, the paper could say "we submitted our implementation of this strategy to Backtest (which includes transaction costs and slippage)."

sgt101 9y ago
I think the pymc3 people have made something available

chvid 9y ago

Does anyone know of any paper that describes a reproducible method of generating above normal returns in the mature western financial markets? Nope. Me neither.

proofofstake 9y ago
Do you consider cryptocurrency mature?
- chvid 9y ago
  Nope. But even for those types of market I doubt that there exists an academic paper describing a reproducible trading strategy/method with above normal returns.
  - proofofstake 9y ago
    https://pdfs.semanticscholar.org/db15/1836543d8a70db1dabef3dee43637a7cd29f.pdf https://pdfs.semanticscholar.org/db15/1836543d8a70db1dabef3d... "Bayesian regression and Bitcoin" > The strategy is able to nearly double the investment in less than 60 day period when run against real data trace. http://cs229.stanford.edu/proj2015/029_report.pdf http://cs229.stanford.edu/proj2015/029_report.pdf "Algorithmic Trading of Cryptocurrency Based on Twitter Sentiment Analysis" http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0177630 http://journals.plos.org/plosone/article?id=10.1371/journal.... "When Bitcoin encounters information in an online forum: Using text mining to analyse user opinions and predict value fluctuation" https://pdfs.semanticscholar.org/e065/3631b4a476abf5276a264f6bbff40b132061.pdf https://pdfs.semanticscholar.org/e065/3631b4a476abf5276a264f... "Automated Bitcoin Trading via Machine Learning Algorithms"
    - chvid 9y ago
      "Based on this price prediction method, we devise a simple strategy for trading Bitcoin. The strategy is able to nearly double the investment in less than 60 day period when run against real data trace." From the paper "Bayesian regression and Bitcoin". I think are more relevant question is: This is clearly not the case in the real world - so why does it appear to be like that? But thanks for the links; I will enjoy reading through them.

lettergram 9y ago

I can't up vote this enough.. I've spend an unreasonable amount of time reading research related to finance for my web app Piglet: https://projectpiglet.com/ https://projectpiglet.com/ Long story short, all "research" is pretty much B.S. I used to assume this is because people want to make money, so they keep the good analysis secret. However, after working in the industry a few years; it's mostly because they just don't know how to apply the algorithms or if it's even possible. I think my favorite example is the seminal paper on using twitter sentiment to predict stock movement[1]. They don't use a large enough data set, and more importantly they use granger causality to identify "casualty"[2] between sentiment and stock value. They then claim they found a specific range which has a p-value indicating they are correlated... Of course you'll find a correlation when you look at two normalized signals and try to match them up. Now, if they had not use the DJIA (Dow Jones Industrial Average) and instead used 500 individual stocks, and found the sentiment on twitter correlated with stock value(s) 90% of the time between 5 and 10 days. I'd argue they probably have something. However, because their method is literally a BFS on only two signals in an attempt to find a correlation, they must correct for the p-value. i.e. "look and you shall find"[3] This is just one of the hundred issues I've found, but really sheds light on how bad that industry is. [1] https://scholar.google.com/scholar?hl=en&q=twitter+stock+sentiment+analysis&btnG=&as_sdt=1%2C47&as_sdtp=&oq=twitter https://scholar.google.com/scholar?hl=en&q=twitter+stock+sen... [2] Granger Causality offsets two signals in an attempt to find a correlation between them at an offset between two times. AKA find "causality" by finding correlation at an offset in time [3] https://stats.stackexchange.com/questions/5750/look-and-you-shall-find-a-correlation https://stats.stackexchange.com/questions/5750/look-and-you-...

mayank 9y ago
> Long story short, all "research" is pretty much B.S. Easy there! When I start feeling cynical about things like almost every sentiment analysis paper, I go back and read some of my favorite papers, especially ones from the 70s and 80s (eg, [1]) Another useful heuristic is to read the best paper award winners from conferences in a particular area. It's not a perfect metric, but the signal/noise ratio is better. [1] the flajolet-martin algorithm: http://algo.inria.fr/flajolet/Publications/FlMa85.pdf http://algo.inria.fr/flajolet/Publications/FlMa85.pdf