4 ms·

One of Quantopian's studies shows that backtesting poorly predicts live trading results: https://blog.quantopian.com/using-machine-learning-to-predict-out-of-s

by nether 9y ago

One of Quantopian's studies shows that backtesting poorly predicts live trading results: https://blog.quantopian.com/using-machine-learning-to-predict-out-of-sample-performance-of-trading-algorithms/ https://blog.quantopian.com/using-machine-learning-to-predic.... From the forums, it looks like 90% of the effort is devoted to getting a good backtest, a yardstick which might not have any bearing on reality. How do real quant traders deal with this discrepancy?

hodder 9y ago
-Out of sample testing. Across markets, across time. -Logical economic explanations.
akrymski 9y ago
Real firms use tick data to backtest, not the 1 minute bars that Quantopian uses, and spend a lot of time simulating network characteristics such as data and order latency. They also use machine learning, which isn't possible on Quantopian, to build models, which requires downloading lots of data (and not just equities). No serious quant will ever use such a tool. You might as well go to the casino.
- akrymski 9y ago
  Edit: tick data is usually collected over time, because data sources have different characteristics, and you should be testing over the same realtime data that you'll be using to trade live. You need to know the bid/ask prices and volumes, in order to know where to place limit orders. Otherwise you are just paying commissions and spreads to brokers and market makers.
- deleted 9y ago
  [deleted]
mabbo 9y ago
> How do real quant traders deal with this discrepancy? A lot of it comes down to access to information and speed. All the major traders and quants have access to what effectively boil down to mailing lists (that their companies pay top dollar for) that have the best analysts in the world producing in-depth analysis on various topics[0]. Also, many/most have access to a Bloomberg terminal[1] to get the Bloomberg new feeds and the AIM clone, which costs $25,000/year/person. This is why amateur day-trading is silly. You're not competing in a level playing field- they know more than you do. [0]http://www.investopedia.com/articles/analyst/03/031803.asp http://www.investopedia.com/articles/analyst/03/031803.asp [1]https://en.wikipedia.org/wiki/Bloomberg_Terminal https://en.wikipedia.org/wiki/Bloomberg_Terminal
- deegles 9y ago
  Sounds like I should get into building tools for traders!
  - mabbo 9y ago
    It's a good idea. The user base is a group of people for whom hundreds of dollars per month is peanuts if it gives them a 1% edge over their competitors.
arkis22 9y ago
There was a great article in the WSJ about quant "factors" >A former student of Mr. Fama, Cliff Asness, founder of quantitative hedge-fund manager AQR Capital Management, said he tries to avoid being caught out by false findings by trading on anomalies he can explain, economically or through investor behavior. To assess whether the market anomalies will continue, he looks for ones which carried on after being identified, can be seen in other markets or asset classes, and where minor changes to how they are defined don’t much affect the result. These include most famously value, momentum and corporate quality, among others. https://www.wsj.com/articles/an-algorithm-an-etf-and-an-academic-study-walk-into-a-bar-1494528113 https://www.wsj.com/articles/an-algorithm-an-etf-and-an-acad...
- thablackbull 9y ago
  He also blogs for those interested - obviously geared more to the finance types: https://www.aqr.com/cliffs-perspective https://www.aqr.com/cliffs-perspective
dia80 9y ago
A Bayesian mindset. When investigating a signal you have to ask: Is it too good to be true? Does it make economic sense? Then simulate rolling optimisation and trading to see what returns could have been achieved with knowledge only available at the time.
dnadler 9y ago
We are very mindful of overfitting our models, and very skeptical of backtests on their own. Successful strategies often have some fundamental theory about why they should work. If not, the backtest needs to be very robust, and demonstrate that it is not overfit by cross-validation, and other methods.
deleted 9y ago
[deleted]
mmargenot 9y ago
You're right, backtest overfitting is a problem. At Quantopian, we spend a lot of time on education helping people avoid overfitting like this (https://www.quantopian.com/lectures#The-Dangers-of-Overfitting https://www.quantopian.com/lectures#The-Dangers-of-Overfitti...). It is easy to make a backtest look good by tweaking things here or there and that's just textbook overfitting. If you follow good statistical principles for model development based on a sound economic basis, the backtest is more of a sanity check to make sure that whatever signal you have found survives the introduction of market impact. If you have consistency between the statistical aspects of in-sample performance (your backtest) and true out-of-sample performance (paper trading on entirely new data), there as an increased probability that you have found a viable signal. Disclaimer, I work at Quantopian.

by nether 9y ago

hodder 9y ago

-Out of sample testing. Across markets, across time. -Logical economic explanations.

akrymski 9y ago

Real firms use tick data to backtest, not the 1 minute bars that Quantopian uses, and spend a lot of time simulating network characteristics such as data and order latency. They also use machine learning, which isn't possible on Quantopian, to build models, which requires downloading lots of data (and not just equities). No serious quant will ever use such a tool. You might as well go to the casino.

akrymski 9y ago
Edit: tick data is usually collected over time, because data sources have different characteristics, and you should be testing over the same realtime data that you'll be using to trade live. You need to know the bid/ask prices and volumes, in order to know where to place limit orders. Otherwise you are just paying commissions and spreads to brokers and market makers.
deleted 9y ago
[deleted]

mabbo 9y ago

> How do real quant traders deal with this discrepancy? A lot of it comes down to access to information and speed. All the major traders and quants have access to what effectively boil down to mailing lists (that their companies pay top dollar for) that have the best analysts in the world producing in-depth analysis on various topics[0]. Also, many/most have access to a Bloomberg terminal[1] to get the Bloomberg new feeds and the AIM clone, which costs $25,000/year/person. This is why amateur day-trading is silly. You're not competing in a level playing field- they know more than you do. [0]http://www.investopedia.com/articles/analyst/03/031803.asp http://www.investopedia.com/articles/analyst/03/031803.asp [1]https://en.wikipedia.org/wiki/Bloomberg_Terminal https://en.wikipedia.org/wiki/Bloomberg_Terminal

deegles 9y ago
Sounds like I should get into building tools for traders!
- mabbo 9y ago
  It's a good idea. The user base is a group of people for whom hundreds of dollars per month is peanuts if it gives them a 1% edge over their competitors.

arkis22 9y ago

There was a great article in the WSJ about quant "factors" >A former student of Mr. Fama, Cliff Asness, founder of quantitative hedge-fund manager AQR Capital Management, said he tries to avoid being caught out by false findings by trading on anomalies he can explain, economically or through investor behavior. To assess whether the market anomalies will continue, he looks for ones which carried on after being identified, can be seen in other markets or asset classes, and where minor changes to how they are defined don’t much affect the result. These include most famously value, momentum and corporate quality, among others. https://www.wsj.com/articles/an-algorithm-an-etf-and-an-academic-study-walk-into-a-bar-1494528113 https://www.wsj.com/articles/an-algorithm-an-etf-and-an-acad...

thablackbull 9y ago
He also blogs for those interested - obviously geared more to the finance types: https://www.aqr.com/cliffs-perspective https://www.aqr.com/cliffs-perspective

dia80 9y ago

A Bayesian mindset. When investigating a signal you have to ask: Is it too good to be true? Does it make economic sense? Then simulate rolling optimisation and trading to see what returns could have been achieved with knowledge only available at the time.

dnadler 9y ago

We are very mindful of overfitting our models, and very skeptical of backtests on their own. Successful strategies often have some fundamental theory about why they should work. If not, the backtest needs to be very robust, and demonstrate that it is not overfit by cross-validation, and other methods.

deleted 9y ago

[deleted]

mmargenot 9y ago

You're right, backtest overfitting is a problem. At Quantopian, we spend a lot of time on education helping people avoid overfitting like this (https://www.quantopian.com/lectures#The-Dangers-of-Overfitting https://www.quantopian.com/lectures#The-Dangers-of-Overfitti...). It is easy to make a backtest look good by tweaking things here or there and that's just textbook overfitting. If you follow good statistical principles for model development based on a sound economic basis, the backtest is more of a sanity check to make sure that whatever signal you have found survives the introduction of market impact. If you have consistency between the statistical aspects of in-sample performance (your backtest) and true out-of-sample performance (paper trading on entirely new data), there as an increased probability that you have found a viable signal. Disclaimer, I work at Quantopian.