4 ms·

Financial Backtesting: A Cautionary Tale

51 points by kawera 11y ago

canttestthis 11y ago
If you were an investor in 1985 and you did this analysis and ran out-of-sample tests, you would have 15 years of profitability before the algorithm starts generating losses. After a few months of losses you would discard the algorithm and search for a new one. What am I missing?
- lintiness 11y ago
  why would you imagine a few months or losses indicated the strategy didn't work anymore?
  - p4wnc6 11y ago
    Exactly. This is a form of counterfactual bias. It's easy to imagine how you were "almost right" and give yourself credit for that, but it's much harder to imagine when you were "almost wrong" and correctly penalize your thinking in proportion to the almost-wrong-ness. It's easy to imagine you would have turned off the strategy at a convenient time, but it's hard to imagine that you would maybe rationalize that it's just a short slump or something and keep the strategy going. And for any hard rule you make like a priori committing to turning off some strategy if there are X consecutive months of bad returns (for reasonably small X), you can construct some counter-example strategy where, if you had just had the foresight to hold on to month X+1, you would have made a ton of money and there was some after-the-fact obvious reason why months 1 through X had the poor return, and "smart" investors would have understood this and...
  - canttestthis 11y ago
    Because the strategy was almost consistently profitable before 2000. So if I entered the algorithm with the assumption that x months of no profit = shut it down, or y% loss = shut it down, I wouldn't be sitting there watching my investment go down 71%.
  - lordnacho 11y ago
    As someone who worked at a shop where this actually happened, I can confirm the strategy most likely would not be shut down. In fact it ended up being run sideways for longer than it made money. Naturally this is due to the manager really depending on the income from having the assets.
chollida1 11y ago
The biggest problem with back-testing in finance that I've come across is that markets change. Nowadays markets change every 5-10 years,this is my own personal belief here, which means you can't do too many meaningful comparisons over longer time periods. Couple that with the fact that most bull markets last for 5-10 years and this means you can't really tell how your active portfolio will perform. Or put another way, strategies that work on a macro level tend not to work very well. Anything of the flavor of "if the market does X for past 5 days, then buy Y and hold until Z happens tend not to be reproducible or testable in a meaningful way." What does work, again from personal experience, is arbitrage. Arbitrage is everywhere in the financial markets. - 5 year bonds that are 2 years old look just like 3 year bonds that were just issued, assuming all terms are the same. - The A and B shares of a single company tend to follow a constant pattern - 2 different ETF's that follow the same index(say SPY) tend to move together, - exchanges that tend to cross at midpoint( most dark pools) and price via the SIP tend to offer a half penny latency arbitrage opportunities - when two companies merge the price of the target will converge on the deal price at the closing. - low latency arb where stock prices follow some commodities price, banks vs currency rates, oil produces vs WTI, etc These are all examples of things that you can meaningfully back-test and model. Jim Simmons of Renaissance Capital once remarked that there is no signal in the market data at a macro level, I tend to agree with him and the author.
- p4wnc6 11y ago
  The way I understand this is that a lot of it comes down to articulating useful prior distributions over some of the outcomes to be predicted. The kind of backtesting that this post explores, which is basically the bread and butter of "quantamental" asset managers (lolz CRSP Fama/French data), is almost always performed in a frequentist set up where you plan to regress outcomes (portfolio returns) on a simple set of "easy to understand" predictors, like price momentum over some trailing time period. You might have to do a few cross-sectional or time-series adjustments, like the Fama/MacBeth regression and some poor man's time series smoothing of coefficients, but it's basically just a really large, really crappy OLS model, where the plan is to just torture the vendor data sets until you get out whatever kind of statistical significance you need for a marketing whitepaper and/or pitchbook entry to shop around to unsophisticated board members in client organizations. This is the whole "which 1000 tests did I do before I showed you the results" thing from this article, and it just amounts to simple p-hacking (or t-stat hacking) which is a well-known Bad Thing To Do. Clients are already very sensitive to things like transaction costs, so the clients probably would have demanded that the analysis be carried out net of transaction costs in the first place. But transaction costs are easy for a paying client to get their head around... for non-quantitative client board members, getting their head around nuances of p-values and regression models is actually quite hard, even if these are statistics 101 for the finance professionals working with them. But the amazing thing is that while everyone's busy worrying about p-hacking (and never Bonferroni-correcting reported results, if they even know how many researcher degrees of freedom [1] are involved in the first place), no one is talking about the fact that all of this is in the service of a frequentist-style analysis of Prob(Data | market hypothesis), rather than a Bayesian-style policy question that seeks to analyze Prob(market hypothesis | Data) and necessitates some use of expert-derived priors about the probability of some market hypothesis in general. In the case of a momentum strategy like the one studied, the market hypothesis would need to be about modern market conditions, the causal underpinnings of price momentum as a predictive factor, and whether or not its efficacy will improve or decrease when you condition on the modern market situation. And crucially, this is the part that active managers are (like everyone else) very bad at. They can't articulate accurate or informative priors about the current status of something like the causal underpinnings of price momentum any better than anyone else. If you are seeking to pay someone for a policy analysis, e.g. what should I think about the posterior probability Prob(market hypothesis | Data), then you shouldn't be paying them for the easy part of that analysis -- the part where you write down some silly likelihood function due to heroic OLS assumptions and optimize it for parameters and update a portfolio position based on those parameters. That part is borderline trivial if you have decent software engineers to make sure the code is not excessively buggy (yet most quantamental shops don't, and they're still letting portfolio managers do this shit directly with SLOPE in Excel and then making Ivy MBAs translate it into poor Python or MATLAB code for "production") -- the hard part is actually articulating informative priors. I think overall, it higlights a need for client organizations to become a bit more sophisticated about the sorts of forecasting models that quantamental managers use. Then they can begin applying market pressure by firing managers that do bang-simple things like OLS in a crappy dev environment, and hopefully create competitive pressure to make quant shops shift to more rigorous Bayesian methods, or at least make the rampant researcher degrees of freedom and p-hacking problems transparent so they can be mitigated. Of course, a lot of this is moot because a very large amount of the business done with quantamental asset managers is based on nepotism and cover-your-ass blame insurance. For example, suppose you are a non-technical board member of the entity charged with managing the pension plans for some state's retired firefighters. You are paid money not just based on whether the pension plan's assets improve, but also on whether you did due diligence when hiring an active manager (read: you hired a fancy consulting firm who told you what you wanted to hear), and whether you are in a position to fire firms if they seem to do a bad job, or create pockets of plausible deniability by crafting "data journalism" accounts about a market downturn that "no one could have avoided." Because these board members are the ones with most of the interaction with quantamental shops, most quantamental shops end up turning into shoddy data journalism mills -- need a story to explain yesterday's bad returns to the rest of your board? We're on it. Whadya want? Something about oil & natural gas? Something data mined about some random earnings reports? Just let us know. We're your one stop shop to rationalizing away yesterday's returns. Just don't ask us what will happen tomorrow. In this sense, clients rarely punish quantamental shops. In fact, many of them are handsomely rewarded for doing p-hacking and shoddy data journalism as long as it covers the ass of some client board member who doesn't want to lose his bonus because of a downturn in the S&P 500 and needs to sell a story to his bosses fast. If that kind of stuff doesn't go away, there will be no actual prediction-based incentive for the asset managers themselves to ever improve. Why work hard to write good code and do proper statistics when you can make bank by selling shoddy data journalism? [1] http://andrewgelman.com/2012/11/01/researcher-degrees-of-freedom/ http://andrewgelman.com/2012/11/01/researcher-degrees-of-fre...
lordnacho 11y ago
Well written article. I kinda fear for some people I used to work with. They've got a "model" that turns over the portfolio 1.5 times a year, backtested since around 2000. Sometimes it's hard to get people to understand that what they're doing is totally crazy. What the article is hinting at is a very specific part of the scientific method: mechanism. When you hypothesise about something, you are also guessing at how exactly the effect comes to pass. So for instance you surmise that incident radiation induces a current in conductors, lessening the amount of radiation beyond the point of incidence. This mechanism would have some interesting experimental results, namely that non conductors would not shield anything, whereas conductors can be used to shield say a phone from EM radiation. It also means if your shield was conducting but very thin, it might not work. In the financial sphere, you rarely have the luxury of being able to test mechanisms. There's a huge amount of data, and it's quite easy to find structure, but the apparatus itself is rarely exposed. This leaves you with two areas of research: arbitrage, as in what Chollida1 says, and presumed mechanisms. Since the first area has been explored by him, I'll look at the second. So let's say you used to work in a large pension fund, and that pension fund always did its trades starting at 3pm, and filled whatever wasn't done in the auction. You meet some colleagues, who do the same. Since you now have a glimpse into a causal mechanism, it's pretty reasonable for you to look at a model where things that have traded a lot since 3pm continue in the way they were going. There could be lots of similar mechanisms in play, coming on and off as the market evolved.
1812Overture 11y ago
"Markets can remain irrational longer than you can remain solvent." - John Maynard Keynes
Nomentatus 11y ago
Gotta agree with Jerf - what happened seems obvious. Someone else got there first - and we know pretty much when - and by now their algorithm includes sophisticated filters and refinements. They're pulling in only the best low hanging fruit now and competing for it, too - by buys near the end of the day, say. Margins are much lower now and the stated crude algorithm can no longer compete. My first paper-and-pencil investment strategy in stocks was precisely this momentum one, when I was about 14, around 1970. So, this wasn't an opportunity that was going to persist forever. (What made it non-viable in the past was the high cost of broker fees.)
mcguire 11y ago
Does anyone have current performance data for the Dogs of the Dow or Motley Fool Foolish Four (?) strategies?

51 points by kawera 11y ago

If you were an investor in 1985 and you did this analysis and ran out-of-sample tests, you would have 15 years of profitability before the algorithm starts generating losses. After a few months of losses you would discard the algorithm and search for a new one. What am I missing?

lintiness 11y ago
why would you imagine a few months or losses indicated the strategy didn't work anymore?
- p4wnc6 11y ago
  Exactly. This is a form of counterfactual bias. It's easy to imagine how you were "almost right" and give yourself credit for that, but it's much harder to imagine when you were "almost wrong" and correctly penalize your thinking in proportion to the almost-wrong-ness. It's easy to imagine you would have turned off the strategy at a convenient time, but it's hard to imagine that you would maybe rationalize that it's just a short slump or something and keep the strategy going. And for any hard rule you make like a priori committing to turning off some strategy if there are X consecutive months of bad returns (for reasonably small X), you can construct some counter-example strategy where, if you had just had the foresight to hold on to month X+1, you would have made a ton of money and there was some after-the-fact obvious reason why months 1 through X had the poor return, and "smart" investors would have understood this and...
- canttestthis 11y ago
  Because the strategy was almost consistently profitable before 2000. So if I entered the algorithm with the assumption that x months of no profit = shut it down, or y% loss = shut it down, I wouldn't be sitting there watching my investment go down 71%.
- lordnacho 11y ago
  As someone who worked at a shop where this actually happened, I can confirm the strategy most likely would not be shut down. In fact it ended up being run sideways for longer than it made money. Naturally this is due to the manager really depending on the income from having the assets.

chollida1 11y ago

The biggest problem with back-testing in finance that I've come across is that markets change. Nowadays markets change every 5-10 years,this is my own personal belief here, which means you can't do too many meaningful comparisons over longer time periods. Couple that with the fact that most bull markets last for 5-10 years and this means you can't really tell how your active portfolio will perform. Or put another way, strategies that work on a macro level tend not to work very well. Anything of the flavor of "if the market does X for past 5 days, then buy Y and hold until Z happens tend not to be reproducible or testable in a meaningful way." What does work, again from personal experience, is arbitrage. Arbitrage is everywhere in the financial markets. - 5 year bonds that are 2 years old look just like 3 year bonds that were just issued, assuming all terms are the same. - The A and B shares of a single company tend to follow a constant pattern - 2 different ETF's that follow the same index(say SPY) tend to move together, - exchanges that tend to cross at midpoint( most dark pools) and price via the SIP tend to offer a half penny latency arbitrage opportunities - when two companies merge the price of the target will converge on the deal price at the closing. - low latency arb where stock prices follow some commodities price, banks vs currency rates, oil produces vs WTI, etc These are all examples of things that you can meaningfully back-test and model. Jim Simmons of Renaissance Capital once remarked that there is no signal in the market data at a macro level, I tend to agree with him and the author.

p4wnc6 11y ago
The way I understand this is that a lot of it comes down to articulating useful prior distributions over some of the outcomes to be predicted. The kind of backtesting that this post explores, which is basically the bread and butter of "quantamental" asset managers (lolz CRSP Fama/French data), is almost always performed in a frequentist set up where you plan to regress outcomes (portfolio returns) on a simple set of "easy to understand" predictors, like price momentum over some trailing time period. You might have to do a few cross-sectional or time-series adjustments, like the Fama/MacBeth regression and some poor man's time series smoothing of coefficients, but it's basically just a really large, really crappy OLS model, where the plan is to just torture the vendor data sets until you get out whatever kind of statistical significance you need for a marketing whitepaper and/or pitchbook entry to shop around to unsophisticated board members in client organizations. This is the whole "which 1000 tests did I do before I showed you the results" thing from this article, and it just amounts to simple p-hacking (or t-stat hacking) which is a well-known Bad Thing To Do. Clients are already very sensitive to things like transaction costs, so the clients probably would have demanded that the analysis be carried out net of transaction costs in the first place. But transaction costs are easy for a paying client to get their head around... for non-quantitative client board members, getting their head around nuances of p-values and regression models is actually quite hard, even if these are statistics 101 for the finance professionals working with them. But the amazing thing is that while everyone's busy worrying about p-hacking (and never Bonferroni-correcting reported results, if they even know how many researcher degrees of freedom [1] are involved in the first place), no one is talking about the fact that all of this is in the service of a frequentist-style analysis of Prob(Data | market hypothesis), rather than a Bayesian-style policy question that seeks to analyze Prob(market hypothesis | Data) and necessitates some use of expert-derived priors about the probability of some market hypothesis in general. In the case of a momentum strategy like the one studied, the market hypothesis would need to be about modern market conditions, the causal underpinnings of price momentum as a predictive factor, and whether or not its efficacy will improve or decrease when you condition on the modern market situation. And crucially, this is the part that active managers are (like everyone else) very bad at. They can't articulate accurate or informative priors about the current status of something like the causal underpinnings of price momentum any better than anyone else. If you are seeking to pay someone for a policy analysis, e.g. what should I think about the posterior probability Prob(market hypothesis | Data), then you shouldn't be paying them for the easy part of that analysis -- the part where you write down some silly likelihood function due to heroic OLS assumptions and optimize it for parameters and update a portfolio position based on those parameters. That part is borderline trivial if you have decent software engineers to make sure the code is not excessively buggy (yet most quantamental shops don't, and they're still letting portfolio managers do this shit directly with SLOPE in Excel and then making Ivy MBAs translate it into poor Python or MATLAB code for "production") -- the hard part is actually articulating informative priors. I think overall, it higlights a need for client organizations to become a bit more sophisticated about the sorts of forecasting models that quantamental managers use. Then they can begin applying market pressure by firing managers that do bang-simple things like OLS in a crappy dev environment, and hopefully create competitive pressure to make quant shops shift to more rigorous Bayesian methods, or at least make the rampant researcher degrees of freedom and p-hacking problems transparent so they can be mitigated. Of course, a lot of this is moot because a very large amount of the business done with quantamental asset managers is based on nepotism and cover-your-ass blame insurance. For example, suppose you are a non-technical board member of the entity charged with managing the pension plans for some state's retired firefighters. You are paid money not just based on whether the pension plan's assets improve, but also on whether you did due diligence when hiring an active manager (read: you hired a fancy consulting firm who told you what you wanted to hear), and whether you are in a position to fire firms if they seem to do a bad job, or create pockets of plausible deniability by crafting "data journalism" accounts about a market downturn that "no one could have avoided." Because these board members are the ones with most of the interaction with quantamental shops, most quantamental shops end up turning into shoddy data journalism mills -- need a story to explain yesterday's bad returns to the rest of your board? We're on it. Whadya want? Something about oil & natural gas? Something data mined about some random earnings reports? Just let us know. We're your one stop shop to rationalizing away yesterday's returns. Just don't ask us what will happen tomorrow. In this sense, clients rarely punish quantamental shops. In fact, many of them are handsomely rewarded for doing p-hacking and shoddy data journalism as long as it covers the ass of some client board member who doesn't want to lose his bonus because of a downturn in the S&P 500 and needs to sell a story to his bosses fast. If that kind of stuff doesn't go away, there will be no actual prediction-based incentive for the asset managers themselves to ever improve. Why work hard to write good code and do proper statistics when you can make bank by selling shoddy data journalism? [1] http://andrewgelman.com/2012/11/01/researcher-degrees-of-freedom/ http://andrewgelman.com/2012/11/01/researcher-degrees-of-fre...

lordnacho 11y ago

Well written article. I kinda fear for some people I used to work with. They've got a "model" that turns over the portfolio 1.5 times a year, backtested since around 2000. Sometimes it's hard to get people to understand that what they're doing is totally crazy. What the article is hinting at is a very specific part of the scientific method: mechanism. When you hypothesise about something, you are also guessing at how exactly the effect comes to pass. So for instance you surmise that incident radiation induces a current in conductors, lessening the amount of radiation beyond the point of incidence. This mechanism would have some interesting experimental results, namely that non conductors would not shield anything, whereas conductors can be used to shield say a phone from EM radiation. It also means if your shield was conducting but very thin, it might not work. In the financial sphere, you rarely have the luxury of being able to test mechanisms. There's a huge amount of data, and it's quite easy to find structure, but the apparatus itself is rarely exposed. This leaves you with two areas of research: arbitrage, as in what Chollida1 says, and presumed mechanisms. Since the first area has been explored by him, I'll look at the second. So let's say you used to work in a large pension fund, and that pension fund always did its trades starting at 3pm, and filled whatever wasn't done in the auction. You meet some colleagues, who do the same. Since you now have a glimpse into a causal mechanism, it's pretty reasonable for you to look at a model where things that have traded a lot since 3pm continue in the way they were going. There could be lots of similar mechanisms in play, coming on and off as the market evolved.

1812Overture 11y ago

"Markets can remain irrational longer than you can remain solvent." - John Maynard Keynes

Nomentatus 11y ago

Gotta agree with Jerf - what happened seems obvious. Someone else got there first - and we know pretty much when - and by now their algorithm includes sophisticated filters and refinements. They're pulling in only the best low hanging fruit now and competing for it, too - by buys near the end of the day, say. Margins are much lower now and the stated crude algorithm can no longer compete. My first paper-and-pencil investment strategy in stocks was precisely this momentum one, when I was about 14, around 1970. So, this wasn't an opportunity that was going to persist forever. (What made it non-viable in the past was the high cost of broker fees.)

mcguire 11y ago

Does anyone have current performance data for the Dogs of the Dow or Motley Fool Foolish Four (?) strategies?