13 ms·
I've worked in quant funds for over a decade. The article is right that there's a lack of rigour, and it's true the number of trials is an issue. However it doe
by lordnacho 11y ago
I've worked in quant funds for over a decade. The article is right that there's a lack of rigour, and it's true the number of trials is an issue. However it doesn't go into why it happens:
- People with phds often have an authority that is unwarranted. People with advanced degrees in similar sounding fields may have spent their time doing very different things. Phds are also coming from an environment where getting credit matters a lot, so you get political struggles that appear to be mathematical in nature to the uninformed manager.
- Programming skills tended to be crap in the places where I worked. Most of the researchers think of coding as a necessary evil, and their code looks like the kind of code you write when you're trying to take shortcuts. What effect does this have on overfit? Well, it's simply impossible to know how many tests are actually being performed. There's a lot of informative errors, too: something gets coded up, you see a backtest, you realise there was an error after. Now you have information that you can use to get a better result, having not incremented your number of tests.
- There's a lot of pressure to come up with something new. Pretty hard when there's loads of people attempting the same. The most natural way to try to make something "new" is to take something old, add a twist to it, and see how it goes. Of course when you add a twist, there's more parameters to fit. And because it's supposed to be complex, you can get away with an elaborate fitting mechanism. Often something that hides the fact you're doing more tests.
- Backtest fetish: it's natural to think about how a system would have done in the past. But unfortunately we all know about the past. Anyone coming up with a strategy is going to know volatility spiked in 2008, so he'll avoid anything that would have lost money then, or worse, just restrict the parameters so they are in a safe zone. Basically, there's a bunch of tests that aren't done, but should have been counted anyway.
- How to fix this: well, there's no magic bullet. You should start by being wary of adding parameters. If you add one, it should work with a number of reaction functions. If it works with a step func, does it work with an s-shape or a ramp? Does it work when the parameters are perturbed? Does it work when the input data itself is perturbed? If you've built a model, does it work on simulated data? Something I rarely see is when someone says "model is an ARMA(2,2)" that they've generated an ARMA22 with the same parameters.