3 ms·

The way I understand this is that a lot of it comes down to articulating useful prior distributions over some of the outcomes to be predicted. The kind of back

by p4wnc6 11y ago

The way I understand this is that a lot of it comes down to articulating useful prior distributions over some of the outcomes to be predicted. The kind of backtesting that this post explores, which is basically the bread and butter of "quantamental" asset managers (lolz CRSP Fama/French data), is almost always performed in a frequentist set up where you plan to regress outcomes (portfolio returns) on a simple set of "easy to understand" predictors, like price momentum over some trailing time period. You might have to do a few cross-sectional or time-series adjustments, like the Fama/MacBeth regression and some poor man's time series smoothing of coefficients, but it's basically just a really large, really crappy OLS model, where the plan is to just torture the vendor data sets until you get out whatever kind of statistical significance you need for a marketing whitepaper and/or pitchbook entry to shop around to unsophisticated board members in client organizations. This is the whole "which 1000 tests did I do before I showed you the results" thing from this article, and it just amounts to simple p-hacking (or t-stat hacking) which is a well-known Bad Thing To Do. Clients are already very sensitive to things like transaction costs, so the clients probably would have demanded that the analysis be carried out net of transaction costs in the first place. But transaction costs are easy for a paying client to get their head around... for non-quantitative client board members, getting their head around nuances of p-values and regression models is actually quite hard, even if these are statistics 101 for the finance professionals working with them. But the amazing thing is that while everyone's busy worrying about p-hacking (and never Bonferroni-correcting reported results, if they even know how many researcher degrees of freedom [1] are involved in the first place), no one is talking about the fact that all of this is in the service of a frequentist-style analysis of Prob(Data | market hypothesis), rather than a Bayesian-style policy question that seeks to analyze Prob(market hypothesis | Data) and necessitates some use of expert-derived priors about the probability of some market hypothesis in general. In the case of a momentum strategy like the one studied, the market hypothesis would need to be about modern market conditions, the causal underpinnings of price momentum as a predictive factor, and whether or not its efficacy will improve or decrease when you condition on the modern market situation. And crucially, this is the part that active managers are (like everyone else) very bad at. They can't articulate accurate or informative priors about the current status of something like the causal underpinnings of price momentum any better than anyone else. If you are seeking to pay someone for a policy analysis, e.g. what should I think about the posterior probability Prob(market hypothesis | Data), then you shouldn't be paying them for the easy part of that analysis -- the part where you write down some silly likelihood function due to heroic OLS assumptions and optimize it for parameters and update a portfolio position based on those parameters. That part is borderline trivial if you have decent software engineers to make sure the code is not excessively buggy (yet most quantamental shops don't, and they're still letting portfolio managers do this shit directly with SLOPE in Excel and then making Ivy MBAs translate it into poor Python or MATLAB code for "production") -- the hard part is actually articulating informative priors. I think overall, it higlights a need for client organizations to become a bit more sophisticated about the sorts of forecasting models that quantamental managers use. Then they can begin applying market pressure by firing managers that do bang-simple things like OLS in a crappy dev environment, and hopefully create competitive pressure to make quant shops shift to more rigorous Bayesian methods, or at least make the rampant researcher degrees of freedom and p-hacking problems transparent so they can be mitigated. Of course, a lot of this is moot because a very large amount of the business done with quantamental asset managers is based on nepotism and cover-your-ass blame insurance. For example, suppose you are a non-technical board member of the entity charged with managing the pension plans for some state's retired firefighters. You are paid money not just based on whether the pension plan's assets improve, but also on whether you did due diligence when hiring an active manager (read: you hired a fancy consulting firm who told you what you wanted to hear), and whether you are in a position to fire firms if they seem to do a bad job, or create pockets of plausible deniability by crafting "data journalism" accounts about a market downturn that "no one could have avoided." Because these board members are the ones with most of the interaction with quantamental shops, most quantamental shops end up turning into shoddy data journalism mills -- need a story to explain yesterday's bad returns to the rest of your board? We're on it. Whadya want? Something about oil & natural gas? Something data mined about some random earnings reports? Just let us know. We're your one stop shop to rationalizing away yesterday's returns. Just don't ask us what will happen tomorrow. In this sense, clients rarely punish quantamental shops. In fact, many of them are handsomely rewarded for doing p-hacking and shoddy data journalism as long as it covers the ass of some client board member who doesn't want to lose his bonus because of a downturn in the S&P 500 and needs to sell a story to his bosses fast. If that kind of stuff doesn't go away, there will be no actual prediction-based incentive for the asset managers themselves to ever improve. Why work hard to write good code and do proper statistics when you can make bank by selling shoddy data journalism? [1] http://andrewgelman.com/2012/11/01/researcher-degrees-of-freedom/ http://andrewgelman.com/2012/11/01/researcher-degrees-of-fre...

xixi77 11y ago
You make it sound like Bayesian-type analysis is somehow fundamentally impervious to p-hacking and is therefore clearly superior to frequentist for this purpose; do you have any actual references in support of this -- because I am personally having serious doubts about that, in particular given how bad people are at coming up with priors (and when they do, wouldn't these priors be usually backward-looking anyway?)
- p4wnc6 11y ago
  No statistical method is impervious to malicious attempts to subvert it. That said, reasoning between two different models on the basis of magnitude of t-stat is inherently flawed. That is, such an inference process doesn't possess the properties necessary to lead to the conclusions people try to draw from it, even if no malicious intent is present. When you add this to a situation where the temptation is high for things like p-hacking, it actively encourages poor statistical hygiene. Whereas, with Bayesian statistics, you'd have to go out of your way and do something that is glaringly fishy to do the same thing. Hiding it from people is generally a lot harder than hiding what you're doing with generic p-hacking. One of the main reasons for this is that the model's reliance upon assumptions is usually far more transparent in a Bayesian model. You are forced to publicly declare your dependence on assumptions encoded in the prior. In the frequentist case, many people don't even consider that there are prior assumptions that may not hold about the data, and indeed they often report results which are incorrect because they did not account for basic things, like how their data is encoded, or non-ignorable experimental designs. One of my favorite papers about this sort of thing is "Let's Put the Garbage-Can Regression and Garbage-Can Probits Where They Belong" by Christopher Achen [1]. Of course, the same thing can happen in a Bayesian setting. Someone can use an off-the-shelf Bayesian model fitting tool without understanding what it is doing, and they could use standard prior distributions for analytical convenience or ease of coding, and then later try to package their result as some "fancy Bayesian analysis" when it really suffers from low stats quality just like the dumb t-stat comparison would have. But it's harder to do that in a Bayesian setting without it drawing a red flag (I guess, unless you just tell bald-faced lies about what code you wrote, but that's a different story). The bigger point is that you don't trust just one thing. You don't look at a single description of model fit or comparison, like two t-stats, and draw a conclusion. You should plot the residuals, make calibration plots, ROC curves, histograms of where your observations fall based on predicted outcome variables, etc. There is never any sort of cookie cutter approach that will tell you whether some backtest should be believed. As for the usefulness of managing this stuff in the form of articulating priors, one quote I always liked was in response to a critcism of Bayesian methods' reliance on a prior, where some academic said something like "So and so is standing on the front porch with a shotgun ready to shoot down any assumption that comes over the hill -- meanwhile, the back door is wide open for any assumption to run in." If we set aside the fact that any statistics procedure can be misused, or can be reported upon without proper skepticism and due diligence, then I still do further believe that frequentism, in general as an entire epistemological half of statistics is simply a failed project, especially for any type of problem that is a policy problem, which inherently requires access to a posterior distribution conditioned on some information. But, I don't want to derail this thread and turn it into yet another philosophical debate about whether frequentism can be used as a basis for probability theory that corresponds with reality or not. [1] http://www.columbia.edu/~gjw10/achen04.pdf http://www.columbia.edu/~gjw10/achen04.pdf

by p4wnc6 11y ago

xixi77 11y ago

You make it sound like Bayesian-type analysis is somehow fundamentally impervious to p-hacking and is therefore clearly superior to frequentist for this purpose; do you have any actual references in support of this -- because I am personally having serious doubts about that, in particular given how bad people are at coming up with priors (and when they do, wouldn't these priors be usually backward-looking anyway?)

p4wnc6 11y ago
No statistical method is impervious to malicious attempts to subvert it. That said, reasoning between two different models on the basis of magnitude of t-stat is inherently flawed. That is, such an inference process doesn't possess the properties necessary to lead to the conclusions people try to draw from it, even if no malicious intent is present. When you add this to a situation where the temptation is high for things like p-hacking, it actively encourages poor statistical hygiene. Whereas, with Bayesian statistics, you'd have to go out of your way and do something that is glaringly fishy to do the same thing. Hiding it from people is generally a lot harder than hiding what you're doing with generic p-hacking. One of the main reasons for this is that the model's reliance upon assumptions is usually far more transparent in a Bayesian model. You are forced to publicly declare your dependence on assumptions encoded in the prior. In the frequentist case, many people don't even consider that there are prior assumptions that may not hold about the data, and indeed they often report results which are incorrect because they did not account for basic things, like how their data is encoded, or non-ignorable experimental designs. One of my favorite papers about this sort of thing is "Let's Put the Garbage-Can Regression and Garbage-Can Probits Where They Belong" by Christopher Achen [1]. Of course, the same thing can happen in a Bayesian setting. Someone can use an off-the-shelf Bayesian model fitting tool without understanding what it is doing, and they could use standard prior distributions for analytical convenience or ease of coding, and then later try to package their result as some "fancy Bayesian analysis" when it really suffers from low stats quality just like the dumb t-stat comparison would have. But it's harder to do that in a Bayesian setting without it drawing a red flag (I guess, unless you just tell bald-faced lies about what code you wrote, but that's a different story). The bigger point is that you don't trust just one thing. You don't look at a single description of model fit or comparison, like two t-stats, and draw a conclusion. You should plot the residuals, make calibration plots, ROC curves, histograms of where your observations fall based on predicted outcome variables, etc. There is never any sort of cookie cutter approach that will tell you whether some backtest should be believed. As for the usefulness of managing this stuff in the form of articulating priors, one quote I always liked was in response to a critcism of Bayesian methods' reliance on a prior, where some academic said something like "So and so is standing on the front porch with a shotgun ready to shoot down any assumption that comes over the hill -- meanwhile, the back door is wide open for any assumption to run in." If we set aside the fact that any statistics procedure can be misused, or can be reported upon without proper skepticism and due diligence, then I still do further believe that frequentism, in general as an entire epistemological half of statistics is simply a failed project, especially for any type of problem that is a policy problem, which inherently requires access to a posterior distribution conditioned on some information. But, I don't want to derail this thread and turn it into yet another philosophical debate about whether frequentism can be used as a basis for probability theory that corresponds with reality or not. [1] http://www.columbia.edu/~gjw10/achen04.pdf http://www.columbia.edu/~gjw10/achen04.pdf