4 ms·

It goes further. Folks who care strongly about the interpretability factor generally only care because it gives them authority and political control -- they def

by p4wnc6 10y ago

It goes further. Folks who care strongly about the interpretability factor generally only care because it gives them authority and political control -- they definitely don't actually believe it's a superior, nor even cost-effective, way of obtaining prediction accuracy. Robin Hanson summarized this well in his article "Who about forecast accuracy?" [0]. Here's a nice quote: > The benefits of creating and monitoring forecast accuracy might be lower than we expect if the function and role of forecasting is less important than we think, relative to the many functions and roles served by our pundits, academics, and managers. > Consider first the many possible functions and roles of media pundits. Media consumers can be educated and entertained by clever, witty, but accessible commentary, and can coordinate to signal that they are smart and well-read by quoting and discussing the words of the same few focal pundits. Also, impressive pundits with prestigious credentials and clear “philosophical” positions can let readers and viewers gain by affiliation with such impressiveness, credentials, and positions. Being easier to understand and classify helps “hedgehogs” to serve many of these functions. > Second, consider the many functions and roles of academics. Academics are primarily selected and rewarded for their impressive mastery and application of difficult academic tools and methods. Students, patrons, and media contacts can gain by affiliation with credentialed academic impressiveness. In forecasts, academic are rewarded much more for showing mastery of impressive tools than for accuracy. > Finally, consider next the many functions and roles of managers, both public and private. By being personally impressive, and by being identified with attractive philosophical positions, leaders can inspire people to work for and affiliate with their organizations. Such support can be threatened by clear tracking of leader forecasts, if that questions leader impressiveness. FWIW, having worked for a while in quant finance, I can testify that this precisely nails it. Even when it would be cheap, easy, obviously worthwhile, and likely impactful, projects to switch from politically-controlled, simplistic regressions couched in the buzzwords of academic finance and instead embrace principled statistical computing, proper model fitting hygiene, machine learning, etc., such improvements will never be considered. There's too much political gatekeeper rent-seeking at the higher levels of the management who oversee the investment process. Sure, they'll claim plausible deniability by saying something is "a black box" or making a fallacious comparison with an algorithmic trading firm that lost money, but the real reason is to protect their fiefdom, over which they are the lord and master because of their credentialed authority in some "interpretable" buzzword. [0] < http://www.cato-unbound.org/2011/07/13/robin-hanson/who-cares-about-forecast-accuracy http://www.cato-unbound.org/2011/07/13/robin-hanson/who-care... >

mathattack 10y ago
I would think quantitative finance is the one area where models are checked, because there's real money on the line. If you're a sell side strategist it might not matter, but if your bonus is on the line, you'll keep an open mind.
- p4wnc6 10y ago
  But that's the problem. Bonuses are awarded based on who can argue politically. It's similar to creating Dutch books. Managers will work hard to set a up a situation such that, no matter what the actual performance outcome really is, they have ready-made, plausible-seeming arguments for why it wasn't their fault and why they did what was asked and it was impossible to avoid the particular outcome. Then they push the burden of blame for a failed outcome somewhere down the hierarchy. It's not a perfect science, and yes, sometimes managers do get dinged badly for failing to be objective enough, but most often even what that does happen the incentive scheme doesn't cause them to become more open-minded with regard to modeling techniques, rather more cynical and aggressive with political techniques. I thought the exact same thing going into finance. If real money is on the line, surely they care about which techniques are most rigorously justified in a given model context, right? Absolutely not. The incentive schemes from clients (who often don't actually fire bad investment managers when they should, and who also engage in chasing returns momentum despite the long and, by now, boring and uncontroversial history of that not paying off) don't actually punish inefficiency. It's a big political mess. Really the best you can hope for is that they'll hire you for marketing purposes. Hey look, we brought in a shiny new expert in deep learning -- we're cutting edge, we swear! They won't actually let you do any real work with deep learning, of course. It will all be Excel jockey bullshit on factor models in which you'll do obviously fallacious things like directly compare the t-stats of two different model fits as a means for model selection. Maybe you'll write an ineffectual white paper on something slightly more advanced from time to time. But the big reason you're hired is to look good on paper and smoke cigars and drink brandy with the right person who wants you as a political darling in order to win arguments from authority about how you definitely should not migrate away from Excel/VBA. This is not hyperbole, sadly.
  - 10y ago

by p4wnc6 10y ago

mathattack 10y ago

I would think quantitative finance is the one area where models are checked, because there's real money on the line. If you're a sell side strategist it might not matter, but if your bonus is on the line, you'll keep an open mind.

p4wnc6 10y ago
But that's the problem. Bonuses are awarded based on who can argue politically. It's similar to creating Dutch books. Managers will work hard to set a up a situation such that, no matter what the actual performance outcome really is, they have ready-made, plausible-seeming arguments for why it wasn't their fault and why they did what was asked and it was impossible to avoid the particular outcome. Then they push the burden of blame for a failed outcome somewhere down the hierarchy. It's not a perfect science, and yes, sometimes managers do get dinged badly for failing to be objective enough, but most often even what that does happen the incentive scheme doesn't cause them to become more open-minded with regard to modeling techniques, rather more cynical and aggressive with political techniques. I thought the exact same thing going into finance. If real money is on the line, surely they care about which techniques are most rigorously justified in a given model context, right? Absolutely not. The incentive schemes from clients (who often don't actually fire bad investment managers when they should, and who also engage in chasing returns momentum despite the long and, by now, boring and uncontroversial history of that not paying off) don't actually punish inefficiency. It's a big political mess. Really the best you can hope for is that they'll hire you for marketing purposes. Hey look, we brought in a shiny new expert in deep learning -- we're cutting edge, we swear! They won't actually let you do any real work with deep learning, of course. It will all be Excel jockey bullshit on factor models in which you'll do obviously fallacious things like directly compare the t-stats of two different model fits as a means for model selection. Maybe you'll write an ineffectual white paper on something slightly more advanced from time to time. But the big reason you're hired is to look good on paper and smoke cigars and drink brandy with the right person who wants you as a political darling in order to win arguments from authority about how you definitely should not migrate away from Excel/VBA. This is not hyperbole, sadly.
- 10y ago