5 ms·

The Mythos of Model Interpretability

56 points by jboynyc 10y ago

cs702 10y ago
When educated people like doctors, lawyers, managers, etc., ask for "interpretability," what they mean is, "what are the 'key variables' we can control that will help produce the outcomes we want?" Simpler statistical models, and particularly linear models, provide easy-to-grasp "answers" to that question -- for example, "to get more of y, we need more of x1 and x2, and less of x3." An entire generation of leaders and professionals has been trained to think this way by statistics professors and teachers who have been rigorously trained to think that way too. (For background, I highly recommend Leo Breiman's now-classic paper, "Statistical Modeling: The Two Cultures:" https://projecteuclid.org/download/pdf_1/euclid.ss/1009213726 https://projecteuclid.org/download/pdf_1/euclid.ss/100921372... ) Alas, there's no going back. Simpler models do not work for complicated tasks like, say, predicting the next best move in a game of Go, for which there are no 'key variables' in the traditional sense.
- p4wnc6 10y ago
  It goes further. Folks who care strongly about the interpretability factor generally only care because it gives them authority and political control -- they definitely don't actually believe it's a superior, nor even cost-effective, way of obtaining prediction accuracy. Robin Hanson summarized this well in his article "Who about forecast accuracy?" [0]. Here's a nice quote: > The benefits of creating and monitoring forecast accuracy might be lower than we expect if the function and role of forecasting is less important than we think, relative to the many functions and roles served by our pundits, academics, and managers. > Consider first the many possible functions and roles of media pundits. Media consumers can be educated and entertained by clever, witty, but accessible commentary, and can coordinate to signal that they are smart and well-read by quoting and discussing the words of the same few focal pundits. Also, impressive pundits with prestigious credentials and clear “philosophical” positions can let readers and viewers gain by affiliation with such impressiveness, credentials, and positions. Being easier to understand and classify helps “hedgehogs” to serve many of these functions. > Second, consider the many functions and roles of academics. Academics are primarily selected and rewarded for their impressive mastery and application of difficult academic tools and methods. Students, patrons, and media contacts can gain by affiliation with credentialed academic impressiveness. In forecasts, academic are rewarded much more for showing mastery of impressive tools than for accuracy. > Finally, consider next the many functions and roles of managers, both public and private. By being personally impressive, and by being identified with attractive philosophical positions, leaders can inspire people to work for and affiliate with their organizations. Such support can be threatened by clear tracking of leader forecasts, if that questions leader impressiveness. FWIW, having worked for a while in quant finance, I can testify that this precisely nails it. Even when it would be cheap, easy, obviously worthwhile, and likely impactful, projects to switch from politically-controlled, simplistic regressions couched in the buzzwords of academic finance and instead embrace principled statistical computing, proper model fitting hygiene, machine learning, etc., such improvements will never be considered. There's too much political gatekeeper rent-seeking at the higher levels of the management who oversee the investment process. Sure, they'll claim plausible deniability by saying something is "a black box" or making a fallacious comparison with an algorithmic trading firm that lost money, but the real reason is to protect their fiefdom, over which they are the lord and master because of their credentialed authority in some "interpretable" buzzword. [0] < http://www.cato-unbound.org/2011/07/13/robin-hanson/who-cares-about-forecast-accuracy http://www.cato-unbound.org/2011/07/13/robin-hanson/who-care... >
  - mathattack 10y ago
    I would think quantitative finance is the one area where models are checked, because there's real money on the line. If you're a sell side strategist it might not matter, but if your bonus is on the line, you'll keep an open mind.
    - p4wnc6 10y ago
      But that's the problem. Bonuses are awarded based on who can argue politically. It's similar to creating Dutch books. Managers will work hard to set a up a situation such that, no matter what the actual performance outcome really is, they have ready-made, plausible-seeming arguments for why it wasn't their fault and why they did what was asked and it was impossible to avoid the particular outcome. Then they push the burden of blame for a failed outcome somewhere down the hierarchy. It's not a perfect science, and yes, sometimes managers do get dinged badly for failing to be objective enough, but most often even what that does happen the incentive scheme doesn't cause them to become more open-minded with regard to modeling techniques, rather more cynical and aggressive with political techniques. I thought the exact same thing going into finance. If real money is on the line, surely they care about which techniques are most rigorously justified in a given model context, right? Absolutely not. The incentive schemes from clients (who often don't actually fire bad investment managers when they should, and who also engage in chasing returns momentum despite the long and, by now, boring and uncontroversial history of that not paying off) don't actually punish inefficiency. It's a big political mess. Really the best you can hope for is that they'll hire you for marketing purposes. Hey look, we brought in a shiny new expert in deep learning -- we're cutting edge, we swear! They won't actually let you do any real work with deep learning, of course. It will all be Excel jockey bullshit on factor models in which you'll do obviously fallacious things like directly compare the t-stats of two different model fits as a means for model selection. Maybe you'll write an ineffectual white paper on something slightly more advanced from time to time. But the big reason you're hired is to look good on paper and smoke cigars and drink brandy with the right person who wants you as a political darling in order to win arguments from authority about how you definitely should not migrate away from Excel/VBA. This is not hyperbole, sadly.
gmisra 10y ago
Maybe the prevalent mental model regarding models is wrong? The real elephant in the room is the idea that "all models" are similar, and therefore occupy the same space regarding interpretability, forecasting, and other uses. Models are designed to solve specific problems, usually within specific scopes. Confusion, misinterpretation, and error occur when consumers of models don't understand (or simply ignore) the limitations of these models. Whenever I sit down to build a new model, or work with a new-to-me model, I start with questions like: - What questions was this model built to answer? Be careful when asking other questions, and know the limitations of those answers. - If it's been in use for a while, how effective has it been? How is that efficacy measured? A highly accurate decomposition of historical behavior may involve a lot of post-fact knowledge that has no predictive power, so be careful interpreting cross-use results. - What are the ranges and distributions of the input variables? What are their boundary conditions? This is not always applicable to all variables, especially unstructured data, but when it is applicable, it is usually straightforward. - What are the ranges of the model outputs? Do they have any boundary conditions? Do boundary effects impact the situations in which the model can be applied? For example, when working with a binned variable, often bins 0 and n include less homogenous data than the rest of the bins. - How is model accuracy calculated? This is usually far less objective than model constructors are willing to admit to. How does your error vary along different dimensions? Which dimensional error do you look at more closely, and why? In my experience, the best way to reason about models is to work with a lot of different models, and to be honest about their flaws. That learning generally happens more efficiently, and more broadly, in the real world than in the classroom. With the recent rise in popularity of applied models, we have lots of inexperience modelers out there, so these growing pains are to be expected.
- scribu 10y ago
  It seems like you're talking about human-created models (e.g. a manually-constructed decision tree), whereas the paper's primary concern is machine-created models (e.g. the weights derived mechanically by a neural network). So the problem would be: how do you get the "modelers" (i.e. the machines) to be "honest about the flaws" of the models that they generated?

56 points by jboynyc 10y ago

cs702 10y ago

When educated people like doctors, lawyers, managers, etc., ask for "interpretability," what they mean is, "what are the 'key variables' we can control that will help produce the outcomes we want?" Simpler statistical models, and particularly linear models, provide easy-to-grasp "answers" to that question -- for example, "to get more of y, we need more of x1 and x2, and less of x3." An entire generation of leaders and professionals has been trained to think this way by statistics professors and teachers who have been rigorously trained to think that way too. (For background, I highly recommend Leo Breiman's now-classic paper, "Statistical Modeling: The Two Cultures:" https://projecteuclid.org/download/pdf_1/euclid.ss/1009213726 https://projecteuclid.org/download/pdf_1/euclid.ss/100921372... ) Alas, there's no going back. Simpler models do not work for complicated tasks like, say, predicting the next best move in a game of Go, for which there are no 'key variables' in the traditional sense.

p4wnc6 10y ago
It goes further. Folks who care strongly about the interpretability factor generally only care because it gives them authority and political control -- they definitely don't actually believe it's a superior, nor even cost-effective, way of obtaining prediction accuracy. Robin Hanson summarized this well in his article "Who about forecast accuracy?" [0]. Here's a nice quote: > The benefits of creating and monitoring forecast accuracy might be lower than we expect if the function and role of forecasting is less important than we think, relative to the many functions and roles served by our pundits, academics, and managers. > Consider first the many possible functions and roles of media pundits. Media consumers can be educated and entertained by clever, witty, but accessible commentary, and can coordinate to signal that they are smart and well-read by quoting and discussing the words of the same few focal pundits. Also, impressive pundits with prestigious credentials and clear “philosophical” positions can let readers and viewers gain by affiliation with such impressiveness, credentials, and positions. Being easier to understand and classify helps “hedgehogs” to serve many of these functions. > Second, consider the many functions and roles of academics. Academics are primarily selected and rewarded for their impressive mastery and application of difficult academic tools and methods. Students, patrons, and media contacts can gain by affiliation with credentialed academic impressiveness. In forecasts, academic are rewarded much more for showing mastery of impressive tools than for accuracy. > Finally, consider next the many functions and roles of managers, both public and private. By being personally impressive, and by being identified with attractive philosophical positions, leaders can inspire people to work for and affiliate with their organizations. Such support can be threatened by clear tracking of leader forecasts, if that questions leader impressiveness. FWIW, having worked for a while in quant finance, I can testify that this precisely nails it. Even when it would be cheap, easy, obviously worthwhile, and likely impactful, projects to switch from politically-controlled, simplistic regressions couched in the buzzwords of academic finance and instead embrace principled statistical computing, proper model fitting hygiene, machine learning, etc., such improvements will never be considered. There's too much political gatekeeper rent-seeking at the higher levels of the management who oversee the investment process. Sure, they'll claim plausible deniability by saying something is "a black box" or making a fallacious comparison with an algorithmic trading firm that lost money, but the real reason is to protect their fiefdom, over which they are the lord and master because of their credentialed authority in some "interpretable" buzzword. [0] < http://www.cato-unbound.org/2011/07/13/robin-hanson/who-cares-about-forecast-accuracy http://www.cato-unbound.org/2011/07/13/robin-hanson/who-care... >
- mathattack 10y ago
  I would think quantitative finance is the one area where models are checked, because there's real money on the line. If you're a sell side strategist it might not matter, but if your bonus is on the line, you'll keep an open mind.
  - p4wnc6 10y ago
    But that's the problem. Bonuses are awarded based on who can argue politically. It's similar to creating Dutch books. Managers will work hard to set a up a situation such that, no matter what the actual performance outcome really is, they have ready-made, plausible-seeming arguments for why it wasn't their fault and why they did what was asked and it was impossible to avoid the particular outcome. Then they push the burden of blame for a failed outcome somewhere down the hierarchy. It's not a perfect science, and yes, sometimes managers do get dinged badly for failing to be objective enough, but most often even what that does happen the incentive scheme doesn't cause them to become more open-minded with regard to modeling techniques, rather more cynical and aggressive with political techniques. I thought the exact same thing going into finance. If real money is on the line, surely they care about which techniques are most rigorously justified in a given model context, right? Absolutely not. The incentive schemes from clients (who often don't actually fire bad investment managers when they should, and who also engage in chasing returns momentum despite the long and, by now, boring and uncontroversial history of that not paying off) don't actually punish inefficiency. It's a big political mess. Really the best you can hope for is that they'll hire you for marketing purposes. Hey look, we brought in a shiny new expert in deep learning -- we're cutting edge, we swear! They won't actually let you do any real work with deep learning, of course. It will all be Excel jockey bullshit on factor models in which you'll do obviously fallacious things like directly compare the t-stats of two different model fits as a means for model selection. Maybe you'll write an ineffectual white paper on something slightly more advanced from time to time. But the big reason you're hired is to look good on paper and smoke cigars and drink brandy with the right person who wants you as a political darling in order to win arguments from authority about how you definitely should not migrate away from Excel/VBA. This is not hyperbole, sadly.

gmisra 10y ago

Maybe the prevalent mental model regarding models is wrong? The real elephant in the room is the idea that "all models" are similar, and therefore occupy the same space regarding interpretability, forecasting, and other uses. Models are designed to solve specific problems, usually within specific scopes. Confusion, misinterpretation, and error occur when consumers of models don't understand (or simply ignore) the limitations of these models. Whenever I sit down to build a new model, or work with a new-to-me model, I start with questions like: - What questions was this model built to answer? Be careful when asking other questions, and know the limitations of those answers. - If it's been in use for a while, how effective has it been? How is that efficacy measured? A highly accurate decomposition of historical behavior may involve a lot of post-fact knowledge that has no predictive power, so be careful interpreting cross-use results. - What are the ranges and distributions of the input variables? What are their boundary conditions? This is not always applicable to all variables, especially unstructured data, but when it is applicable, it is usually straightforward. - What are the ranges of the model outputs? Do they have any boundary conditions? Do boundary effects impact the situations in which the model can be applied? For example, when working with a binned variable, often bins 0 and n include less homogenous data than the rest of the bins. - How is model accuracy calculated? This is usually far less objective than model constructors are willing to admit to. How does your error vary along different dimensions? Which dimensional error do you look at more closely, and why? In my experience, the best way to reason about models is to work with a lot of different models, and to be honest about their flaws. That learning generally happens more efficiently, and more broadly, in the real world than in the classroom. With the recent rise in popularity of applied models, we have lots of inexperience modelers out there, so these growing pains are to be expected.

scribu 10y ago
It seems like you're talking about human-created models (e.g. a manually-constructed decision tree), whereas the paper's primary concern is machine-created models (e.g. the weights derived mechanically by a neural network). So the problem would be: how do you get the "modelers" (i.e. the machines) to be "honest about the flaws" of the models that they generated?