5 ms·

This article is a truly fantastic critique of how modern social sciences (besides economics) are carried out and publicized in modern society. Any critique one

This article is a truly fantastic critique of how modern social sciences (besides economics) are carried out and publicized in modern society. Any critique one dislikes - typically "your theory predicts X which is false" - can be dismissed by "nuance", "things are more complicated", "that's different in unspecified way", etc. In statistical terms, these dismissals are little more than a call for overfitting - if I can arbitrarily add more dimensions to a theory, I can make it fit any data set. This is a fantastic article that can be applied in so many places. In applied social sciences this critique is far easier to work around. "Yeah, my theory is pretty one-dimensional and does lack nuance. My code is in $REPO - feel free to run an experiment and see if your multidimensional theory generates any alpha over mine."

p4wnc6 11y ago
Why do you exclude economics? I see economics as practically leading the way with regard to dismissals like you suggest. One example I have first-hand experience with is the low-volatility anomaly. CAPM-like theories predict a roughly linear, positive-sloping relationship between risk and return. Yet decades worth of various empirical measurements of the riskiness of assets and portfolios unequivocally demonstrates that it is a negative-sloping relationship (e.g. for your willingness to bear an additional unit of risk, you actually receive less return). But everyone says basically the same thing. "Things are more complicated in real markets." It would be reasonable to say that CAPM is a mathematical result, and the evidence suggests the mathematical model required for CAPM simply fails to correspond to reality. But many finance folks, especially in academia, won't admit it. For them, CAPM must also be a descriptive theory. Two sources that illustrate what I'm talking about: [0] Baker, Bradley, and Wurgler, "Benchmarks as Limits to Arbitrage: Understanding the Low-Volatility Anomaly" FAJ Volume 61, 2011. < http://www.cfapubs.org/doi/pdf/10.2469/faj.v67.n1.4 http://www.cfapubs.org/doi/pdf/10.2469/faj.v67.n1.4 > [1] Baker, Malcolm P. and Bradley, Brendan and Taliaferro, Ryan, The Low Risk Anomaly: A Decomposition into Micro and Macro Effects (September 13, 2013). Financial Analysts Journal. < http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2210003 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2210003 > *disclaimer: The authors of these two papers were my bosses in a previous role, and I wrote the code and generated the results for the second paper.
- ryanmonroe 11y ago
  >t would be reasonable to say that CAPM is a mathematical result, and the evidence suggests the mathematical model required for CAPM simply fails to correspond to reality. But many finance folks, especially in academia, won't admit it. In other words, "Things are more complicated in real markets". I don't see any problem with wanting descriptive theories. Economists are interested in building models whose parameters can be tweaked in ways that correspond in a clear meaningful way to a possible actions (e.g. change government expenditure on X by Y). If you try to just maximize predictive power you're missing the point. Economics can be used to predict the future, but I would say the main goal of the field is to produce theory that will provide valid methods of altering the future.
  - p4wnc6 11y ago
    This doesn't make a lot of sense in this example with CAPM. The economic theory provides something that is abjectly wrong. It is literally claiming there is a positive relationship between two quantities when the evidence suggests the relationship is negative. The theory is grossly qualitatively wrong, not merely just on the edge of being quantitatively wrong in a few cases or something.
  - tormeh 11y ago
    So you're saying that we need wrong theories that can guide policymakers in the right direction? That sounds like a bad idea. It's basically "I want you to do this thing. Let me invent a theory that makes me sound more persuasive. Don't worry about the details!" Yes, we need theories that are easy enough to reason about that they provide useful insight, but I don't think this is even close to being too complicated.
- yummyfajitas 11y ago
  What you are describing in economics - while perhaps a failure mode - is the exact opposite of what this article discusses. The critique of this article is that valid abstractions and theories are rejected due to a lack of "nuance", rather than a failure to explain and predict reality: Second is the ever more extensive expansion of some theoretical system in a way that effectively closes it o from rebuttal or disconrmation by anything in the world...It is an evasion of the demand that a theory be refutable. Your critique of some economic theories is a totally different one - that they make specific predictions which don't come true.
jerf 11y ago
At first "overfitting" was the concept that leaped to mind as I read it, but overfitting is sort of something different. You have a complicated data set, and when you overfit, you keep adding dimensions of freedom, and then setting those dimensions of freedom to produce the original values that were passed in, such that it fails to correctly predict any other instance drawn from some conceptual "same distribution". In this case, the theories do the "adding many dimensions of freedom" part, but they don't really fix the values; they probably haven't got data to fix in the first place, and it's not mathematical enough to be that precise about the fitting. I think it's more like these theories are often not so much hypotheses themselves, but the descriptions of hypotheses spaces, and the addition of "nuance" means that in a lot of cases, what sounds like a "theory" is really just the description of a space that happens to contain all or most of the possibility space in question. As much fun as it may be to do that, and as much as it may even be a necessary first step to a real theory, in practice that is not generally the difficult part of the process. (There are exceptions to that which may leap to mind but I submit they would be the exceptions, rather than the rule.) When one is working with fuzzy words (even if by necessity!) and not math, the fuzzy words make it very easy to make this mistake, I think. English does not have a strong distinction between a hypothesis space and a specific hypothesis in that space, as evidenced by the very fact I have to dip into jargon to even make that point! From that perspective, the solution that leaps to my mind is implement the observation that in many ways, the important thing about a theory is not what it describes, but what it excludes. It is too easy and cognitively tempting to try to create a theory that, when you fiddle with the parameters, describes everything you see, but the real questions are, what is excluded by your theory? Can I fiddle with the knobs in your theory to describe something that you say ought to exist, but does not seem to? Is your theory even well-enough specified that these are meaningful questions to ask about it? I don't know the answers to these questions for sociology. I have to admit I've tried to engage with it a couple of times, because what fan of Isaac Asimov doesn't want to learn about some real-life psychohistory, but I've always found it difficult to get traction with with my engineer mind. I can engage from a philosophical point of view, but that is not generally what I was looking for; I've got, well, philosophy itself for that. (Repeat previous sentence for history, psychology, and economics.) To p4wnc6's question, I'd submit that as dismal as the Dismal Science may sometimes be, it does often meet this bar. I fear economists themselves may often be quite sluggish at recognizing when their own theories are contradicted by the evidence because the theories are saying that what is currently happening (in the general sense, not the literal March 27, 2016 sense) is excluded by the theory, but that is a human problem, not an economic theory problem. Economic theories do exclude things; if supply rises and demand holds steady, the vast majority of possibilities where price goes up are excluded, and if price were to go up, the theories would be useful in investigating how such a thing actually happened by examining what such theories would still permit (e.g., perhaps supply isn't really rising because the new supply is actually a higher quantity of lower quality, and too much of the demand is for a higher quality than the new supply is supplying which means "real" supply is dropping, or something like that). They certainly aren't perfect and a given theory may even be wrong, but, well, the very fact that a theory can be wrong is evidence that it is falsifiable, and thus not "nuanced" into infinite psudo-explanatory power.
Olscore 11y ago
> In statistical terms, these dismissals are little more than a call for overfitting - if I can arbitrarily add more dimensions to a theory, I can make it fit any data set. Sure. However, this leads me to believe it is statistics that are the problem, and not nuance. You are in my opinion admitting that nuance invalidates most statistics, or at least allows for heavy manipulation of the statistics to fit any narrative. This is precisely why public policy should not be made using statistics. Society should not be likened to a turing machine. People, history and so many topics of discussion are full of nuance, detail and perspective; and disregarding nuance (or perspectives) is insulting to the truth.
- yummyfajitas 11y ago
  No, on the contrary I'm saying that calls for nuance are often excuses to do bad statistics and bad science. Rather than throwing away our theories because they predict X and the world does Y, they instead say "actually the theory only applies under vague unspecified circumstances which you'd understand if you were as nuanced as I". Whether or not the theory fits the data, they cry "intersectionality" or "humans are so complex". In any case, your post begs the question. If we should ignore statistics, how should we make decisions? Emotionally pleasing verbal narratives? See also the related essay, "Pragmatism is Poison". It argues that the word "pragmatism" is used in software a lot to shut down discussion of bad choices, much the same way "nuance" is used to shut down discussion of good theories. https://einarwh.wordpress.com/2016/03/10/pragmatism-is-poison/ https://einarwh.wordpress.com/2016/03/10/pragmatism-is-poiso...
  - platz 11y ago
    No true scottsman will allow their theory to be critiqued. A true scottsman is more nuanced.
  - pron 11y ago
    > I'm saying that calls for nuance are often excuses to do bad statistics and bad science. Absolutely. But at least as often, calls for nuance are calls for better statistics and better science. How do you know which is which? Judgement. You can only say "fuck nuance" once you've demonstrated good judgment.
pron 11y ago
It is a fantastic article, but I don't think excessive, obsessive nuance is what plagues most people here. In any event, don't take it too far. In dynamical system, your theory can be simple and predict observation 100% yet be completely wrong -- not just a little wrong -- and fail to model any of the actual dynamics (e.g. x = 0 and x = t are both completely predictive, yet entirely false, models of x" = 0). It is those simplified theories that are no more than empirical observation, and suffer from another kind of overfitting. > In statistical terms, these dismissals are little more than a call for overfitting. can arbitrarily add more dimensions to a theory, I can make it fit any data set. In order to fit any social data set, you may need dozens if not hundreds of variables. If your data set contains many millions of data points, telling you that there may be four variables instead than two is certainly not "a call for overfitting"; it is no more excessive nuance than the entirety of special+general relativity. Or, don't use this as an excuse to stop questioning your theory once it predicts behavior in one setting. You may have completely missed the mark by misjudging the order of the equation. Let alone, when your theory doesn't even fit the data we have -- just the data you have -- it's nothing more than amateurism; don't confuse amateur smart-assedness with any sort of profundity; it is often wrong in the most boring way: uninformed wrong. Intuition that comes from years of study of human behavior may help a lot, but some believe they can come up with theories without spending the time to earn their intuition first. Whether overly nuanced or over-simplified, I think the chances that any of their theories are correct -- especially if they put little effort in forming them -- are low. And one thing is certain: the less you know, the far more likely it is that you'll err on the side of over-simplification than over-nuance. This paper is a warning to experts, who know enough to err with over-care. Amateurs (of the non-humble kind) are indeed completely protected from this particular kind of error by their ignorance, especially if they don't have the basic tools to even assess the system's level of complexity. This paper is a call against obsessive, exaggerated nuance. Not a call for oversimplification and sloppiness. And while one can err on both sides, historically, errors of over-simplification (when it comes to social theories) have been much more detrimental to the human race than errors of nuance. The latter kind may cause boring science; the former has led to atrocities. I think that many of the calls for more nuance are just warnings against careless errors of the first kind, simply because they are far more dangerous than errors of the second kind.
- 11y ago

p4wnc6 11y ago

Why do you exclude economics? I see economics as practically leading the way with regard to dismissals like you suggest. One example I have first-hand experience with is the low-volatility anomaly. CAPM-like theories predict a roughly linear, positive-sloping relationship between risk and return. Yet decades worth of various empirical measurements of the riskiness of assets and portfolios unequivocally demonstrates that it is a negative-sloping relationship (e.g. for your willingness to bear an additional unit of risk, you actually receive less return). But everyone says basically the same thing. "Things are more complicated in real markets." It would be reasonable to say that CAPM is a mathematical result, and the evidence suggests the mathematical model required for CAPM simply fails to correspond to reality. But many finance folks, especially in academia, won't admit it. For them, CAPM must also be a descriptive theory. Two sources that illustrate what I'm talking about: [0] Baker, Bradley, and Wurgler, "Benchmarks as Limits to Arbitrage: Understanding the Low-Volatility Anomaly" FAJ Volume 61, 2011. < http://www.cfapubs.org/doi/pdf/10.2469/faj.v67.n1.4 http://www.cfapubs.org/doi/pdf/10.2469/faj.v67.n1.4 > [1] Baker, Malcolm P. and Bradley, Brendan and Taliaferro, Ryan, The Low Risk Anomaly: A Decomposition into Micro and Macro Effects (September 13, 2013). Financial Analysts Journal. < http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2210003 http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2210003 > *disclaimer: The authors of these two papers were my bosses in a previous role, and I wrote the code and generated the results for the second paper.

ryanmonroe 11y ago
>t would be reasonable to say that CAPM is a mathematical result, and the evidence suggests the mathematical model required for CAPM simply fails to correspond to reality. But many finance folks, especially in academia, won't admit it. In other words, "Things are more complicated in real markets". I don't see any problem with wanting descriptive theories. Economists are interested in building models whose parameters can be tweaked in ways that correspond in a clear meaningful way to a possible actions (e.g. change government expenditure on X by Y). If you try to just maximize predictive power you're missing the point. Economics can be used to predict the future, but I would say the main goal of the field is to produce theory that will provide valid methods of altering the future.
- p4wnc6 11y ago
  This doesn't make a lot of sense in this example with CAPM. The economic theory provides something that is abjectly wrong. It is literally claiming there is a positive relationship between two quantities when the evidence suggests the relationship is negative. The theory is grossly qualitatively wrong, not merely just on the edge of being quantitatively wrong in a few cases or something.
- tormeh 11y ago
  So you're saying that we need wrong theories that can guide policymakers in the right direction? That sounds like a bad idea. It's basically "I want you to do this thing. Let me invent a theory that makes me sound more persuasive. Don't worry about the details!" Yes, we need theories that are easy enough to reason about that they provide useful insight, but I don't think this is even close to being too complicated.
yummyfajitas 11y ago
What you are describing in economics - while perhaps a failure mode - is the exact opposite of what this article discusses. The critique of this article is that valid abstractions and theories are rejected due to a lack of "nuance", rather than a failure to explain and predict reality: Second is the ever more extensive expansion of some theoretical system in a way that effectively closes it o from rebuttal or disconrmation by anything in the world...It is an evasion of the demand that a theory be refutable. Your critique of some economic theories is a totally different one - that they make specific predictions which don't come true.

jerf 11y ago

At first "overfitting" was the concept that leaped to mind as I read it, but overfitting is sort of something different. You have a complicated data set, and when you overfit, you keep adding dimensions of freedom, and then setting those dimensions of freedom to produce the original values that were passed in, such that it fails to correctly predict any other instance drawn from some conceptual "same distribution". In this case, the theories do the "adding many dimensions of freedom" part, but they don't really fix the values; they probably haven't got data to fix in the first place, and it's not mathematical enough to be that precise about the fitting. I think it's more like these theories are often not so much hypotheses themselves, but the descriptions of hypotheses spaces, and the addition of "nuance" means that in a lot of cases, what sounds like a "theory" is really just the description of a space that happens to contain all or most of the possibility space in question. As much fun as it may be to do that, and as much as it may even be a necessary first step to a real theory, in practice that is not generally the difficult part of the process. (There are exceptions to that which may leap to mind but I submit they would be the exceptions, rather than the rule.) When one is working with fuzzy words (even if by necessity!) and not math, the fuzzy words make it very easy to make this mistake, I think. English does not have a strong distinction between a hypothesis space and a specific hypothesis in that space, as evidenced by the very fact I have to dip into jargon to even make that point! From that perspective, the solution that leaps to my mind is implement the observation that in many ways, the important thing about a theory is not what it describes, but what it excludes. It is too easy and cognitively tempting to try to create a theory that, when you fiddle with the parameters, describes everything you see, but the real questions are, what is excluded by your theory? Can I fiddle with the knobs in your theory to describe something that you say ought to exist, but does not seem to? Is your theory even well-enough specified that these are meaningful questions to ask about it? I don't know the answers to these questions for sociology. I have to admit I've tried to engage with it a couple of times, because what fan of Isaac Asimov doesn't want to learn about some real-life psychohistory, but I've always found it difficult to get traction with with my engineer mind. I can engage from a philosophical point of view, but that is not generally what I was looking for; I've got, well, philosophy itself for that. (Repeat previous sentence for history, psychology, and economics.) To p4wnc6's question, I'd submit that as dismal as the Dismal Science may sometimes be, it does often meet this bar. I fear economists themselves may often be quite sluggish at recognizing when their own theories are contradicted by the evidence because the theories are saying that what is currently happening (in the general sense, not the literal March 27, 2016 sense) is excluded by the theory, but that is a human problem, not an economic theory problem. Economic theories do exclude things; if supply rises and demand holds steady, the vast majority of possibilities where price goes up are excluded, and if price were to go up, the theories would be useful in investigating how such a thing actually happened by examining what such theories would still permit (e.g., perhaps supply isn't really rising because the new supply is actually a higher quantity of lower quality, and too much of the demand is for a higher quality than the new supply is supplying which means "real" supply is dropping, or something like that). They certainly aren't perfect and a given theory may even be wrong, but, well, the very fact that a theory can be wrong is evidence that it is falsifiable, and thus not "nuanced" into infinite psudo-explanatory power.

Olscore 11y ago

> In statistical terms, these dismissals are little more than a call for overfitting - if I can arbitrarily add more dimensions to a theory, I can make it fit any data set. Sure. However, this leads me to believe it is statistics that are the problem, and not nuance. You are in my opinion admitting that nuance invalidates most statistics, or at least allows for heavy manipulation of the statistics to fit any narrative. This is precisely why public policy should not be made using statistics. Society should not be likened to a turing machine. People, history and so many topics of discussion are full of nuance, detail and perspective; and disregarding nuance (or perspectives) is insulting to the truth.

yummyfajitas 11y ago
No, on the contrary I'm saying that calls for nuance are often excuses to do bad statistics and bad science. Rather than throwing away our theories because they predict X and the world does Y, they instead say "actually the theory only applies under vague unspecified circumstances which you'd understand if you were as nuanced as I". Whether or not the theory fits the data, they cry "intersectionality" or "humans are so complex". In any case, your post begs the question. If we should ignore statistics, how should we make decisions? Emotionally pleasing verbal narratives? See also the related essay, "Pragmatism is Poison". It argues that the word "pragmatism" is used in software a lot to shut down discussion of bad choices, much the same way "nuance" is used to shut down discussion of good theories. https://einarwh.wordpress.com/2016/03/10/pragmatism-is-poison/ https://einarwh.wordpress.com/2016/03/10/pragmatism-is-poiso...
- platz 11y ago
  No true scottsman will allow their theory to be critiqued. A true scottsman is more nuanced.
- pron 11y ago
  > I'm saying that calls for nuance are often excuses to do bad statistics and bad science. Absolutely. But at least as often, calls for nuance are calls for better statistics and better science. How do you know which is which? Judgement. You can only say "fuck nuance" once you've demonstrated good judgment.

pron 11y ago

It is a fantastic article, but I don't think excessive, obsessive nuance is what plagues most people here. In any event, don't take it too far. In dynamical system, your theory can be simple and predict observation 100% yet be completely wrong -- not just a little wrong -- and fail to model any of the actual dynamics (e.g. x = 0 and x = t are both completely predictive, yet entirely false, models of x" = 0). It is those simplified theories that are no more than empirical observation, and suffer from another kind of overfitting. > In statistical terms, these dismissals are little more than a call for overfitting. can arbitrarily add more dimensions to a theory, I can make it fit any data set. In order to fit any social data set, you may need dozens if not hundreds of variables. If your data set contains many millions of data points, telling you that there may be four variables instead than two is certainly not "a call for overfitting"; it is no more excessive nuance than the entirety of special+general relativity. Or, don't use this as an excuse to stop questioning your theory once it predicts behavior in one setting. You may have completely missed the mark by misjudging the order of the equation. Let alone, when your theory doesn't even fit the data we have -- just the data you have -- it's nothing more than amateurism; don't confuse amateur smart-assedness with any sort of profundity; it is often wrong in the most boring way: uninformed wrong. Intuition that comes from years of study of human behavior may help a lot, but some believe they can come up with theories without spending the time to earn their intuition first. Whether overly nuanced or over-simplified, I think the chances that any of their theories are correct -- especially if they put little effort in forming them -- are low. And one thing is certain: the less you know, the far more likely it is that you'll err on the side of over-simplification than over-nuance. This paper is a warning to experts, who know enough to err with over-care. Amateurs (of the non-humble kind) are indeed completely protected from this particular kind of error by their ignorance, especially if they don't have the basic tools to even assess the system's level of complexity. This paper is a call against obsessive, exaggerated nuance. Not a call for oversimplification and sloppiness. And while one can err on both sides, historically, errors of over-simplification (when it comes to social theories) have been much more detrimental to the human race than errors of nuance. The latter kind may cause boring science; the former has led to atrocities. I think that many of the calls for more nuance are just warnings against careless errors of the first kind, simply because they are far more dangerous than errors of the second kind.

11y ago