4 ms·

Recognizing that not many will care about my opinion on the subject, I will link to this excellent (and IMO accurate) paper by Guido Imbens: https://arxiv.org/

Recognizing that not many will care about my opinion on the subject, I will link to this excellent (and IMO accurate) paper by Guido Imbens: https://arxiv.org/pdf/1907.07271.pdf https://arxiv.org/pdf/1907.07271.pdf One of the problems in groups is that there is a belief that the loudest and most self-confident individual is correct. I believe that is the case with Pearl. He attracts a following based on his personality, but when it comes to actually doing empirical research, he comes up short. "Separate from the theoretical merits of the two approaches, another reason for the lack of adoption in economics is that the DAG literature has not shown much evidence of the alleged benefits for empirical practice in settings that resonate with economists....In contrast in the DAG literature, TBOW, [Pearl, 2000], and[Peters, Janzing, and Sch ̈olkopf, 2017] have no substantive empirical examples, focusing largely on identification questions in what TBOW refers to as “toy” models. Compare the lack of impact of the DAG literature in economics with the recent embrace of regression discontinuity designs imported from the psychology literature, or with the current rapid spread of the machine learning methods from computer science, or the recent quick adoption of synthetic control methods developed in economics [Abadie and Gardeazabal, 2003, Abadie, Diamond, and Hainmueller,2010]. All three came with multiple concrete and detailed examples that highlighted their benefits over traditional methods. In the absence of such concrete examples the toy models in the DAG literature sometimes appear to be a set of solutions in search of problems, rather than a set of clever solutions for substantive problems previously posed in social sciences, bringing to mind the discussion of Leamer on the Tobit model ([Leamer, 1997])."

longtimegoogler 6y ago
That's a long paper and I am interested to read it. From what I have heard, there is more adoption of Pearl's methods in Epidemiology and in Social Sciences, in contrast with Econometrics. To me, it reminds me of Dyson Freeman's, "Birds and Frogs". For most Statistician's and Economists Regression is their goto tool and the Potential Outcome Framework is much more natural to them. You could say the same about the synthetic controls methods you mentioned.
squibbles 6y ago
I find this dialog to be reminiscent of the controversy surrounding Wolfram's A New Kind of Science. Perhaps one could frame it as the theorists versus the pragmatists.
- longtimegoogler 6y ago
  I disagree with that characterization of Pearl's work. Pearl's methods are actually already being used in many fields like Epidemiology.
  - squibbles 6y ago
    I didn't mean to imply that theorists don't produce valuable work, or that theory does not necessarily lead to practice. The distinction I intended is that some people prefer to work off of an underlying model, while others are content so long as they get the results they desire. For example, contrast symbolic processing with artificial neural nets. Symbolic processing has a very solid philosophical basis and it can be used to solve many meaningful problems. Some problems, however, are so complex or nuanced that there are insufficient computing resources to implement a solution based on symbolic processing. Artificial neural networks can be used to address those complex problems, yet we lack the theory (at this time) to really understand the full limits or capabilities of complex artificial neural networks. My reference to Wolfram was about his proposal to cast everything as automata. Even though that does not seem practical for all or even most problems, it provides a certain comfort, much like lambda calculus provides. Beyond the comfort of a solid philosophical grounding, automata also give us a way to approach many problems (such as simulation) in a principled manner. Put another way, some people like to understand first, and use that understanding to discover results. Other people want the results first, and then seek to understand based on the results they found.
currymj 6y ago
i get the sense that back in the 90s, statisticians must have been really rude and dismissive to Judea Pearl, and he might not have gotten over it. it seems very plausible to me that at the time, many people outside CS would have considered a graph to be a type of illustration not a "legitimate" mathematical structure, and wouldn't have understood or cared about computational complexity arguments about performing inference or computing independence. it does seem to be the case that for various reasons, for economics and population-level social science, there's limited advantage to using DAGs. I have also noticed as a causal inference outsider in machine learning land, people use DAGs or SWIGs to write down assumptions, and then translate to potential outcomes to derive estimators. And for people who do automated causal discovery, I haven't heard of anyone trying to use a potential-outcomes-based representation for their system. so i buy Pearl's argument that DAGs are the "right" data structure for representing causality, both for humans and computers. But many social scientists may not have any reason to care about this, because automated causal discovery is hopeless for their applications, and for methodological reasons any reasonable model has to be such that representing things directly as potential outcomes is manageable anyway.
- TeMPOraL 6y ago
  > it does seem to be the case that for various reasons, for economics and population-level social science, there's limited advantage to using DAGs. A tangent: on the one hand, I do think DAGs are the best thing since sliced bread, and I wish more people would embrace them and add them to their toolbox as a fundamental modelling tool. On the other hand, the next step is realization that DAGs aren't sufficient. Lack of cycles is nice for analysis and implementation, but it's also a drawback. The world is running on feedback loops, and yet this seems to be a secret restricted only to specialists deep inside their respective fields. I'd wish the public was more accustomed to working with dynamic models. (I don't think cyclic graphs help much with causality analysis, as presumably cycles would imply time travel.)
  - currymj 6y ago
    yeah, i've wondered about this too. the natural thing to do is to just unroll the models in time. the classic case of this is a hidden Markov Model, but you can easily have a more complicated time-indexed DAG. There are also some interesting connections to reinforcement learning (MDP can be expressed as DAG e.g. https://www.microsoft.com/en-us/research/wp-content/uploads/2013/11/bottou13a.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/..., and then many RL algorithms are equivalent to causal inference estimators). it seems like people deal with equilibria and feedback loops in a pretty ad-hoc way though.

longtimegoogler 6y ago

That's a long paper and I am interested to read it. From what I have heard, there is more adoption of Pearl's methods in Epidemiology and in Social Sciences, in contrast with Econometrics. To me, it reminds me of Dyson Freeman's, "Birds and Frogs". For most Statistician's and Economists Regression is their goto tool and the Potential Outcome Framework is much more natural to them. You could say the same about the synthetic controls methods you mentioned.

squibbles 6y ago

I find this dialog to be reminiscent of the controversy surrounding Wolfram's A New Kind of Science. Perhaps one could frame it as the theorists versus the pragmatists.

longtimegoogler 6y ago
I disagree with that characterization of Pearl's work. Pearl's methods are actually already being used in many fields like Epidemiology.
- squibbles 6y ago
  I didn't mean to imply that theorists don't produce valuable work, or that theory does not necessarily lead to practice. The distinction I intended is that some people prefer to work off of an underlying model, while others are content so long as they get the results they desire. For example, contrast symbolic processing with artificial neural nets. Symbolic processing has a very solid philosophical basis and it can be used to solve many meaningful problems. Some problems, however, are so complex or nuanced that there are insufficient computing resources to implement a solution based on symbolic processing. Artificial neural networks can be used to address those complex problems, yet we lack the theory (at this time) to really understand the full limits or capabilities of complex artificial neural networks. My reference to Wolfram was about his proposal to cast everything as automata. Even though that does not seem practical for all or even most problems, it provides a certain comfort, much like lambda calculus provides. Beyond the comfort of a solid philosophical grounding, automata also give us a way to approach many problems (such as simulation) in a principled manner. Put another way, some people like to understand first, and use that understanding to discover results. Other people want the results first, and then seek to understand based on the results they found.

currymj 6y ago

i get the sense that back in the 90s, statisticians must have been really rude and dismissive to Judea Pearl, and he might not have gotten over it. it seems very plausible to me that at the time, many people outside CS would have considered a graph to be a type of illustration not a "legitimate" mathematical structure, and wouldn't have understood or cared about computational complexity arguments about performing inference or computing independence. it does seem to be the case that for various reasons, for economics and population-level social science, there's limited advantage to using DAGs. I have also noticed as a causal inference outsider in machine learning land, people use DAGs or SWIGs to write down assumptions, and then translate to potential outcomes to derive estimators. And for people who do automated causal discovery, I haven't heard of anyone trying to use a potential-outcomes-based representation for their system. so i buy Pearl's argument that DAGs are the "right" data structure for representing causality, both for humans and computers. But many social scientists may not have any reason to care about this, because automated causal discovery is hopeless for their applications, and for methodological reasons any reasonable model has to be such that representing things directly as potential outcomes is manageable anyway.

TeMPOraL 6y ago
> it does seem to be the case that for various reasons, for economics and population-level social science, there's limited advantage to using DAGs. A tangent: on the one hand, I do think DAGs are the best thing since sliced bread, and I wish more people would embrace them and add them to their toolbox as a fundamental modelling tool. On the other hand, the next step is realization that DAGs aren't sufficient. Lack of cycles is nice for analysis and implementation, but it's also a drawback. The world is running on feedback loops, and yet this seems to be a secret restricted only to specialists deep inside their respective fields. I'd wish the public was more accustomed to working with dynamic models. (I don't think cyclic graphs help much with causality analysis, as presumably cycles would imply time travel.)
- currymj 6y ago
  yeah, i've wondered about this too. the natural thing to do is to just unroll the models in time. the classic case of this is a hidden Markov Model, but you can easily have a more complicated time-indexed DAG. There are also some interesting connections to reinforcement learning (MDP can be expressed as DAG e.g. https://www.microsoft.com/en-us/research/wp-content/uploads/2013/11/bottou13a.pdf https://www.microsoft.com/en-us/research/wp-content/uploads/..., and then many RL algorithms are equivalent to causal inference estimators). it seems like people deal with equilibria and feedback loops in a pretty ad-hoc way though.