3 ms·

My comment on the blog was (at least initially) eaten by the spam filter, so I reproduce it here. --- JP: There is no way to answer causal questions without s

by nkurz 8y ago

My comment on the blog was (at least initially) eaten by the spam filter, so I reproduce it here. --- JP: There is no way to answer causal questions without snapping out of statistical vocabulary. AG: We disagree. That’s fine. Science is full of disagreements, and there’s lots of room for progress using different methods. I'm only an amateur, but from the outside it sure doesn't feel "fine" for the two of you to disagree on what seems like such a fundamental issue. Instead, this seems like a case where two extremely smart individuals should be able to reach an common understanding instead of accepting disagreement as a final outcome. JP: I have tried to demonstrate it to you in the past several years, but was not able to get you to solve ONE toy problem from beginning to end. AG: For me and many others, one can indeed answer causal questions within statistical vocabulary. Pearl obviously disagrees that standard statistical vocabulary is sufficient to answer all simple causal questions. You seem to think he's wrong. I think you'd be doing a great service to encourage him to formulate such a "toy" question that he thinks is unanswerable without resorting to the do-calculus, which you then try to answer to the audiences' satisfaction using more standard techniques. Maybe the two of you turn out to be in agreement but using different terminology, maybe you are right that his tools are optional, or maybe he's right that they are essential. Any of these outcomes would feel much more satisfactory and productive than agreement to disagree. Please consider offering him a platform with which to make his case.

bachmeier 8y ago
> it sure doesn't feel "fine" for the two of you to disagree on what seems like such a fundamental issue. Instead, this seems like a case where two extremely smart individuals should be able to reach an common understanding instead of accepting disagreement as a final outcome. Once you've reached the limits of what you can know (after examining all data, worked through all arguments, etc.) this is pretty much the only possible outcome. What works in one situation might not work in others. One person's interpretation of our limited knowledge might very well appear implausible to someone else. I'll admit to being turned off by Pearl's insistence on working with "toy problems". That might be fine for a philosophical discussion, but it's not of much practical value. I want Pearl to write a few empirical papers attacking important issues, then let's have a discussion about putting ideas into practice.
- nkurz 8y ago
  I have much more sympathy for Pearl. He's constructed "toy" problems that yield different answers depending on the hidden causal structure chosen. He then shows that unless one takes account of the causal structure, it's impossible to correctly answer the problem. The logical conclusion of this is that any technique that doesn't use the causal structure as an explicit assumption is instead using it as an hidden implicit assumption. His hope is that anyone who goes through his exercise will come to the same conclusion. Starting with real-world problems is tricky, because one doesn't know the correct answer in advance, thus it's much harder show the reliance on the assumptions. Consider a parallel with computer programming. A user complains that they fear a program is giving them the wrong answer on a complex real world problem. They report this, and get back the unhelpful answer "Works for me, will not fix". Unable to shake the feeling that the answer is unreliable, they reduce the problem down to a proof of concept that serves as a simple self-contained test case. Two different inputs produce the same answer, but only one of them can be right! But now they are unable to convince the maintainer to even look at the test case, because now the maintainer says "I need to focus on the real world, and don't want to waste my time on toy examples". It's a discouraging position to find oneself in.
  - bachmeier 8y ago
    Oh, I'm completely on board with being explicit about any assumptions you're making, and for thinking deeply about the causal relations, and for thinking about the reliability of the analysis. I've been critical of economists for not thinking things through. And I'm definitely on board with teaching using toy problems. The problem is that, as far as I can tell, Pearl doesn't go beyond making points with toy problems. He hasn't done much empirical work (has he published a single empirical paper?) or even read much of the empirical literature he's criticizing as worthless. Ultimately, the question is whether policy and other decisions will be better using a particular framework. The fact that Pearl writes with the aggressive confidence of a Hacker News commenter does not mean he's right.
    - harperlee 8y ago
      But from a mathematical point of view this insistence of focusing on empirical work is a little bit disheartening isn’t it? And frankly Judea Pearl is hardly a random internet tinfoil hat guy, nor is he asking people to invest a massive effort checking his looong work (as we eg often see with random NP-completeness “proofs”, or was discussed with the time investment needed to check Mochizuki’s ABC proof - no, he just asks Gellman to apply his own familiar techniques to a toy problem. It does not sound unreasonable at all.
conjectures 8y ago
"you'd be doing a great service to encourage him to formulate such a "toy" question that he thinks is unanswerable without resorting to the do-calculus, which you then try to answer to the audiences' satisfaction using more standard techniques." This would be very helpful indeed. I think a lot of the issue comes down to how idiosyncratic Pearl's work is. It will have to accumulate quite a lot of victories before enough people will bother with it. Until then I suspect Causality will remain something of a statistical Finnegan's Wake. I'll probably read The Book of Why to try and get a better handle on motivation for the technical material.
- deleted 8y ago
  [deleted]
- notafraudster 8y ago
  Thankfully, "Causality: A Primer" is sort of like the CliffNotes to Finnegan's Wake ;)
  - conjectures 8y ago
    Thanks, may check it out. This stuff has long been on my to-do list.
mycorrhizal 8y ago
I think you are overstating Judea Pearl's argument. He doesn't claim it is impossible to answer causal questions (on observational data) without resorting to do-calculus. What he does claim is that to answer causal (interventional or counterfactual) questions you need to define a model and reason on that model in a way that transforms the question into conditional probabilities. Do-calculus provides a framework to help you perform that reasoning, but certainly people have been using things like the backdoor criterion before the invention of do-calculus to answer interventional questions. He might claim that any other method can be reduced to do-calculus. I'm not sure. I do believe at the core of his argument is need for an explicit model.
- nkurz 8y ago
  Yes, that's a better and more accurate phrasing than I used. The issue is that the correctness of the answer depends on the whether the causal structure of the model matches the underlying generative process. Graphs and do-calculus aren't required for this, but can help to make things clearer. In a later comment on the blog post, Pearl links to a paper that describes one of his "toy" examples: http://ftp.cs.ucla.edu/pub/stat_ser/r400-reprint.pdf http://ftp.cs.ucla.edu/pub/stat_ser/r400-reprint.pdf. Section 3 of page 584 is the beginning of the example. I'm sad to say that I'm sufficiently amateur at this that I found even the "toy" example to be at the limits of my reasoning ability, but I thought it still illustrated the argument Pearl is making.
  - mycorrhizal 8y ago
    I absolutely agree if your model is wrong any causal inferences you draw will be wrong. However his framework does provide mechanisms for determining if your data is inconsistent with your model (ie certain independences should or should not exist). So hopefully if your model is wrong your data will tell you so and you can change your model.
    - deleted 8y ago
      [deleted]
- 8y ago
tunesmith 8y ago
I'm not an expert on this stuff at all, but are the toy problems basically formulations of Simpson's paradox? My sense was that if you have a set of a data, where if you cut it into two slices and each slice suggests the same (identical) conclusion, but then if you look at the data overall it suggests a different conclusion, then how do you know which approach is correct? And it seems the only way to know that is to use your judgment about causal factors that generated the data, and the do-calculus helps make that explicit. For instance, when thinking about gender ratios and grad programs, it's more probable that the choice of what grad program to apply to is affected by what gender you are, but it's less probable that the choice of your grad program affects your gender. Well I just reviewed section 3 of page 584 of that pdf that was shared and it seems different, so I probably have a long way to go before I understand this stuff.
- xtacy 8y ago
  Not all "paradoxes" are similar to Simpson's paradox. See this PDF for more similar "paradoxes": https://ftp.cs.ucla.edu/pub/stat_ser/r409-corrected-reprint.pdf https://ftp.cs.ucla.edu/pub/stat_ser/r409-corrected-reprint.... (I quote the word paradox, because they aren't really paradoxes once we have an understanding of what's going on.)

by nkurz 8y ago

bachmeier 8y ago

> it sure doesn't feel "fine" for the two of you to disagree on what seems like such a fundamental issue. Instead, this seems like a case where two extremely smart individuals should be able to reach an common understanding instead of accepting disagreement as a final outcome. Once you've reached the limits of what you can know (after examining all data, worked through all arguments, etc.) this is pretty much the only possible outcome. What works in one situation might not work in others. One person's interpretation of our limited knowledge might very well appear implausible to someone else. I'll admit to being turned off by Pearl's insistence on working with "toy problems". That might be fine for a philosophical discussion, but it's not of much practical value. I want Pearl to write a few empirical papers attacking important issues, then let's have a discussion about putting ideas into practice.

nkurz 8y ago
I have much more sympathy for Pearl. He's constructed "toy" problems that yield different answers depending on the hidden causal structure chosen. He then shows that unless one takes account of the causal structure, it's impossible to correctly answer the problem. The logical conclusion of this is that any technique that doesn't use the causal structure as an explicit assumption is instead using it as an hidden implicit assumption. His hope is that anyone who goes through his exercise will come to the same conclusion. Starting with real-world problems is tricky, because one doesn't know the correct answer in advance, thus it's much harder show the reliance on the assumptions. Consider a parallel with computer programming. A user complains that they fear a program is giving them the wrong answer on a complex real world problem. They report this, and get back the unhelpful answer "Works for me, will not fix". Unable to shake the feeling that the answer is unreliable, they reduce the problem down to a proof of concept that serves as a simple self-contained test case. Two different inputs produce the same answer, but only one of them can be right! But now they are unable to convince the maintainer to even look at the test case, because now the maintainer says "I need to focus on the real world, and don't want to waste my time on toy examples". It's a discouraging position to find oneself in.
- bachmeier 8y ago
  Oh, I'm completely on board with being explicit about any assumptions you're making, and for thinking deeply about the causal relations, and for thinking about the reliability of the analysis. I've been critical of economists for not thinking things through. And I'm definitely on board with teaching using toy problems. The problem is that, as far as I can tell, Pearl doesn't go beyond making points with toy problems. He hasn't done much empirical work (has he published a single empirical paper?) or even read much of the empirical literature he's criticizing as worthless. Ultimately, the question is whether policy and other decisions will be better using a particular framework. The fact that Pearl writes with the aggressive confidence of a Hacker News commenter does not mean he's right.
  - harperlee 8y ago
    But from a mathematical point of view this insistence of focusing on empirical work is a little bit disheartening isn’t it? And frankly Judea Pearl is hardly a random internet tinfoil hat guy, nor is he asking people to invest a massive effort checking his looong work (as we eg often see with random NP-completeness “proofs”, or was discussed with the time investment needed to check Mochizuki’s ABC proof - no, he just asks Gellman to apply his own familiar techniques to a toy problem. It does not sound unreasonable at all.

conjectures 8y ago

"you'd be doing a great service to encourage him to formulate such a "toy" question that he thinks is unanswerable without resorting to the do-calculus, which you then try to answer to the audiences' satisfaction using more standard techniques." This would be very helpful indeed. I think a lot of the issue comes down to how idiosyncratic Pearl's work is. It will have to accumulate quite a lot of victories before enough people will bother with it. Until then I suspect Causality will remain something of a statistical Finnegan's Wake. I'll probably read The Book of Why to try and get a better handle on motivation for the technical material.

deleted 8y ago
[deleted]
notafraudster 8y ago
Thankfully, "Causality: A Primer" is sort of like the CliffNotes to Finnegan's Wake ;)
- conjectures 8y ago
  Thanks, may check it out. This stuff has long been on my to-do list.

mycorrhizal 8y ago

I think you are overstating Judea Pearl's argument. He doesn't claim it is impossible to answer causal questions (on observational data) without resorting to do-calculus. What he does claim is that to answer causal (interventional or counterfactual) questions you need to define a model and reason on that model in a way that transforms the question into conditional probabilities. Do-calculus provides a framework to help you perform that reasoning, but certainly people have been using things like the backdoor criterion before the invention of do-calculus to answer interventional questions. He might claim that any other method can be reduced to do-calculus. I'm not sure. I do believe at the core of his argument is need for an explicit model.

nkurz 8y ago
Yes, that's a better and more accurate phrasing than I used. The issue is that the correctness of the answer depends on the whether the causal structure of the model matches the underlying generative process. Graphs and do-calculus aren't required for this, but can help to make things clearer. In a later comment on the blog post, Pearl links to a paper that describes one of his "toy" examples: http://ftp.cs.ucla.edu/pub/stat_ser/r400-reprint.pdf http://ftp.cs.ucla.edu/pub/stat_ser/r400-reprint.pdf. Section 3 of page 584 is the beginning of the example. I'm sad to say that I'm sufficiently amateur at this that I found even the "toy" example to be at the limits of my reasoning ability, but I thought it still illustrated the argument Pearl is making.
- mycorrhizal 8y ago
  I absolutely agree if your model is wrong any causal inferences you draw will be wrong. However his framework does provide mechanisms for determining if your data is inconsistent with your model (ie certain independences should or should not exist). So hopefully if your model is wrong your data will tell you so and you can change your model.
  - deleted 8y ago
    [deleted]
8y ago

tunesmith 8y ago

I'm not an expert on this stuff at all, but are the toy problems basically formulations of Simpson's paradox? My sense was that if you have a set of a data, where if you cut it into two slices and each slice suggests the same (identical) conclusion, but then if you look at the data overall it suggests a different conclusion, then how do you know which approach is correct? And it seems the only way to know that is to use your judgment about causal factors that generated the data, and the do-calculus helps make that explicit. For instance, when thinking about gender ratios and grad programs, it's more probable that the choice of what grad program to apply to is affected by what gender you are, but it's less probable that the choice of your grad program affects your gender. Well I just reviewed section 3 of page 584 of that pdf that was shared and it seems different, so I probably have a long way to go before I understand this stuff.

xtacy 8y ago
Not all "paradoxes" are similar to Simpson's paradox. See this PDF for more similar "paradoxes": https://ftp.cs.ucla.edu/pub/stat_ser/r409-corrected-reprint.pdf https://ftp.cs.ucla.edu/pub/stat_ser/r409-corrected-reprint.... (I quote the word paradox, because they aren't really paradoxes once we have an understanding of what's going on.)