4 ms·

> it sure doesn't feel "fine" for the two of you to disagree on what seems like such a fundamental issue. Instead, this seems like a case where two extremely sm

> it sure doesn't feel "fine" for the two of you to disagree on what seems like such a fundamental issue. Instead, this seems like a case where two extremely smart individuals should be able to reach an common understanding instead of accepting disagreement as a final outcome. Once you've reached the limits of what you can know (after examining all data, worked through all arguments, etc.) this is pretty much the only possible outcome. What works in one situation might not work in others. One person's interpretation of our limited knowledge might very well appear implausible to someone else. I'll admit to being turned off by Pearl's insistence on working with "toy problems". That might be fine for a philosophical discussion, but it's not of much practical value. I want Pearl to write a few empirical papers attacking important issues, then let's have a discussion about putting ideas into practice.

nkurz 8y ago
I have much more sympathy for Pearl. He's constructed "toy" problems that yield different answers depending on the hidden causal structure chosen. He then shows that unless one takes account of the causal structure, it's impossible to correctly answer the problem. The logical conclusion of this is that any technique that doesn't use the causal structure as an explicit assumption is instead using it as an hidden implicit assumption. His hope is that anyone who goes through his exercise will come to the same conclusion. Starting with real-world problems is tricky, because one doesn't know the correct answer in advance, thus it's much harder show the reliance on the assumptions. Consider a parallel with computer programming. A user complains that they fear a program is giving them the wrong answer on a complex real world problem. They report this, and get back the unhelpful answer "Works for me, will not fix". Unable to shake the feeling that the answer is unreliable, they reduce the problem down to a proof of concept that serves as a simple self-contained test case. Two different inputs produce the same answer, but only one of them can be right! But now they are unable to convince the maintainer to even look at the test case, because now the maintainer says "I need to focus on the real world, and don't want to waste my time on toy examples". It's a discouraging position to find oneself in.
- bachmeier 8y ago
  Oh, I'm completely on board with being explicit about any assumptions you're making, and for thinking deeply about the causal relations, and for thinking about the reliability of the analysis. I've been critical of economists for not thinking things through. And I'm definitely on board with teaching using toy problems. The problem is that, as far as I can tell, Pearl doesn't go beyond making points with toy problems. He hasn't done much empirical work (has he published a single empirical paper?) or even read much of the empirical literature he's criticizing as worthless. Ultimately, the question is whether policy and other decisions will be better using a particular framework. The fact that Pearl writes with the aggressive confidence of a Hacker News commenter does not mean he's right.
  - harperlee 8y ago
    But from a mathematical point of view this insistence of focusing on empirical work is a little bit disheartening isn’t it? And frankly Judea Pearl is hardly a random internet tinfoil hat guy, nor is he asking people to invest a massive effort checking his looong work (as we eg often see with random NP-completeness “proofs”, or was discussed with the time investment needed to check Mochizuki’s ABC proof - no, he just asks Gellman to apply his own familiar techniques to a toy problem. It does not sound unreasonable at all.
    - bachmeier 8y ago
      > But from a mathematical point of view this insistence of focusing on empirical work is a little bit disheartening isn’t it? Why? The only thing that matters is the quality of the empirical work. You can get caught up in philosophical debates about the best way to do research. If it has no impact on empirical work, it's useless. That's not to say Pearl's arguments are wrong or that his work is useless. The problem is the incompleteness of his arguments. You can't arrogantly dismiss empirical work just because it's imperfect. There's no reason a priori to expect that Pearl's approach will lead to better decisions. There are many self-proclaimed experts who can point out the flaws in programming language designs, but that doesn't mean they can design a better language, and it doesn't mean existing programming languages are useless. Pearl's approach is not some kind of magic pixie dust that suddenly guarantees your empirical work is more trustworthy. It's unfortunate that Pearl thinks it is, and it prevents him from having a reasonable conversation about the topic.
Fomite 8y ago
Some of these "toy" problems have had pretty major practical effects. For example, reconciling the disagreement between cohort studies and RCTs, in many medical contexts, come down to remarkably simple causal diagrams.
- bachmeier 8y ago
  I don't disagree. I'm not debating the value of Pearl's approach. I'm saying other methods can also be helpful. Pearl does not accept that.
  - kgwgk 8y ago
    What methods? His main point is that "To properly define causal problems, let alone solve them, requires a vocabulary that resides outside the language of probability theory. This means that all the smart and brilliant statisticians who used joint density functions, correlation analysis, contingency tables, ANOVA, Entropy, Risk Ratios, etc., etc., and did not enrich them with either diagrams or counterfactual symbols have been laboring in vain — orthogonally to the question — you can’t answer a question if you have no words to ask it." http://causality.cs.ucla.edu/blog/index.php/2018/06/11/statisticians-reaction-to-the-book-of-why/ http://causality.cs.ucla.edu/blog/index.php/2018/06/11/stati...
fromMars 8y ago
I think Pearl is more a theoretician. I believe some of his descendants have done some work that might qualify. But, I agree that it would be nice if Pearl used real world problems using his methodology.
tunesmith 8y ago
Why does JP have to be the one to write the empirical papers? Why is that a prerequisite here? Judea Pearl is attempting to develop (or has developed) and evangelize the approach for others to use it on empirical problems. So it seems the response is to be in favor of anyone (not just JP) to use it for empirical work. Do we expect every developer of a theory to put it into practice before it is found convincing? Shouldn't the reasoned explanation of a theory be sufficient for someone else to understand and attempt it?

nkurz 8y ago

I have much more sympathy for Pearl. He's constructed "toy" problems that yield different answers depending on the hidden causal structure chosen. He then shows that unless one takes account of the causal structure, it's impossible to correctly answer the problem. The logical conclusion of this is that any technique that doesn't use the causal structure as an explicit assumption is instead using it as an hidden implicit assumption. His hope is that anyone who goes through his exercise will come to the same conclusion. Starting with real-world problems is tricky, because one doesn't know the correct answer in advance, thus it's much harder show the reliance on the assumptions. Consider a parallel with computer programming. A user complains that they fear a program is giving them the wrong answer on a complex real world problem. They report this, and get back the unhelpful answer "Works for me, will not fix". Unable to shake the feeling that the answer is unreliable, they reduce the problem down to a proof of concept that serves as a simple self-contained test case. Two different inputs produce the same answer, but only one of them can be right! But now they are unable to convince the maintainer to even look at the test case, because now the maintainer says "I need to focus on the real world, and don't want to waste my time on toy examples". It's a discouraging position to find oneself in.

bachmeier 8y ago
Oh, I'm completely on board with being explicit about any assumptions you're making, and for thinking deeply about the causal relations, and for thinking about the reliability of the analysis. I've been critical of economists for not thinking things through. And I'm definitely on board with teaching using toy problems. The problem is that, as far as I can tell, Pearl doesn't go beyond making points with toy problems. He hasn't done much empirical work (has he published a single empirical paper?) or even read much of the empirical literature he's criticizing as worthless. Ultimately, the question is whether policy and other decisions will be better using a particular framework. The fact that Pearl writes with the aggressive confidence of a Hacker News commenter does not mean he's right.
- harperlee 8y ago
  But from a mathematical point of view this insistence of focusing on empirical work is a little bit disheartening isn’t it? And frankly Judea Pearl is hardly a random internet tinfoil hat guy, nor is he asking people to invest a massive effort checking his looong work (as we eg often see with random NP-completeness “proofs”, or was discussed with the time investment needed to check Mochizuki’s ABC proof - no, he just asks Gellman to apply his own familiar techniques to a toy problem. It does not sound unreasonable at all.
  - bachmeier 8y ago
    > But from a mathematical point of view this insistence of focusing on empirical work is a little bit disheartening isn’t it? Why? The only thing that matters is the quality of the empirical work. You can get caught up in philosophical debates about the best way to do research. If it has no impact on empirical work, it's useless. That's not to say Pearl's arguments are wrong or that his work is useless. The problem is the incompleteness of his arguments. You can't arrogantly dismiss empirical work just because it's imperfect. There's no reason a priori to expect that Pearl's approach will lead to better decisions. There are many self-proclaimed experts who can point out the flaws in programming language designs, but that doesn't mean they can design a better language, and it doesn't mean existing programming languages are useless. Pearl's approach is not some kind of magic pixie dust that suddenly guarantees your empirical work is more trustworthy. It's unfortunate that Pearl thinks it is, and it prevents him from having a reasonable conversation about the topic.

Fomite 8y ago

Some of these "toy" problems have had pretty major practical effects. For example, reconciling the disagreement between cohort studies and RCTs, in many medical contexts, come down to remarkably simple causal diagrams.

bachmeier 8y ago
I don't disagree. I'm not debating the value of Pearl's approach. I'm saying other methods can also be helpful. Pearl does not accept that.
- kgwgk 8y ago
  What methods? His main point is that "To properly define causal problems, let alone solve them, requires a vocabulary that resides outside the language of probability theory. This means that all the smart and brilliant statisticians who used joint density functions, correlation analysis, contingency tables, ANOVA, Entropy, Risk Ratios, etc., etc., and did not enrich them with either diagrams or counterfactual symbols have been laboring in vain — orthogonally to the question — you can’t answer a question if you have no words to ask it." http://causality.cs.ucla.edu/blog/index.php/2018/06/11/statisticians-reaction-to-the-book-of-why/ http://causality.cs.ucla.edu/blog/index.php/2018/06/11/stati...

fromMars 8y ago

I think Pearl is more a theoretician. I believe some of his descendants have done some work that might qualify. But, I agree that it would be nice if Pearl used real world problems using his methodology.

tunesmith 8y ago

Why does JP have to be the one to write the empirical papers? Why is that a prerequisite here? Judea Pearl is attempting to develop (or has developed) and evangelize the approach for others to use it on empirical problems. So it seems the response is to be in favor of anyone (not just JP) to use it for empirical work. Do we expect every developer of a theory to put it into practice before it is found convincing? Shouldn't the reasoned explanation of a theory be sufficient for someone else to understand and attempt it?