4 ms·

None of this is causal. For a problem to be, in a statistical sense, causally identified there must be some random or as-if random manipulation of treatment. Th

None of this is causal. For a problem to be, in a statistical sense, causally identified there must be some random or as-if random manipulation of treatment. The two major ways of thinking about causality in statistics are Judea Pearl's DAGS (a representation of causes and effects as an acyclic graph where pathways between variables must be clear of "colliders" which threaten causal validity) and the Neyman-Rubin causal model (also called "potential outcomes", where a unit's outcomes under treatment and control, hypothetically, are considered). One example of an identification strategy here would be to find two languages that are identical in all regards (including in their overall popularity and exposure across the world), but where one was more popular than the other on SO specifically. This would be a matched selection on observables strategy, where selection into "treatment" (popularity on SO) is not globally random, but random conditional on certain pre-treatment covariates, such that the non-treatment potential outcome of both languages would be expected to be the same and the difference between the observed outcomes (the level of popularity of both languages on HN) is only a product of the treatment. Here's a simple inferential threat; the author ascribes the popularity of technology on SO as causing the popularity of a technology on HN. What if, instead, some third common cause caused both, but it caused SO spikes faster than HN spikes. Now, in the world I've described (where the true treatment effect is zero), what statistical test involving comparing SO and HN data, even incorporating temporal ordering, would correctly come up with an estimate of 0? If your answer does not come up with an estimate of 0, then its real-world causal estimate is also presumptively wrong. I also have concerns about how the author measured both treatment and outcome. Overall I think there is an interesting DESCRIPTIVE (non-causal) question somewhere in this article, but it's bogged down by the author trying to apply something they heard about from a Wikipedia article as though it were a substitute for taking causality seriously. We've all heard Alexander Pope's adage that "a little knowledge is a dangerous thing".

chopete 8y ago
>> For a problem to be, in a statistical sense, causally identified there must be some random or as-if random manipulation of treatment. It would be good to see your ideal example of a casual, for non statistics people to understand your long note better.
- nerdponx 8y ago
  You take a random sample of 1,000 white men between the ages of 45 and 55, who have lived in New England for at least 10 years, with no known history of heart disease. Your randomly split them in half. You give half of them a supplement to take every day for 12 months, and you give the other half a placebo. If the number of heart attacks in the placebo sample is greater than in the treatment sample, you have some believable evidence that the supplement can help prevent unexpected heart attacks, at least in white men in their 40s and 50s. The idea is that you've controlled for just about every factor that could affect the rate of unexpected heart attacks, or those factors are evenly distributed throughout both samples because you were careful to sample randomly. Therefore, if there is a difference between the groups, on average, it must be because of the treatment that you introduced to one group and not the other. I'm hand-waving, of course, and I'm sure there are medical researchers out there who will read my study design and laugh at how badly controlled it is. But that should give you the general picture of one comon method used to perform "causal" analysis.
  - notafraudster 8y ago
    Great example. In the design you propose, in expectation, you would have an unbiased causal inference. We would probably want to check for pre-treatment balance between groups to make sure that stochastic (chance) imbalance did not emerge even though the process itself is good. I don't know anything about heart attacks so I don't have the subject matter knowledge here, but imagine that smoking causes heart attacks. If that's the case, although your design should not cause the presence of smokers among treated and control units to systematically vary, maybe it did by chance. We'd want to assess balance. Same with any other potential confounders. Another technique we might use is a blocked (or stratified) random sample. Knowing that there will be both smokers and non-smokers, we recruit two separate samples, and randomize treatment assignment within each. This ensures that smoking status does not predict treatment assignment and guards against some potential threat from overall randomization. We could also mitigate the imbalance that does exist by doing a matched analysis, where each treated unit is paired with a control unit that looks most like him (some control units are reused). Or we could match on propensity scores. Or we could weight on inverse propensity weights. Or we could weight using covariate balancing. Or... My point in doing this info dump is to a) back up nerdponx's example, which is great and b) illustrate how there's a lot to learn about how statisticians have taken the problem of causal analysis seriously and developed techniques appropriate for answering causal questions. People in the CS side of things tend to use Pearl's DAGS for conceptualizing this stuff. I'm in the stats/econ side of things so I use Neyman-Rubin. They're equivalent. Allow me to suggest Rubin and Imbens - Causal Inference for Statistics, Social and Biomedical Sciences as a good textbook that we assign to graduate students learning this stuff. Some of my students tell me the "Causal Inference Mixtape" is popular among people who want less statistical theory and more "what should I do as a practitioner". A virtue of both the resources I just mentioned is that they discuss not just experimental designs but also observational data studies, like the one the original post would have wanted to conduct.
stared 8y ago
Measuring causality is hard. Without randomized control trials, you pretty much need to assume some causal graph structure (plain data is not enough). See: "ML beyond Curve Fitting: An Intro to Causal Inference and do-Calculus" http://www.inference.vc/untitled/ http://www.inference.vc/untitled/ When it comes to this SO vs HN article based on data one can look at correlations. Even if some delayed correlations, then still - it does not imply causation (e.g. HN crowd my by faster to post links to new technologies than SO crowd to questions).
- nerdponx 8y ago
  Sure, but you need to make your assumed graph convincing. As the GP said, Here's a simple inferential threat; the author ascribes the popularity of technology on SO as causing the popularity of a technology on HN. What if, instead, some third common cause caused both, but it caused SO spikes faster than HN spikes. You either need to account for this, or make a convincing argument that no such common cause exists.
  - stared 8y ago
    Sure, fully I agree with that.
l____ 8y ago
Fair point , thank you for the feedback. As I wrote, I did not detect causation (except for a couple of examples in which the null hypothesis of Granger causality test was rejected). The aim of the analysis was to find relationships between data from Hacker News and Stack Overflow without verification of other data sources , which could influence both websites. Nor did I say I had found causation neither did write that I had not. I specifically stated that I tested the hypothesis of Granger causality which does not mean causality. Even if the main question of the article was causality, I could not answer it (and I stated this in the text). Nevertheless, I agree that I should have probably been more careful with the term "causality" so thank you once again for pointing this out.

chopete 8y ago

>> For a problem to be, in a statistical sense, causally identified there must be some random or as-if random manipulation of treatment. It would be good to see your ideal example of a casual, for non statistics people to understand your long note better.

nerdponx 8y ago
You take a random sample of 1,000 white men between the ages of 45 and 55, who have lived in New England for at least 10 years, with no known history of heart disease. Your randomly split them in half. You give half of them a supplement to take every day for 12 months, and you give the other half a placebo. If the number of heart attacks in the placebo sample is greater than in the treatment sample, you have some believable evidence that the supplement can help prevent unexpected heart attacks, at least in white men in their 40s and 50s. The idea is that you've controlled for just about every factor that could affect the rate of unexpected heart attacks, or those factors are evenly distributed throughout both samples because you were careful to sample randomly. Therefore, if there is a difference between the groups, on average, it must be because of the treatment that you introduced to one group and not the other. I'm hand-waving, of course, and I'm sure there are medical researchers out there who will read my study design and laugh at how badly controlled it is. But that should give you the general picture of one comon method used to perform "causal" analysis.
- notafraudster 8y ago
  Great example. In the design you propose, in expectation, you would have an unbiased causal inference. We would probably want to check for pre-treatment balance between groups to make sure that stochastic (chance) imbalance did not emerge even though the process itself is good. I don't know anything about heart attacks so I don't have the subject matter knowledge here, but imagine that smoking causes heart attacks. If that's the case, although your design should not cause the presence of smokers among treated and control units to systematically vary, maybe it did by chance. We'd want to assess balance. Same with any other potential confounders. Another technique we might use is a blocked (or stratified) random sample. Knowing that there will be both smokers and non-smokers, we recruit two separate samples, and randomize treatment assignment within each. This ensures that smoking status does not predict treatment assignment and guards against some potential threat from overall randomization. We could also mitigate the imbalance that does exist by doing a matched analysis, where each treated unit is paired with a control unit that looks most like him (some control units are reused). Or we could match on propensity scores. Or we could weight on inverse propensity weights. Or we could weight using covariate balancing. Or... My point in doing this info dump is to a) back up nerdponx's example, which is great and b) illustrate how there's a lot to learn about how statisticians have taken the problem of causal analysis seriously and developed techniques appropriate for answering causal questions. People in the CS side of things tend to use Pearl's DAGS for conceptualizing this stuff. I'm in the stats/econ side of things so I use Neyman-Rubin. They're equivalent. Allow me to suggest Rubin and Imbens - Causal Inference for Statistics, Social and Biomedical Sciences as a good textbook that we assign to graduate students learning this stuff. Some of my students tell me the "Causal Inference Mixtape" is popular among people who want less statistical theory and more "what should I do as a practitioner". A virtue of both the resources I just mentioned is that they discuss not just experimental designs but also observational data studies, like the one the original post would have wanted to conduct.

stared 8y ago

Measuring causality is hard. Without randomized control trials, you pretty much need to assume some causal graph structure (plain data is not enough). See: "ML beyond Curve Fitting: An Intro to Causal Inference and do-Calculus" http://www.inference.vc/untitled/ http://www.inference.vc/untitled/ When it comes to this SO vs HN article based on data one can look at correlations. Even if some delayed correlations, then still - it does not imply causation (e.g. HN crowd my by faster to post links to new technologies than SO crowd to questions).

nerdponx 8y ago
Sure, but you need to make your assumed graph convincing. As the GP said, Here's a simple inferential threat; the author ascribes the popularity of technology on SO as causing the popularity of a technology on HN. What if, instead, some third common cause caused both, but it caused SO spikes faster than HN spikes. You either need to account for this, or make a convincing argument that no such common cause exists.
- stared 8y ago
  Sure, fully I agree with that.

l____ 8y ago

Fair point , thank you for the feedback. As I wrote, I did not detect causation (except for a couple of examples in which the null hypothesis of Granger causality test was rejected). The aim of the analysis was to find relationships between data from Hacker News and Stack Overflow without verification of other data sources , which could influence both websites. Nor did I say I had found causation neither did write that I had not. I specifically stated that I tested the hypothesis of Granger causality which does not mean causality. Even if the main question of the article was causality, I could not answer it (and I stated this in the text). Nevertheless, I agree that I should have probably been more careful with the term "causality" so thank you once again for pointing this out.