6 ms·

This statistical test for causation (X->Y) is based on the idea that X and Y each contain noise - noise present in X flows causally to Y but noise present in Y

This statistical test for causation (X->Y) is based on the idea that X and Y each contain noise - noise present in X flows causally to Y but noise present in Y won't flow back to X. But, even if true, it isn't clear that this makes for a good test. For example, it's plausible that Y could have a damping effect and remove noise, which would reverse the results of the test. "They say the additive noise model is up to 80 per cent accurate in correctly determining cause-and-effect." This has been exaggerated by Medium from "accuracies between 65% and 80%" in the original article. But a coin-flip model should be 50% accurate. 65% accuracy is unconvincing. The journal article's conclusion admits that their results are not statistically significant in any sense. As such, the results do not even meet the weakest possible scientific standard. They couldn't reproduce earlier published results in this field (typical of publication bias). Their final paragraph concludes that there is surely a method of doing this, but they just haven't found that method here. In my opinion, the results do not support that conclusion.

Matumio 12y ago
I think we are talking about independent samples of X and Y, not about time series. If X causes Y, then you model y = f(x, u), where U is a random variable independent of X (think: unexplained, e.g. noise). I don't think there can be any dampening effect in this setup. This model is generic: you can find a f(x, u) for any relationship between X and Y. But you may get a much simpler noise model (like additive gaussian noise) in one direction. It's no proof, but a strong hint (Occam's razor). There is also the family of algorithms like IC* and FCI that can recover a causal graph from statistical dependencies between random variables. As output you get a set of causal graphs that are still plausible given the observed dependencies, including constraints about the presence or absence of latent common causes.
- rcthompson 12y ago
  i.i.d. variables are uncorrelated.
  - Matumio 12y ago
    pardon, I meant independent samples of course (edited)
pdabbadabba 12y ago
If the results are 65% accurate, could one not conduct a meta-study to achieve the requisite level of confidence?
- gwern 12y ago
  They don't have that many cause-effect pairs in their dataset (~80, IIRC), so it could be considered insufficient data depending on how skeptical you are. But you don't really need to do a meta-analysis: the cause-effect pair collection is available, so if you dig up more, you can simply rerun the algorithms and see what the new results are with more pairs available. I don't know how hard it would be to collect more examples: on the one hand, any randomized experiment should be a source of such examples, but on the other hand, you might want datasets large enough to definitively establish a causal effect before you go using it as a ground-truth/benchmark in statistical studies like this (and larger datasets tend to be harder to get the individual datapoints for).
gwern 12y ago
> But, even if true, it isn't clear that this makes for a good test. For example, it's plausible that Y could have a damping effect and remove noise, which would reverse the results of the test. It would still be a good test of bivariate causation even if it doesn't handle the case of feedback well. > The journal article's conclusion admits that their results are not statistically significant in any sense. ...and in the same paragraph, they point out that a standard Bonferroni correction would be inappropriate and very conservative, because many of the methods are very similar & the results correlated, and that almost all the variant methodologies (except 1 or 2) do >50%. (If you really want to be a smart-ass about this point, please calculate the odds of almost all the variants performing >50%.)
- learnstats2 12y ago
  The Bonferroni part of the paper is horrible; I almost picked up on that earlier. By my reading, they don't apply Bonferroni, nor any other correction for multiple-testing. The results are already not significant before the correction, so there is no point in making a correction step. Read this part again: they are saying their results are insignificant and would be further away from signficance if they had bothered to apply any correction. Criticising Bonferroni as being too conservative for this purpose might be fashionable, but it's irrelevant and highly misleading here (neither is it evidenced). They haven't applied Bonferroni and they haven't even identified an appropriate replacement - this is an irrelevant comment which seeks to distract.
  - gwern 12y ago
    > The results are already not significant before the correction, so there is no point in making a correction step. Read this part again: they are saying their results are insignificant and would be further away from signficance if they had bothered to apply any correction. Give me a break. Let me go back and quote from the paper: "Whether the performance of a method is significantly better than random guessing is not so clear-cut. For example, the p-value for ANM-pHSIC under the null hypothesis of random guessing is 0.068 on the CEP benchmark, only slightly larger than the popular α = 5% threshold. However, because we tested many different methods, one should correct for multiple testing. Using a conservative Bonferroni correction, the results on a single data set would be far from being significant. In other words, from this study we cannot conclude that the ANM-pHSIC method significantly outperforms random guessing on the CauseEffectPairs benchmark. However, the Bonferroni correction would be overly conservative for our purposes, as many of the methods that we compare are small variations of each other, and their results are clearly dependent. In addition, good performance across data sets increases the significance of the results. Although a proper quantitative evaluation of the significance of our results is a nontrivial exercise, we believe that when combining the results on the CEP benchmark with those on the simulated data, the hypothesis that the good performance of the methods ANM-pHSIC, ANM-HSIC, ANM-PSD and ANM-MML is only due to chance is implausible." A p-value of 0.06 is not horrible, and they do not say it is the smallest p-value obtained. And since they test multiple methods, in different conditions, Bonferonni would divide 0.05 by combinatorially many, which is absurd because clearly these are not independent p-values (which is the assumption behind Bonferroni) and it is not irrelevant at all (and you saying it is suggests you don't understand how Bonferroni works). In fact, you're not even right about "The results are already not significant before the correction" - ANM-pHSIC was almost statistically-significant at the normal threshold, and it didn't perform the best on the full unmodified dataset, that was ANM-HSIC (see figure 11, pg28, left table; or pg27, figure 9). A more reasonable interpretation would be, rather than shoehorning in a Bonferroni where it has no business, to ask whether there's an overall difference of the various algorithms from baseline chance of 50-50, with something like an ANOVA. How would that work? Well, we can estimate the fraction right by eye, use the known 88 pairs, and estimate from there. (There's probably a table of exact results somewhere but I didn't see it.) What is the p-value of all the algorithms, lumped together? Does it suggest outperformance? Why yes it does: R> cep <- read.csv(stdin(),header=TRUE) "Type","Fraction" pHSIC,0.62 HSIC,0.65 ent1sp,0.45 entsp1,0.62 entKDP,0.70 endPSD,0.67 end3nn,0.20 entEdE,0.55 FN,0.49 Gauss,0.51 MML,0.57 entME1,0.60 entME2,0.55 R> cep$Correct <- round(cep$Fraction * 88) R> cep$N <- 88 R> binom.test(sum(cep$Correct), sum(cep$N), p=0.5) Exact binomial test data: sum(cep$Correct) and sum(cep$N) number of successes = 633, number of trials = 1144, p-value = 0.0003429 alternative hypothesis: true probability of success is not equal to 0.5 95 percent confidence interval: 0.523970021 0.582398490 sample estimates: probability of success 0.553321678 So even including all the crappy algorithms, they still outperform. An ANOVA would show the same thing. As the authors say: "the hypothesis that the good performance of the methods ANM-pHSIC, ANM-HSIC, ANM-PSD and ANM-MML is only due to chance is implausible."

Matumio 12y ago

I think we are talking about independent samples of X and Y, not about time series. If X causes Y, then you model y = f(x, u), where U is a random variable independent of X (think: unexplained, e.g. noise). I don't think there can be any dampening effect in this setup. This model is generic: you can find a f(x, u) for any relationship between X and Y. But you may get a much simpler noise model (like additive gaussian noise) in one direction. It's no proof, but a strong hint (Occam's razor). There is also the family of algorithms like IC* and FCI that can recover a causal graph from statistical dependencies between random variables. As output you get a set of causal graphs that are still plausible given the observed dependencies, including constraints about the presence or absence of latent common causes.

rcthompson 12y ago
i.i.d. variables are uncorrelated.
- Matumio 12y ago
  pardon, I meant independent samples of course (edited)

pdabbadabba 12y ago

If the results are 65% accurate, could one not conduct a meta-study to achieve the requisite level of confidence?

gwern 12y ago
They don't have that many cause-effect pairs in their dataset (~80, IIRC), so it could be considered insufficient data depending on how skeptical you are. But you don't really need to do a meta-analysis: the cause-effect pair collection is available, so if you dig up more, you can simply rerun the algorithms and see what the new results are with more pairs available. I don't know how hard it would be to collect more examples: on the one hand, any randomized experiment should be a source of such examples, but on the other hand, you might want datasets large enough to definitively establish a causal effect before you go using it as a ground-truth/benchmark in statistical studies like this (and larger datasets tend to be harder to get the individual datapoints for).

gwern 12y ago

> But, even if true, it isn't clear that this makes for a good test. For example, it's plausible that Y could have a damping effect and remove noise, which would reverse the results of the test. It would still be a good test of bivariate causation even if it doesn't handle the case of feedback well. > The journal article's conclusion admits that their results are not statistically significant in any sense. ...and in the same paragraph, they point out that a standard Bonferroni correction would be inappropriate and very conservative, because many of the methods are very similar & the results correlated, and that almost all the variant methodologies (except 1 or 2) do >50%. (If you really want to be a smart-ass about this point, please calculate the odds of almost all the variants performing >50%.)

learnstats2 12y ago
The Bonferroni part of the paper is horrible; I almost picked up on that earlier. By my reading, they don't apply Bonferroni, nor any other correction for multiple-testing. The results are already not significant before the correction, so there is no point in making a correction step. Read this part again: they are saying their results are insignificant and would be further away from signficance if they had bothered to apply any correction. Criticising Bonferroni as being too conservative for this purpose might be fashionable, but it's irrelevant and highly misleading here (neither is it evidenced). They haven't applied Bonferroni and they haven't even identified an appropriate replacement - this is an irrelevant comment which seeks to distract.
- gwern 12y ago
  > The results are already not significant before the correction, so there is no point in making a correction step. Read this part again: they are saying their results are insignificant and would be further away from signficance if they had bothered to apply any correction. Give me a break. Let me go back and quote from the paper: "Whether the performance of a method is significantly better than random guessing is not so clear-cut. For example, the p-value for ANM-pHSIC under the null hypothesis of random guessing is 0.068 on the CEP benchmark, only slightly larger than the popular α = 5% threshold. However, because we tested many different methods, one should correct for multiple testing. Using a conservative Bonferroni correction, the results on a single data set would be far from being significant. In other words, from this study we cannot conclude that the ANM-pHSIC method significantly outperforms random guessing on the CauseEffectPairs benchmark. However, the Bonferroni correction would be overly conservative for our purposes, as many of the methods that we compare are small variations of each other, and their results are clearly dependent. In addition, good performance across data sets increases the significance of the results. Although a proper quantitative evaluation of the significance of our results is a nontrivial exercise, we believe that when combining the results on the CEP benchmark with those on the simulated data, the hypothesis that the good performance of the methods ANM-pHSIC, ANM-HSIC, ANM-PSD and ANM-MML is only due to chance is implausible." A p-value of 0.06 is not horrible, and they do not say it is the smallest p-value obtained. And since they test multiple methods, in different conditions, Bonferonni would divide 0.05 by combinatorially many, which is absurd because clearly these are not independent p-values (which is the assumption behind Bonferroni) and it is not irrelevant at all (and you saying it is suggests you don't understand how Bonferroni works). In fact, you're not even right about "The results are already not significant before the correction" - ANM-pHSIC was almost statistically-significant at the normal threshold, and it didn't perform the best on the full unmodified dataset, that was ANM-HSIC (see figure 11, pg28, left table; or pg27, figure 9). A more reasonable interpretation would be, rather than shoehorning in a Bonferroni where it has no business, to ask whether there's an overall difference of the various algorithms from baseline chance of 50-50, with something like an ANOVA. How would that work? Well, we can estimate the fraction right by eye, use the known 88 pairs, and estimate from there. (There's probably a table of exact results somewhere but I didn't see it.) What is the p-value of all the algorithms, lumped together? Does it suggest outperformance? Why yes it does: R> cep <- read.csv(stdin(),header=TRUE) "Type","Fraction" pHSIC,0.62 HSIC,0.65 ent1sp,0.45 entsp1,0.62 entKDP,0.70 endPSD,0.67 end3nn,0.20 entEdE,0.55 FN,0.49 Gauss,0.51 MML,0.57 entME1,0.60 entME2,0.55 R> cep$Correct <- round(cep$Fraction * 88) R> cep$N <- 88 R> binom.test(sum(cep$Correct), sum(cep$N), p=0.5) Exact binomial test data: sum(cep$Correct) and sum(cep$N) number of successes = 633, number of trials = 1144, p-value = 0.0003429 alternative hypothesis: true probability of success is not equal to 0.5 95 percent confidence interval: 0.523970021 0.582398490 sample estimates: probability of success 0.553321678 So even including all the crappy algorithms, they still outperform. An ANOVA would show the same thing. As the authors say: "the hypothesis that the good performance of the methods ANM-pHSIC, ANM-HSIC, ANM-PSD and ANM-MML is only due to chance is implausible."