4 ms·

"because randomized experiments are the gold standard for this type of research, we know the effect we are seeing is causation, not just correlation" No, they

by pdonis 4y ago

"because randomized experiments are the gold standard for this type of research, we know the effect we are seeing is causation, not just correlation" No, they don't. Correlation is not causation, even if you see it in a randomized experiment. With shoddy reasoning like this, it's no wonder science has a replication crisis.

Spivak 4y ago
https://nulib.github.io/moderndive_book/7-causality.html https://nulib.github.io/moderndive_book/7-causality.html https://bolt.mph.ufl.edu/6050-6052/unit-2/causation-and-experiments/ https://bolt.mph.ufl.edu/6050-6052/unit-2/causation-and-expe... https://towardsdatascience.com/establishing-causality-part-1-49cb9230884c https://towardsdatascience.com/establishing-causality-part-1... https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6235704/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6235704/ https://escholarship.org/uc/item/42v4w8k1 https://escholarship.org/uc/item/42v4w8k1 http://ippsr.msu.edu/public-policy/michigan-wonk-blog/randomized-controlled-trials-public-policy http://ippsr.msu.edu/public-policy/michigan-wonk-blog/random... https://www.cs.cornell.edu/courses/cs1380/2018sp/textbook/chapters/16/3/causality.html https://www.cs.cornell.edu/courses/cs1380/2018sp/textbook/ch... I’ll leave open the possibility that it’s everyone else that’s wrong but RCTs are used to establish causality and are as much “proof” as you’re gonna get in science. Hell ya know what I’ll just let the actual paper explain it. > The second, more important advantage of randomized field experiments is that they can distinguish causation from correlation. The ability to prove causal relationships derives from the combination of two characteristics. The first is having a control group, that is, a group unaffected by the intervention (in our case, publication of a Wikipedia article on the topic) that can be used as a counterfactual to estimate the size of causal effects. The second is randomization, that is, random assignment into the control and intervention groups. With sufficient data and a sound experimental design, the experiment can reduce the probability of being misled by correlation or noise to whatever arbitrarily small value is desired.
- pdonis 4y ago
  > RCTs are used to establish causality and are as much “proof” as you’re gonna get in science. No, they're not. The real "gold standard" in science--the standard that prevails in, for example, physics or chemistry--is a controlled experiment. Not just a "randomized controlled trial", but a controlled experiment, where you can actually dictate exactly what state the things you are going to experiment on start out in. And the eventual output of controlled experiments is a predictive model--a model that can predict, accurately, what will happen if you run further experiments. That is what it takes to truly "establish causality". But in most other domains, including the one under study here, controlled experiments simply cannot be done and predictive models with any kind of accuracy simply don't exist. The correct response to that unfortunate fact is to realize that we can never achieve the same level of confidence in these other domains as we can in domains like physics or chemistry where we can do controlled experiments. Unfortunately, the response "science" has settled on instead is to pretend that it doesn't matter--that because we can't do controlled experiments in these other domains, the universe will somehow magically lower its standards of what it takes to achieve the level of confidence we want. But the universe doesn't care what we can or can't achieve.
  - asdff 4y ago
    They did do a controlled experiment here. They had articles in a treatment group and articles in a controlled group. What are the shortcomings you have in mind when you say that "controlled experiments simply cannot be done" when it appears that they have done a controlled experiment?
    - DiggyJohnson 4y ago
      Because the experiment itself isn’t closed.
      - asdff 4y ago
        What do you mean by closed?
        4y ago
asdff 4y ago
What they did was fine with this setup. You might enjoy reading this article that covers the very perceived issue you are having with the research: https://towardsdatascience.com/establishing-causality-part-1-49cb9230884c https://towardsdatascience.com/establishing-causality-part-1...
- bjornsing 4y ago
  As I understand it a randomized controlled trial (RCT) can determine causation. But I don’t think “what they did with this setup” was an RCT. A real RCT in this context should IMHO assign new cases randomly to a “treatment” or “control” group. In this setup they assign legal precedents randomly, but all new cases are effectively under “treatment”. That’s not the same thing.
  - asdff 4y ago
    How are new cases effectively under treatment? They identified 150 decisions, wrote articles for each, and assigned half randomly to treatment or control. they did not consider newer cases beyond this set of 150. An RCT is a type of randomized experiment, but all determine causation: https://en.wikipedia.org/wiki/Randomized_experiment https://en.wikipedia.org/wiki/Randomized_experiment
    - bjornsing 4y ago
      You’re right. It’s the legal precedents / decisions that are the “patients” here. The new cases (that cite those precedents / decisions) are just… let’s say “test results”. So yes, this is a way to determine causation in relation to what legal precedents / decisions will be cited in new cases depending on the contents of Wikipedia. I was under the (perhaps mistaken) impression that they claimed a causal link between their Wikipedia articles and the outcome of new cases. The chosen setup cannot IMHO prove such a causal relationship, in the strict statistical sense. That would require randomizing new cases and applying the “treatment” (new Wikipedia articles) only to some of them.

by pdonis 4y ago

Spivak 4y ago

https://nulib.github.io/moderndive_book/7-causality.html https://nulib.github.io/moderndive_book/7-causality.html https://bolt.mph.ufl.edu/6050-6052/unit-2/causation-and-experiments/ https://bolt.mph.ufl.edu/6050-6052/unit-2/causation-and-expe... https://towardsdatascience.com/establishing-causality-part-1-49cb9230884c https://towardsdatascience.com/establishing-causality-part-1... https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6235704/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6235704/ https://escholarship.org/uc/item/42v4w8k1 https://escholarship.org/uc/item/42v4w8k1 http://ippsr.msu.edu/public-policy/michigan-wonk-blog/randomized-controlled-trials-public-policy http://ippsr.msu.edu/public-policy/michigan-wonk-blog/random... https://www.cs.cornell.edu/courses/cs1380/2018sp/textbook/chapters/16/3/causality.html https://www.cs.cornell.edu/courses/cs1380/2018sp/textbook/ch... I’ll leave open the possibility that it’s everyone else that’s wrong but RCTs are used to establish causality and are as much “proof” as you’re gonna get in science. Hell ya know what I’ll just let the actual paper explain it. > The second, more important advantage of randomized field experiments is that they can distinguish causation from correlation. The ability to prove causal relationships derives from the combination of two characteristics. The first is having a control group, that is, a group unaffected by the intervention (in our case, publication of a Wikipedia article on the topic) that can be used as a counterfactual to estimate the size of causal effects. The second is randomization, that is, random assignment into the control and intervention groups. With sufficient data and a sound experimental design, the experiment can reduce the probability of being misled by correlation or noise to whatever arbitrarily small value is desired.

pdonis 4y ago
> RCTs are used to establish causality and are as much “proof” as you’re gonna get in science. No, they're not. The real "gold standard" in science--the standard that prevails in, for example, physics or chemistry--is a controlled experiment. Not just a "randomized controlled trial", but a controlled experiment, where you can actually dictate exactly what state the things you are going to experiment on start out in. And the eventual output of controlled experiments is a predictive model--a model that can predict, accurately, what will happen if you run further experiments. That is what it takes to truly "establish causality". But in most other domains, including the one under study here, controlled experiments simply cannot be done and predictive models with any kind of accuracy simply don't exist. The correct response to that unfortunate fact is to realize that we can never achieve the same level of confidence in these other domains as we can in domains like physics or chemistry where we can do controlled experiments. Unfortunately, the response "science" has settled on instead is to pretend that it doesn't matter--that because we can't do controlled experiments in these other domains, the universe will somehow magically lower its standards of what it takes to achieve the level of confidence we want. But the universe doesn't care what we can or can't achieve.
- asdff 4y ago
  They did do a controlled experiment here. They had articles in a treatment group and articles in a controlled group. What are the shortcomings you have in mind when you say that "controlled experiments simply cannot be done" when it appears that they have done a controlled experiment?
  - DiggyJohnson 4y ago
    Because the experiment itself isn’t closed.
    - asdff 4y ago
      What do you mean by closed?
      - 4y ago

asdff 4y ago

What they did was fine with this setup. You might enjoy reading this article that covers the very perceived issue you are having with the research: https://towardsdatascience.com/establishing-causality-part-1-49cb9230884c https://towardsdatascience.com/establishing-causality-part-1...

bjornsing 4y ago
As I understand it a randomized controlled trial (RCT) can determine causation. But I don’t think “what they did with this setup” was an RCT. A real RCT in this context should IMHO assign new cases randomly to a “treatment” or “control” group. In this setup they assign legal precedents randomly, but all new cases are effectively under “treatment”. That’s not the same thing.
- asdff 4y ago
  How are new cases effectively under treatment? They identified 150 decisions, wrote articles for each, and assigned half randomly to treatment or control. they did not consider newer cases beyond this set of 150. An RCT is a type of randomized experiment, but all determine causation: https://en.wikipedia.org/wiki/Randomized_experiment https://en.wikipedia.org/wiki/Randomized_experiment
  - bjornsing 4y ago
    You’re right. It’s the legal precedents / decisions that are the “patients” here. The new cases (that cite those precedents / decisions) are just… let’s say “test results”. So yes, this is a way to determine causation in relation to what legal precedents / decisions will be cited in new cases depending on the contents of Wikipedia. I was under the (perhaps mistaken) impression that they claimed a causal link between their Wikipedia articles and the outcome of new cases. The chosen setup cannot IMHO prove such a causal relationship, in the strict statistical sense. That would require randomizing new cases and applying the “treatment” (new Wikipedia articles) only to some of them.