3 ms·

What they did was fine with this setup. You might enjoy reading this article that covers the very perceived issue you are having with the research: https://tow

by asdff 4y ago

What they did was fine with this setup. You might enjoy reading this article that covers the very perceived issue you are having with the research: https://towardsdatascience.com/establishing-causality-part-1-49cb9230884c https://towardsdatascience.com/establishing-causality-part-1...

bjornsing 4y ago
As I understand it a randomized controlled trial (RCT) can determine causation. But I don’t think “what they did with this setup” was an RCT. A real RCT in this context should IMHO assign new cases randomly to a “treatment” or “control” group. In this setup they assign legal precedents randomly, but all new cases are effectively under “treatment”. That’s not the same thing.
- asdff 4y ago
  How are new cases effectively under treatment? They identified 150 decisions, wrote articles for each, and assigned half randomly to treatment or control. they did not consider newer cases beyond this set of 150. An RCT is a type of randomized experiment, but all determine causation: https://en.wikipedia.org/wiki/Randomized_experiment https://en.wikipedia.org/wiki/Randomized_experiment
  - bjornsing 4y ago
    You’re right. It’s the legal precedents / decisions that are the “patients” here. The new cases (that cite those precedents / decisions) are just… let’s say “test results”. So yes, this is a way to determine causation in relation to what legal precedents / decisions will be cited in new cases depending on the contents of Wikipedia. I was under the (perhaps mistaken) impression that they claimed a causal link between their Wikipedia articles and the outcome of new cases. The chosen setup cannot IMHO prove such a causal relationship, in the strict statistical sense. That would require randomizing new cases and applying the “treatment” (new Wikipedia articles) only to some of them.

by asdff 4y ago

bjornsing 4y ago

As I understand it a randomized controlled trial (RCT) can determine causation. But I don’t think “what they did with this setup” was an RCT. A real RCT in this context should IMHO assign new cases randomly to a “treatment” or “control” group. In this setup they assign legal precedents randomly, but all new cases are effectively under “treatment”. That’s not the same thing.

asdff 4y ago
How are new cases effectively under treatment? They identified 150 decisions, wrote articles for each, and assigned half randomly to treatment or control. they did not consider newer cases beyond this set of 150. An RCT is a type of randomized experiment, but all determine causation: https://en.wikipedia.org/wiki/Randomized_experiment https://en.wikipedia.org/wiki/Randomized_experiment
- bjornsing 4y ago
  You’re right. It’s the legal precedents / decisions that are the “patients” here. The new cases (that cite those precedents / decisions) are just… let’s say “test results”. So yes, this is a way to determine causation in relation to what legal precedents / decisions will be cited in new cases depending on the contents of Wikipedia. I was under the (perhaps mistaken) impression that they claimed a causal link between their Wikipedia articles and the outcome of new cases. The chosen setup cannot IMHO prove such a causal relationship, in the strict statistical sense. That would require randomizing new cases and applying the “treatment” (new Wikipedia articles) only to some of them.