4 ms·

https://en.wikipedia.org/wiki/Berkson%27s_paradox https://en.wikipedia.org/wiki/Berkson%27s_paradox is also one to be aware of. There are lots of ways for error

• pdonis 3y ago
  > It is common for data to suggest the opposite of the truth. Actually, I think the best takeaway from phenomena like these is that just doing statistics on a set of data can't tell you "the truth". If you don't understand the actual causal factors in play, your knowledge is very limited, no matter how much data you have or how many different ways you slice the statistics. For example, in the UC Berkeley case described in the Simpson's Paradox article, the data actually doesn't tell you anything useful about "bias" in the sense of "something people are doing that they should do differently to make the admissions process fairer". It doesn't even tell you where to look for possible "bias" without knowing more about the admissions process: it is controlled primarily by departments or by the university as a whole?
  • staunton 3y ago
    > just doing statistics on a set of data can't tell you "the truth". If you don't understand the actual causal factors in play, your knowledge is very limited I would argue that ultimately, all your knowledge and understanding comes from "doing statistics on data". Maybe the statistics is done by sloppy slurpy things in the brain instead of in R, and maybe it's actually mathematically unsound most of the time, but it's still some sort of statistics.
    • yccs27 3y ago
      I think the key difference is between statistics on passively collected data vs results from active experiments. The former will only ever show correlations, while the latter can prove causal results from the actions of the experimenter.
      • pdonis 3y ago
        Also, results from active experiments aren't limited to statistics. You can set up experiments to have discrete results, where no statistics is required to test a hypothesis. For example, the GHZ experiment [1] can rule out local hidden variable models and confirm QM predictions with no statistics at all: the two different models make contradictory predictions with no continuous variation between them. [1] https://en.wikipedia.org/wiki/GHZ_experiment https://en.wikipedia.org/wiki/GHZ_experiment
        • staunton 3y ago
          If you read a good experimental paper on GHZ, you will find quite a bit of statistics.
          • pdonis 3y ago
            Sure, but that doesn't contradict what I said. From the Wikipedia article I referenced: "For specific combinations of orientations, perfect (rather than statistical) correlations between the three polarizations are predicted by both local hidden variable theory (aka "local realism") and by quantum mechanical theory, and the predictions may be contradictory." "Perfect" correlations means, as the parenthetical comment shows, "doesn't require statistics to check".
            • staunton 3y ago
              You said > the GHZ experiment [1] can rule out local hidden variable models and confirm QM predictions with no statistics at all However, one needs to use statistics to even show GHZ works. That does sound contradictory to me. The correlations you get in experiments are never perfect and in this case they can be pretty far from perfect.
              • pdonis 3y ago
                > one needs to use statistics to even show GHZ works Not for the particular cases described in the quote I gave. For a complete verification of all the GHZ theorem's predictions, yes, you need to do statistics, because some of those predictions are probabilistic. > The correlations you get in experiments are never perfect In some cases, like the ones described in the quote I gave, it isn't a matter of correlations. You have contradictory results predicted by two different models, each prediction being 100% certain according to the model. You don't need any statistics to test that: just do one single run and see which way it comes out.
  • andirk 3y ago
    Is the UC Berkeley case a good example of the importance of normalizing data before analyzing? Where things need to be put on a level playing field and handicaps applied to remove auxiliary noise.
    • pdonis 3y ago
      Normalizing data doesn't fix the issue in the UC Berkeley case, because you still have to pick what to normalize over: do you normalize over the entire university, or separately over each department? The answer to questions like that can't be found in the data. You have to go look at how the university admission process actually works, and what roles the university vs. the individual departments play in it.
• o11c 3y ago
  Of course, an empiricist can just point to https://en.wikipedia.org/wiki/Goodhart%27s_law https://en.wikipedia.org/wiki/Goodhart%27s_law
  • tunesmith 3y ago
    > Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes. It may be true in some cases, but I think the "Any" is disproven by the existence of proper scoring rules.
    • o11c 3y ago
      It's a very big assumption that such rules can, let alone do, exist in the first place.
• JacobAldridge 3y ago
  As the aphorism goes, there are Lies, Damned Lies, and Statistics.