4 ms·

> It is common for data to suggest the opposite of the truth. Actually, I think the best takeaway from phenomena like these is that just doing statistics on a

by pdonis 3y ago

> It is common for data to suggest the opposite of the truth. Actually, I think the best takeaway from phenomena like these is that just doing statistics on a set of data can't tell you "the truth". If you don't understand the actual causal factors in play, your knowledge is very limited, no matter how much data you have or how many different ways you slice the statistics. For example, in the UC Berkeley case described in the Simpson's Paradox article, the data actually doesn't tell you anything useful about "bias" in the sense of "something people are doing that they should do differently to make the admissions process fairer". It doesn't even tell you where to look for possible "bias" without knowing more about the admissions process: it is controlled primarily by departments or by the university as a whole?

staunton 3y ago
> just doing statistics on a set of data can't tell you "the truth". If you don't understand the actual causal factors in play, your knowledge is very limited I would argue that ultimately, all your knowledge and understanding comes from "doing statistics on data". Maybe the statistics is done by sloppy slurpy things in the brain instead of in R, and maybe it's actually mathematically unsound most of the time, but it's still some sort of statistics.
- yccs27 3y ago
  I think the key difference is between statistics on passively collected data vs results from active experiments. The former will only ever show correlations, while the latter can prove causal results from the actions of the experimenter.
  - pdonis 3y ago
    Also, results from active experiments aren't limited to statistics. You can set up experiments to have discrete results, where no statistics is required to test a hypothesis. For example, the GHZ experiment [1] can rule out local hidden variable models and confirm QM predictions with no statistics at all: the two different models make contradictory predictions with no continuous variation between them. [1] https://en.wikipedia.org/wiki/GHZ_experiment https://en.wikipedia.org/wiki/GHZ_experiment
    - staunton 3y ago
      If you read a good experimental paper on GHZ, you will find quite a bit of statistics.
      - pdonis 3y ago
        Sure, but that doesn't contradict what I said. From the Wikipedia article I referenced: "For specific combinations of orientations, perfect (rather than statistical) correlations between the three polarizations are predicted by both local hidden variable theory (aka "local realism") and by quantum mechanical theory, and the predictions may be contradictory." "Perfect" correlations means, as the parenthetical comment shows, "doesn't require statistics to check".
        staunton 3y ago
        You said > the GHZ experiment [1] can rule out local hidden variable models and confirm QM predictions with no statistics at all However, one needs to use statistics to even show GHZ works. That does sound contradictory to me. The correlations you get in experiments are never perfect and in this case they can be pretty far from perfect.
        pdonis 3y ago
        > one needs to use statistics to even show GHZ works Not for the particular cases described in the quote I gave. For a complete verification of all the GHZ theorem's predictions, yes, you need to do statistics, because some of those predictions are probabilistic. > The correlations you get in experiments are never perfect In some cases, like the ones described in the quote I gave, it isn't a matter of correlations. You have contradictory results predicted by two different models, each prediction being 100% certain according to the model. You don't need any statistics to test that: just do one single run and see which way it comes out.
andirk 3y ago
Is the UC Berkeley case a good example of the importance of normalizing data before analyzing? Where things need to be put on a level playing field and handicaps applied to remove auxiliary noise.
- pdonis 3y ago
  Normalizing data doesn't fix the issue in the UC Berkeley case, because you still have to pick what to normalize over: do you normalize over the entire university, or separately over each department? The answer to questions like that can't be found in the data. You have to go look at how the university admission process actually works, and what roles the university vs. the individual departments play in it.

by pdonis 3y ago

staunton 3y ago

> just doing statistics on a set of data can't tell you "the truth". If you don't understand the actual causal factors in play, your knowledge is very limited I would argue that ultimately, all your knowledge and understanding comes from "doing statistics on data". Maybe the statistics is done by sloppy slurpy things in the brain instead of in R, and maybe it's actually mathematically unsound most of the time, but it's still some sort of statistics.

yccs27 3y ago
I think the key difference is between statistics on passively collected data vs results from active experiments. The former will only ever show correlations, while the latter can prove causal results from the actions of the experimenter.
- pdonis 3y ago
  Also, results from active experiments aren't limited to statistics. You can set up experiments to have discrete results, where no statistics is required to test a hypothesis. For example, the GHZ experiment [1] can rule out local hidden variable models and confirm QM predictions with no statistics at all: the two different models make contradictory predictions with no continuous variation between them. [1] https://en.wikipedia.org/wiki/GHZ_experiment https://en.wikipedia.org/wiki/GHZ_experiment
  - staunton 3y ago
    If you read a good experimental paper on GHZ, you will find quite a bit of statistics.
    - pdonis 3y ago
      Sure, but that doesn't contradict what I said. From the Wikipedia article I referenced: "For specific combinations of orientations, perfect (rather than statistical) correlations between the three polarizations are predicted by both local hidden variable theory (aka "local realism") and by quantum mechanical theory, and the predictions may be contradictory." "Perfect" correlations means, as the parenthetical comment shows, "doesn't require statistics to check".
      - staunton 3y ago
        You said > the GHZ experiment [1] can rule out local hidden variable models and confirm QM predictions with no statistics at all However, one needs to use statistics to even show GHZ works. That does sound contradictory to me. The correlations you get in experiments are never perfect and in this case they can be pretty far from perfect.
        pdonis 3y ago
        > one needs to use statistics to even show GHZ works Not for the particular cases described in the quote I gave. For a complete verification of all the GHZ theorem's predictions, yes, you need to do statistics, because some of those predictions are probabilistic. > The correlations you get in experiments are never perfect In some cases, like the ones described in the quote I gave, it isn't a matter of correlations. You have contradictory results predicted by two different models, each prediction being 100% certain according to the model. You don't need any statistics to test that: just do one single run and see which way it comes out.

andirk 3y ago

Is the UC Berkeley case a good example of the importance of normalizing data before analyzing? Where things need to be put on a level playing field and handicaps applied to remove auxiliary noise.

pdonis 3y ago
Normalizing data doesn't fix the issue in the UC Berkeley case, because you still have to pick what to normalize over: do you normalize over the entire university, or separately over each department? The answer to questions like that can't be found in the data. You have to go look at how the university admission process actually works, and what roles the university vs. the individual departments play in it.