5 ms·

Why focus on outliers and not all quantitative causal relationships in the graph like "tell me what causes X"?

• kqr 4y ago
  Because there's nothing in particular that causes what's happening in the central part of the distribution. The centrally located outcomes are the result of combining a myriad of small factors this way and that way. Outliers usually have an identifiable, single cause. This is why in statistical process control these types of outcome are known as "common-cause variation" and "assignable-cause variation".
• jazzyjackson 4y ago
  I think it would be very useful to look at an outlier and know if it's a legitimate measurement that should be kept in the dataset or an aberration that can be safely removed from the dataset. I am not a statistician, so I don't know under what circumstances outliers are usually thrown out.
  • kqr 4y ago
    > I am not a statistician, so I don't know under what circumstances outliers are usually thrown out. As an industrial statistician, I can tell you: way too often. Outliers are the signal among the noise. They indicate something. It is nearly always worth finding out what, instead of removing them. If they indicate a flaw with measurement or the process, then fix that flaw and re-do the measurement or re-run the process. Outlier gone! But in a much more informative way.
    • mcswell 4y ago
      I have absolutely no knowledge of this stuff (in particular of your industry), but I would have thought that most outliers indicate either a measurement error (which you suggest) or a cause that hasn't been taken into account. For the latter, I'm thinking for example about countries whose life expectancy is completely out of line with their per capita income, or some such. Do unexpected causes appear to cause outliers in your field?
• anigbrowl 4y ago
  There's already a technique for that called principal component analysis which works quite well (in many but obviously not all situations).