6 ms·

New method identifies the root causes of statistical outliers

107 points by jonbaer 4y ago

ubbr 4y ago
Why focus on outliers and not all quantitative causal relationships in the graph like "tell me what causes X"?
- kqr 4y ago
  Because there's nothing in particular that causes what's happening in the central part of the distribution. The centrally located outcomes are the result of combining a myriad of small factors this way and that way. Outliers usually have an identifiable, single cause. This is why in statistical process control these types of outcome are known as "common-cause variation" and "assignable-cause variation".
- jazzyjackson 4y ago
  I think it would be very useful to look at an outlier and know if it's a legitimate measurement that should be kept in the dataset or an aberration that can be safely removed from the dataset. I am not a statistician, so I don't know under what circumstances outliers are usually thrown out.
  - kqr 4y ago
    > I am not a statistician, so I don't know under what circumstances outliers are usually thrown out. As an industrial statistician, I can tell you: way too often. Outliers are the signal among the noise. They indicate something. It is nearly always worth finding out what, instead of removing them. If they indicate a flaw with measurement or the process, then fix that flaw and re-do the measurement or re-run the process. Outlier gone! But in a much more informative way.
    - mcswell 4y ago
      I have absolutely no knowledge of this stuff (in particular of your industry), but I would have thought that most outliers indicate either a measurement error (which you suggest) or a cause that hasn't been taken into account. For the latter, I'm thinking for example about countries whose life expectancy is completely out of line with their per capita income, or some such. Do unexpected causes appear to cause outliers in your field?
- anigbrowl 4y ago
  There's already a technique for that called principal component analysis which works quite well (in many but obviously not all situations).
deleted 4y ago
[deleted]
clircle 4y ago
Very bold of them to assume that outliers even exist. Most statisticians would say that there's not really a difference between an outlier and a non-outlier, but actually the data just have fatter tails than you originally thought, and you need to use a robust modeling method to account for that.
- WastingMyTime89 4y ago
  By definition outliers are just points which differ significantly from the others. Obviously if your distribution has heavy tails they are likely to be there. They could also be noise or simply errors. The only things you know before analysing is that they are different which is why they are outliers.
  - whatshisface 4y ago
    If you carefully read the article, what they're doing is classifying points by how unlikely they'd be according to their model of their system, and they are using that to find inaccuracies in their model. Without a model there is no difference between outliers and "regular" data points, because a data point will always match a distribution made from itself.
    - WastingMyTime89 4y ago
      You don’t need a model for a point to be significantly different. It just needs to have characteristics making it stand out. Outliers are an important concept for the theory of estimators for exemple and that doesn’t involve modelling at all. This discussion is why I regret that people doing CS have so little exposure to frequentist statistics.
      - whatshisface 4y ago
        If you measure 5, 5, 5, 9999999999, then your measured distribution is a peak at 5 and another at 9999999999. You can't say that 9999999999 is a surprising measurement unless you have a theoretical distribution - usually an implicit one - to evaluate at it and find a low probability. In other words, if being surprised means having your expectations violated, an expression of surprise implies that you must have had expectations to violate.
uoaei 4y ago
This "new" method seems to be an old method, indistinguishable from Pearl's do-calculus. What is the innovation here, aside from a new software library? The quantification of each candidate root cause's influence on the outcome? I am surprised the authors found nothing similar throughout the entire corpus of academic research. I must say, Amazon's "science" blog is the most unimpressive of the big tech companies. It churns out PR like the rest, but the others at least have some substance behind them.
- tpoacher 4y ago
  It's ironic that they say they find no good definition for the term, but then go on to cite the same people that use those exact terms to describe the exact same thing: https://www.semanticscholar.org/paper/Causality%2C-Responsibility%2C-and-Blame%3A-A-Approach-Halpern/212ee07c3b76f0fd0ec603584d9f6034cf77b6b9 https://www.semanticscholar.org/paper/Causality%2C-Responsib...
  - svnt 4y ago
    It would be more ironic if they made a mistake like this in a way that wasn’t quite so ego-reinforcing.
- just_boost_it 4y ago
  Every one that I see is a pretty weak summary of a decades old idea, but presented like its new research.
tpoacher 4y ago
Is it me or has this paper gone to great lengths to effectively rediscover the humble (but oft misunderstood) p-value?
deleted 4y ago
[deleted]
emmelaich 4y ago
Relatedly? Amazon books used to have a list of commonly used and rarely used words within the book. Seems they haven't had them for a while, anyone know why? They were interesting, especially for academic titles.
zcw100 4y ago
I've always felt like there was something wrong with how outliers were framed, as though they were wrong, errors, illegitimate, mistakes, noise, etc. They are just, possibly, points from some other process than the one you're interested in. Seems like a very unscientific way to conveniently ignore something that doesn't fit into the way you want it to be. "Look at all this rigorous math I'm doing! Oh, those points? In my expert opinion they're no good. Just delete them"

107 points by jonbaer 4y ago

ubbr 4y ago

Why focus on outliers and not all quantitative causal relationships in the graph like "tell me what causes X"?

kqr 4y ago
Because there's nothing in particular that causes what's happening in the central part of the distribution. The centrally located outcomes are the result of combining a myriad of small factors this way and that way. Outliers usually have an identifiable, single cause. This is why in statistical process control these types of outcome are known as "common-cause variation" and "assignable-cause variation".
jazzyjackson 4y ago
I think it would be very useful to look at an outlier and know if it's a legitimate measurement that should be kept in the dataset or an aberration that can be safely removed from the dataset. I am not a statistician, so I don't know under what circumstances outliers are usually thrown out.
- kqr 4y ago
  > I am not a statistician, so I don't know under what circumstances outliers are usually thrown out. As an industrial statistician, I can tell you: way too often. Outliers are the signal among the noise. They indicate something. It is nearly always worth finding out what, instead of removing them. If they indicate a flaw with measurement or the process, then fix that flaw and re-do the measurement or re-run the process. Outlier gone! But in a much more informative way.
  - mcswell 4y ago
    I have absolutely no knowledge of this stuff (in particular of your industry), but I would have thought that most outliers indicate either a measurement error (which you suggest) or a cause that hasn't been taken into account. For the latter, I'm thinking for example about countries whose life expectancy is completely out of line with their per capita income, or some such. Do unexpected causes appear to cause outliers in your field?
anigbrowl 4y ago
There's already a technique for that called principal component analysis which works quite well (in many but obviously not all situations).

deleted 4y ago

[deleted]

clircle 4y ago

Very bold of them to assume that outliers even exist. Most statisticians would say that there's not really a difference between an outlier and a non-outlier, but actually the data just have fatter tails than you originally thought, and you need to use a robust modeling method to account for that.

WastingMyTime89 4y ago
By definition outliers are just points which differ significantly from the others. Obviously if your distribution has heavy tails they are likely to be there. They could also be noise or simply errors. The only things you know before analysing is that they are different which is why they are outliers.
- whatshisface 4y ago
  If you carefully read the article, what they're doing is classifying points by how unlikely they'd be according to their model of their system, and they are using that to find inaccuracies in their model. Without a model there is no difference between outliers and "regular" data points, because a data point will always match a distribution made from itself.
  - WastingMyTime89 4y ago
    You don’t need a model for a point to be significantly different. It just needs to have characteristics making it stand out. Outliers are an important concept for the theory of estimators for exemple and that doesn’t involve modelling at all. This discussion is why I regret that people doing CS have so little exposure to frequentist statistics.
    - whatshisface 4y ago
      If you measure 5, 5, 5, 9999999999, then your measured distribution is a peak at 5 and another at 9999999999. You can't say that 9999999999 is a surprising measurement unless you have a theoretical distribution - usually an implicit one - to evaluate at it and find a low probability. In other words, if being surprised means having your expectations violated, an expression of surprise implies that you must have had expectations to violate.

uoaei 4y ago

This "new" method seems to be an old method, indistinguishable from Pearl's do-calculus. What is the innovation here, aside from a new software library? The quantification of each candidate root cause's influence on the outcome? I am surprised the authors found nothing similar throughout the entire corpus of academic research. I must say, Amazon's "science" blog is the most unimpressive of the big tech companies. It churns out PR like the rest, but the others at least have some substance behind them.

tpoacher 4y ago
It's ironic that they say they find no good definition for the term, but then go on to cite the same people that use those exact terms to describe the exact same thing: https://www.semanticscholar.org/paper/Causality%2C-Responsibility%2C-and-Blame%3A-A-Approach-Halpern/212ee07c3b76f0fd0ec603584d9f6034cf77b6b9 https://www.semanticscholar.org/paper/Causality%2C-Responsib...
- svnt 4y ago
  It would be more ironic if they made a mistake like this in a way that wasn’t quite so ego-reinforcing.
just_boost_it 4y ago
Every one that I see is a pretty weak summary of a decades old idea, but presented like its new research.

tpoacher 4y ago

Is it me or has this paper gone to great lengths to effectively rediscover the humble (but oft misunderstood) p-value?

deleted 4y ago

[deleted]

emmelaich 4y ago

Relatedly? Amazon books used to have a list of commonly used and rarely used words within the book. Seems they haven't had them for a while, anyone know why? They were interesting, especially for academic titles.

zcw100 4y ago

I've always felt like there was something wrong with how outliers were framed, as though they were wrong, errors, illegitimate, mistakes, noise, etc. They are just, possibly, points from some other process than the one you're interested in. Seems like a very unscientific way to conveniently ignore something that doesn't fit into the way you want it to be. "Look at all this rigorous math I'm doing! Oh, those points? In my expert opinion they're no good. Just delete them"