6 ms·

Very bold of them to assume that outliers even exist. Most statisticians would say that there's not really a difference between an outlier and a non-outlier, bu

Very bold of them to assume that outliers even exist. Most statisticians would say that there's not really a difference between an outlier and a non-outlier, but actually the data just have fatter tails than you originally thought, and you need to use a robust modeling method to account for that.

WastingMyTime89 4y ago
By definition outliers are just points which differ significantly from the others. Obviously if your distribution has heavy tails they are likely to be there. They could also be noise or simply errors. The only things you know before analysing is that they are different which is why they are outliers.
- whatshisface 4y ago
  If you carefully read the article, what they're doing is classifying points by how unlikely they'd be according to their model of their system, and they are using that to find inaccuracies in their model. Without a model there is no difference between outliers and "regular" data points, because a data point will always match a distribution made from itself.
  - WastingMyTime89 4y ago
    You don’t need a model for a point to be significantly different. It just needs to have characteristics making it stand out. Outliers are an important concept for the theory of estimators for exemple and that doesn’t involve modelling at all. This discussion is why I regret that people doing CS have so little exposure to frequentist statistics.
    - whatshisface 4y ago
      If you measure 5, 5, 5, 9999999999, then your measured distribution is a peak at 5 and another at 9999999999. You can't say that 9999999999 is a surprising measurement unless you have a theoretical distribution - usually an implicit one - to evaluate at it and find a low probability. In other words, if being surprised means having your expectations violated, an expression of surprise implies that you must have had expectations to violate.
      - strangedejavu 4y ago
        This argument doesn't make much sense to me. You can definitely associate a probability based just on the measured data. What if you measured 5, 5, 5, 5, 5, 5,5 (1000 more 5s) , 5, 999999999? I think it's pretty intuitive to say that 999999999 is unexpected relative to the 5s. And you certainly can have expectations that are generated purely based off of the measured distribution and not off of some theoretical model.
        whatshisface 4y ago
        It is unexpected relative to the 5s, but it is also rare in your sample, meaning that its occurrence in the sample is in exact proportion to how often you'd expect it to occur, given the distribution you inferred from your sample. That means you have no grounds for singling it out as "abnormal."
        nl 4y ago
        And here we see the "Bayesian vs Frequentist" argument. If this gets resolved in a HN thread you should publish it.
      - WastingMyTime89 4y ago
        Obviously you can say that 9999999999 is very different from the rest of the sample consisting of 5 without an implicit distribution. It’s incredibly far from the median compared to all the other value.
        whatshisface 4y ago
        But is it surprising to find it far from the median?
        WastingMyTime89 4y ago
        It doesn’t matter if it’s surprising. It might be a measurement mistake, a mixed population, a fat tail or a poor sample. It’s different therefore an outlier.
mhh__ 4y ago
Sometimes there's a horse rubbing against your air pressure sensor though.
aquafox 4y ago
That's not true. There's a variety of reasons why outliers can exist, for example data quality issues where an experiment failed or the method to measure the data did not work. The approach presented there in my opinion has one big caveat: It tries to learn the cause of outliers from the data itself instead of incorporating domain knowledge! Having and using domain knowledge is time consuming and difficult, but nothing can replace it. One cannot automate away to critically explore and understand the data, although machine learning folks think they do.
- Retric 4y ago
  If your population is sensor readings rather than actual objects then malfunctions should be completely expected behavior. Treating them as exceptional is simply a sign you don’t have enough data.
  - kqr 4y ago
    There's a difference between expected malfunctions (due to common causes) and exceptional malfunctions (due to assignable causes) though.
magicalhippo 4y ago
Lets say you time some code, and get a nice Gaussian at around 1ms say, but with a smaller peak far to the right (ie +20ms or so). The second peak is because your thread got scheduled. You don't care about those samples, they don't matter. Can't you just say those are outliers and not consider them? Ideally you'd build some bimodal model, but given that they don't matter, what's wrong with throwing away those samples?
- kqr 4y ago
  This is the problem with how most people deal with outliers. You should absolutely care about your code being scheduled away. If you care about speed and then you change your code in ways that makes it get pre-empted more often, that matters! Outliers are still useful data!
  - magicalhippo 4y ago
    If you write code that gets pre-empted more often, then you've deliberately done that by inserting system calls or similar, and that would be an entirely different scenario. I agree that in such a scenario, it would not be an outlier.
civilized 4y ago
> Most statisticians would say that there's not really a difference between an outlier and a non-outlier Definitely untrue. Suppose we throw 1,000 baseballs, shoot one rocket (45 degree angle), record the distance each one flew, and present the 1,001 data points to someone for analysis. Most people, statisticians or not, will recognize the rocket's distance as an outlier, even if they don't know the cause of it. And they'll be right to do so. It's true that there is no universal definition of outlier. But outliers are very clearly defined in some contexts.
- timy2shoes 4y ago
  > Suppose we throw 1,000 baseballs, shoot one rocket (45 degree angle), record the distance each one flew, and present the 1,001 data points to someone for analysis. But I would argue that's not really a case of an outlier, that's a mixture distribution because the underlying identifying feature (baseball or rocket) is missing.
  - civilized 4y ago
    You've almost grasped the point. Mixture distributions with a missing underlying feature are how outliers (in the meaningful, well-defined sense) are generated. And if you dig into this paper (I have not yet), I can almost guarantee you that this is foundational to the paper.
    - timy2shoes 4y ago
      I can tell you from extensive experience that domain knowledge is more critical to understanding such issues than this algorithm. Fundamentally, such causes can not be identified if they don't exist in the data and the algorithm doesn't "see" them. Like the previous example where the identity of the object being hurled is hidden and all the data shows is the distance. And the causes change a lot from field to field.
      - civilized 4y ago
        I don't believe that your extensive experience tells you about the utility of an algorithm you've never used that was developed by Amazon researchers most likely to help with problems at Amazon.
nl 4y ago
> Most statisticians would say that there's not really a difference between an outlier and a non-outlier, but actually the data just have fatter tails than you originally thought, and you need to use a robust modeling method to account for that. No statisticians would ever propose that as a practical industrial method. Sometimes a bat falls in your vat of beer at the exact moment the lid is off and no sensible statistician proposes modelling the flight path of bats. Understanding and removal of outliers has been part of the statisticians toolbox forever. Gosset's[1] (The inventor of modern statistics) whole work on Student's t-distribution was because he had to deal with small sample size and know when sample weren't representative, and how to discard outliers. Studentized Residual is named for him and is (to quote Wikipedia)[2]: an important technique in the detection of outliers. The statisticians I've worked with are very pragmatic, and none want to try to model the whole universe. It was a statistician who came up with "all models are wrong but some are useful" after all. [1] https://en.wikipedia.org/wiki/William_Sealy_Gosset https://en.wikipedia.org/wiki/William_Sealy_Gosset [2] https://en.wikipedia.org/wiki/Studentized_residual https://en.wikipedia.org/wiki/Studentized_residual
dekhn 4y ago
Some statisticians may say that, but as the two cultures swap places (http://www2.math.uu.se/~thulin/mm/breiman.pdf http://www2.math.uu.se/~thulin/mm/breiman.pdf) more folks take a mechanistic view of data collection, in which some observations are truly what we would call "out of class where the class boundaries are well-defined". I suppose this gets into your definition of reality and how that relates to class boundaries.

WastingMyTime89 4y ago

By definition outliers are just points which differ significantly from the others. Obviously if your distribution has heavy tails they are likely to be there. They could also be noise or simply errors. The only things you know before analysing is that they are different which is why they are outliers.

whatshisface 4y ago
If you carefully read the article, what they're doing is classifying points by how unlikely they'd be according to their model of their system, and they are using that to find inaccuracies in their model. Without a model there is no difference between outliers and "regular" data points, because a data point will always match a distribution made from itself.
- WastingMyTime89 4y ago
  You don’t need a model for a point to be significantly different. It just needs to have characteristics making it stand out. Outliers are an important concept for the theory of estimators for exemple and that doesn’t involve modelling at all. This discussion is why I regret that people doing CS have so little exposure to frequentist statistics.
  - whatshisface 4y ago
    If you measure 5, 5, 5, 9999999999, then your measured distribution is a peak at 5 and another at 9999999999. You can't say that 9999999999 is a surprising measurement unless you have a theoretical distribution - usually an implicit one - to evaluate at it and find a low probability. In other words, if being surprised means having your expectations violated, an expression of surprise implies that you must have had expectations to violate.
    - strangedejavu 4y ago
      This argument doesn't make much sense to me. You can definitely associate a probability based just on the measured data. What if you measured 5, 5, 5, 5, 5, 5,5 (1000 more 5s) , 5, 999999999? I think it's pretty intuitive to say that 999999999 is unexpected relative to the 5s. And you certainly can have expectations that are generated purely based off of the measured distribution and not off of some theoretical model.
      - whatshisface 4y ago
        It is unexpected relative to the 5s, but it is also rare in your sample, meaning that its occurrence in the sample is in exact proportion to how often you'd expect it to occur, given the distribution you inferred from your sample. That means you have no grounds for singling it out as "abnormal."
        nl 4y ago
        And here we see the "Bayesian vs Frequentist" argument. If this gets resolved in a HN thread you should publish it.
    - WastingMyTime89 4y ago
      Obviously you can say that 9999999999 is very different from the rest of the sample consisting of 5 without an implicit distribution. It’s incredibly far from the median compared to all the other value.
      - whatshisface 4y ago
        But is it surprising to find it far from the median?
        WastingMyTime89 4y ago
        It doesn’t matter if it’s surprising. It might be a measurement mistake, a mixed population, a fat tail or a poor sample. It’s different therefore an outlier.

mhh__ 4y ago

Sometimes there's a horse rubbing against your air pressure sensor though.

aquafox 4y ago

That's not true. There's a variety of reasons why outliers can exist, for example data quality issues where an experiment failed or the method to measure the data did not work. The approach presented there in my opinion has one big caveat: It tries to learn the cause of outliers from the data itself instead of incorporating domain knowledge! Having and using domain knowledge is time consuming and difficult, but nothing can replace it. One cannot automate away to critically explore and understand the data, although machine learning folks think they do.

Retric 4y ago
If your population is sensor readings rather than actual objects then malfunctions should be completely expected behavior. Treating them as exceptional is simply a sign you don’t have enough data.
- kqr 4y ago
  There's a difference between expected malfunctions (due to common causes) and exceptional malfunctions (due to assignable causes) though.

magicalhippo 4y ago

Lets say you time some code, and get a nice Gaussian at around 1ms say, but with a smaller peak far to the right (ie +20ms or so). The second peak is because your thread got scheduled. You don't care about those samples, they don't matter. Can't you just say those are outliers and not consider them? Ideally you'd build some bimodal model, but given that they don't matter, what's wrong with throwing away those samples?

kqr 4y ago
This is the problem with how most people deal with outliers. You should absolutely care about your code being scheduled away. If you care about speed and then you change your code in ways that makes it get pre-empted more often, that matters! Outliers are still useful data!
- magicalhippo 4y ago
  If you write code that gets pre-empted more often, then you've deliberately done that by inserting system calls or similar, and that would be an entirely different scenario. I agree that in such a scenario, it would not be an outlier.

civilized 4y ago

> Most statisticians would say that there's not really a difference between an outlier and a non-outlier Definitely untrue. Suppose we throw 1,000 baseballs, shoot one rocket (45 degree angle), record the distance each one flew, and present the 1,001 data points to someone for analysis. Most people, statisticians or not, will recognize the rocket's distance as an outlier, even if they don't know the cause of it. And they'll be right to do so. It's true that there is no universal definition of outlier. But outliers are very clearly defined in some contexts.

timy2shoes 4y ago
> Suppose we throw 1,000 baseballs, shoot one rocket (45 degree angle), record the distance each one flew, and present the 1,001 data points to someone for analysis. But I would argue that's not really a case of an outlier, that's a mixture distribution because the underlying identifying feature (baseball or rocket) is missing.
- civilized 4y ago
  You've almost grasped the point. Mixture distributions with a missing underlying feature are how outliers (in the meaningful, well-defined sense) are generated. And if you dig into this paper (I have not yet), I can almost guarantee you that this is foundational to the paper.
  - timy2shoes 4y ago
    I can tell you from extensive experience that domain knowledge is more critical to understanding such issues than this algorithm. Fundamentally, such causes can not be identified if they don't exist in the data and the algorithm doesn't "see" them. Like the previous example where the identity of the object being hurled is hidden and all the data shows is the distance. And the causes change a lot from field to field.
    - civilized 4y ago
      I don't believe that your extensive experience tells you about the utility of an algorithm you've never used that was developed by Amazon researchers most likely to help with problems at Amazon.

nl 4y ago

> Most statisticians would say that there's not really a difference between an outlier and a non-outlier, but actually the data just have fatter tails than you originally thought, and you need to use a robust modeling method to account for that. No statisticians would ever propose that as a practical industrial method. Sometimes a bat falls in your vat of beer at the exact moment the lid is off and no sensible statistician proposes modelling the flight path of bats. Understanding and removal of outliers has been part of the statisticians toolbox forever. Gosset's[1] (The inventor of modern statistics) whole work on Student's t-distribution was because he had to deal with small sample size and know when sample weren't representative, and how to discard outliers. Studentized Residual is named for him and is (to quote Wikipedia)[2]: an important technique in the detection of outliers. The statisticians I've worked with are very pragmatic, and none want to try to model the whole universe. It was a statistician who came up with "all models are wrong but some are useful" after all. [1] https://en.wikipedia.org/wiki/William_Sealy_Gosset https://en.wikipedia.org/wiki/William_Sealy_Gosset [2] https://en.wikipedia.org/wiki/Studentized_residual https://en.wikipedia.org/wiki/Studentized_residual

dekhn 4y ago

Some statisticians may say that, but as the two cultures swap places (http://www2.math.uu.se/~thulin/mm/breiman.pdf http://www2.math.uu.se/~thulin/mm/breiman.pdf) more folks take a mechanistic view of data collection, in which some observations are truly what we would call "out of class where the class boundaries are well-defined". I suppose this gets into your definition of reality and how that relates to class boundaries.