3 ms·

I love how he(?) wears the "I'm not a datascientist" statement proudly like a badge of honour. Hah! Nice article. My only comment would be that performing anom

I love how he(?) wears the "I'm not a datascientist" statement proudly like a badge of honour. Hah! Nice article. My only comment would be that performing anomaly detection on the same data that you use to "train" your detector in the first place, is a classic example of a "data-leak" in data science, and very likely to lead to errors in itself.

ithrow 6y ago
so, how do you gather the correct data to "train" the detector?
- surething123 6y ago
  It's not really a question of the "correct" data. To evaluate whether an observation is anomalous, you typically define a threshold for what's considered not-normal, e.g. an observation 2 or 3 standard deviations above the mean. This may leave you with some false positives (like any system of this nature would). Of course, you could go the route of actually defining an anomaly yourself and building a more sophisticated model (i.e. one not using z-scores as the measure of anomaly), but that's obviously a different scope.
whoisburbansky 6y ago
What would you consider the "training" step here? The part where the author performs backtesting to settle on an adequate value for the threshold? In the case I don't really see a problem, since the author seems to be looking at historical data to come up with a value for a parameter that will be used to determine the model that acts on future data. I agree with your comment in general that using test data in your training set is a clear example of an error that will lead to overfitting and a model that generalizes poorly, but am having a hard time seeing where the author commits this mistake.
- tpoacher 6y ago
  The way he describes the problem in the introduction alludes to this. I.e. "here's a some data, 12 is a clear outlier, let's see if we can confirm this using the mean and standard deviation of the sample to derive lower and upper bounds". Then he proceeds to include 12 in the calculation deriving these bounds. This is not really the way to do it. In fact, if he had excluded the anomalous measurement from the training data, the '5' values would have been excluded as outliers as well, given his criteria for defining the bounds. I agree that this is a trivial point made on a trivial example though, and that it is more a matter of 'sensible definitions' of what counts as anomalous or as training set in the first place. But it's still worth thinking about explicitly though, so I thought I'd mention it.
noblebeast 6y ago
Why do you need a train/test split for a z-score? It's just telling you if data is extreme beyond some threshold defined by the distribution from which it originated.
- tpoacher 6y ago
  Replace the value 12 in the introduction by 1,000,000 and see if you still agree with your premise :)
codeulike 6y ago
My only comment would be that performing anomaly detection on the same data that you use to "train" your detector in the first place, is a classic example of a "data-leak" in data science, and very likely to lead to errors in itself. But thats the whole point of the very well established concept of standard deviation - to look at a dataset in isolation and analyse it.
yters 6y ago
In machine learning there is the concept of a generalization bound where you can determine if your model has successfully generalized without reference to a validation dataset. In which case, your training and real world dataset can be the same.

ithrow 6y ago

so, how do you gather the correct data to "train" the detector?

surething123 6y ago
It's not really a question of the "correct" data. To evaluate whether an observation is anomalous, you typically define a threshold for what's considered not-normal, e.g. an observation 2 or 3 standard deviations above the mean. This may leave you with some false positives (like any system of this nature would). Of course, you could go the route of actually defining an anomaly yourself and building a more sophisticated model (i.e. one not using z-scores as the measure of anomaly), but that's obviously a different scope.

whoisburbansky 6y ago

What would you consider the "training" step here? The part where the author performs backtesting to settle on an adequate value for the threshold? In the case I don't really see a problem, since the author seems to be looking at historical data to come up with a value for a parameter that will be used to determine the model that acts on future data. I agree with your comment in general that using test data in your training set is a clear example of an error that will lead to overfitting and a model that generalizes poorly, but am having a hard time seeing where the author commits this mistake.

tpoacher 6y ago
The way he describes the problem in the introduction alludes to this. I.e. "here's a some data, 12 is a clear outlier, let's see if we can confirm this using the mean and standard deviation of the sample to derive lower and upper bounds". Then he proceeds to include 12 in the calculation deriving these bounds. This is not really the way to do it. In fact, if he had excluded the anomalous measurement from the training data, the '5' values would have been excluded as outliers as well, given his criteria for defining the bounds. I agree that this is a trivial point made on a trivial example though, and that it is more a matter of 'sensible definitions' of what counts as anomalous or as training set in the first place. But it's still worth thinking about explicitly though, so I thought I'd mention it.

noblebeast 6y ago

Why do you need a train/test split for a z-score? It's just telling you if data is extreme beyond some threshold defined by the distribution from which it originated.

tpoacher 6y ago
Replace the value 12 in the introduction by 1,000,000 and see if you still agree with your premise :)

codeulike 6y ago

My only comment would be that performing anomaly detection on the same data that you use to "train" your detector in the first place, is a classic example of a "data-leak" in data science, and very likely to lead to errors in itself. But thats the whole point of the very well established concept of standard deviation - to look at a dataset in isolation and analyse it.

yters 6y ago

In machine learning there is the concept of a generalization bound where you can determine if your model has successfully generalized without reference to a validation dataset. In which case, your training and real world dataset can be the same.