3 ms·

Sadly the paper uses for benchmarks datasets that: - are known to be pretty useless - contain mistakes - can be misleading with a naive F1-score measure. (to b

Sadly the paper uses for benchmarks datasets that: - are known to be pretty useless - contain mistakes - can be misleading with a naive F1-score measure. (to be fair they write "we looked at the F1-Score, under which both partial and full anomaly detection are considered correct identification" so this may be mitigated, but it's not clear) See https://kdd-milets.github.io/milets2021/slides/Irrational%20Exuberance_Eammon_Keogh.pdf https://kdd-milets.github.io/milets2021/slides/Irrational%20... So it's hard to take any benchmark from the paper seriously. The paper is also ignoring any recent work (like > 2018) on univariate timeseries anomaly detection in the matrix profile space (eg MADRID). The "practicality of usage" and conclusion sections are pretty correct though: it's expensive, slow, and no-shot is worthless if some other methods can train and infer in orders of magnitude less time. It would have been interesting to see how the DETECTOR method performs when the LLM forecasting is replaced with some standard forecasting. (eg some auto ETS, if possible robust to anomalies in the training data). It looks like the natural follow up of this article is to remove the LLM altogether.

ericpauley 2y ago
> It looks like the natural follow up of this article is to remove the LLM altogether. Welcome to scientific papers in 2024…
nerdponx 2y ago
> For the second approach, called Detector, they use the LLM as a forecaster to predict the next value from a time series. The researchers compare the predicted value to the actual value. A large discrepancy suggests that the real value is likely an anomaly. Unless there's more to it in the actual paper, this is how just about every anomaly detection technique already works. You fit a model of the distribution of data under normal-enough circumstances, and an observation is an "anomaly" if it seems very improbable (based on your model), or is otherwise extreme if your model isn't explicitly probabilistic. So yes, this technique would be great if you removed the LLM: it's already the industry standard framework. There's nothing wrong conceptually with trying to plug in a transformer model here. The problem is the presumption that a "large" pre-trained transformer model can actually work effectively on arbitrary time series.

ericpauley 2y ago

> It looks like the natural follow up of this article is to remove the LLM altogether. Welcome to scientific papers in 2024…

nerdponx 2y ago

> For the second approach, called Detector, they use the LLM as a forecaster to predict the next value from a time series. The researchers compare the predicted value to the actual value. A large discrepancy suggests that the real value is likely an anomaly. Unless there's more to it in the actual paper, this is how just about every anomaly detection technique already works. You fit a model of the distribution of data under normal-enough circumstances, and an observation is an "anomaly" if it seems very improbable (based on your model), or is otherwise extreme if your model isn't explicitly probabilistic. So yes, this technique would be great if you removed the LLM: it's already the industry standard framework. There's nothing wrong conceptually with trying to plug in a transformer model here. The problem is the presumption that a "large" pre-trained transformer model can actually work effectively on arbitrary time series.