5 ms·

Some people, when confronted with a complex system, think "I know, I'll use a large language model." Now they have two complex systems interacting.

cqqxo4zV46cp 2y ago
Some people trivialise things in bad faith to score internet points on Hacker News, I guess. The status quo, per the article, was already using “deep learning” models to perform anomaly detection. We are already talking about complex, black-box systems. The stated advantage was that they didn’t require deployment-specific training. You can just throw a pre-trained LLM at it. There is also a stated benefit: early-stage detection, without needing to pay for, or wait for, training a custom ML model. The article is quite open about the fact that the LLM approach doesn’t beat the state of the art in terms of accuracy. It’s like you didn’t click the link.
- mdp2021 2y ago
  The metrics OP suggest increase with the transition from DL to LLMs (and OP had no immediate duty to discuss other metrics).
- glutamate 2y ago
  Colour me old-fashioned, but i am really sceptical about a zero-shot general-purpose LLM-based anomaly detector being able to outperform a trivial, not-obviously-wrong ML/statistical model calibrated on a small subset of the data that makes training extremely fast, in entirely new domains. And in anomaly detection, false negatives (and even false positives) can be very expensive. The most promising role for LLMs in timeseries prediction is in extracting covariates from unstructured data, in the wind example, weather reports and geographically proximal social media posts.
  - thomashop 2y ago
    You are skeptical about something that is explained quite thoroughly in the article. Did you read the article? Edit: I meant they never claim to outperform SOTA domain-specific approaches. So, I don't know whose claim you are refuting.
    - glutamate 2y ago
      > While LLMs could not beat state-of-the-art deep learning models at anomaly detection, they did perform as well as some other AI approaches. I think one should also be sceptical about the current hype train
- Kiro 2y ago
  Funny how one of the the few comments actually discussing the content of the article gets downvoted while generic memes that have nothing to do with it get upvoted.
- dotnet00 2y ago
  >early-stage detection, without needing to pay for, or wait for, training a custom ML model This doesn't make much sense though, if you're in a hurry to get something working and don't have the data for training a custom model, you'd use the knowledge you have of the system to setup a typical statistical anomaly detector. That way you'd have the insight into the system behavior that you'd want early on, would be able to enforce all sorts of guarantees and would not have the weight and complexity of an LLM on the system's resources, rather than worrying about this LLM that has nowhere near the same info on the system. Using an LLM for this would be like hiring a random person off the street to keep your network infrastructure working because you're in such a rush that you don't even want to document how your infra is setup, let alone wait to find someone who would know how to decipher it.
visarga 2y ago
> Now they have two complex systems interacting. As opposed to sending crowds of people to interact with the system? We are even more complex and less benchmarked/tested than LLMs. At least the LLM is a known thing, we know its limitations, we can evaluate and compensate for their lacks.
mensetmanusman 2y ago
The human body is orders more complex than LLMs and we can already use LLMs to improve anomaly detection rates for cancer.
- theGnuMe 2y ago
  > improve anomaly detection rates for cancer. What do you mean by that?
  - mensetmanusman 2y ago
    https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11003452/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11003452/ “We propose and demonstrate the use of these models to assist in the manual curation process which results in higher accuracy and F1 score with lesser time and cost, thus improving efforts of cancer research.”
    - theGnuMe 2y ago
      How does this relate to anomaly detection?
      - mensetmanusman 2y ago
        Anomalys are markers to keep track of when diagnosing.
        theGnuMe 2y ago
        That paper doesn't even mention the word anomaly or detecting them.
        mensetmanusman 2y ago
        Cancer is anomalous cell behavior.

cqqxo4zV46cp 2y ago

Some people trivialise things in bad faith to score internet points on Hacker News, I guess. The status quo, per the article, was already using “deep learning” models to perform anomaly detection. We are already talking about complex, black-box systems. The stated advantage was that they didn’t require deployment-specific training. You can just throw a pre-trained LLM at it. There is also a stated benefit: early-stage detection, without needing to pay for, or wait for, training a custom ML model. The article is quite open about the fact that the LLM approach doesn’t beat the state of the art in terms of accuracy. It’s like you didn’t click the link.

mdp2021 2y ago
The metrics OP suggest increase with the transition from DL to LLMs (and OP had no immediate duty to discuss other metrics).
glutamate 2y ago
Colour me old-fashioned, but i am really sceptical about a zero-shot general-purpose LLM-based anomaly detector being able to outperform a trivial, not-obviously-wrong ML/statistical model calibrated on a small subset of the data that makes training extremely fast, in entirely new domains. And in anomaly detection, false negatives (and even false positives) can be very expensive. The most promising role for LLMs in timeseries prediction is in extracting covariates from unstructured data, in the wind example, weather reports and geographically proximal social media posts.
- thomashop 2y ago
  You are skeptical about something that is explained quite thoroughly in the article. Did you read the article? Edit: I meant they never claim to outperform SOTA domain-specific approaches. So, I don't know whose claim you are refuting.
  - glutamate 2y ago
    > While LLMs could not beat state-of-the-art deep learning models at anomaly detection, they did perform as well as some other AI approaches. I think one should also be sceptical about the current hype train
Kiro 2y ago
Funny how one of the the few comments actually discussing the content of the article gets downvoted while generic memes that have nothing to do with it get upvoted.
dotnet00 2y ago
>early-stage detection, without needing to pay for, or wait for, training a custom ML model This doesn't make much sense though, if you're in a hurry to get something working and don't have the data for training a custom model, you'd use the knowledge you have of the system to setup a typical statistical anomaly detector. That way you'd have the insight into the system behavior that you'd want early on, would be able to enforce all sorts of guarantees and would not have the weight and complexity of an LLM on the system's resources, rather than worrying about this LLM that has nowhere near the same info on the system. Using an LLM for this would be like hiring a random person off the street to keep your network infrastructure working because you're in such a rush that you don't even want to document how your infra is setup, let alone wait to find someone who would know how to decipher it.

visarga 2y ago

> Now they have two complex systems interacting. As opposed to sending crowds of people to interact with the system? We are even more complex and less benchmarked/tested than LLMs. At least the LLM is a known thing, we know its limitations, we can evaluate and compensate for their lacks.

mensetmanusman 2y ago

The human body is orders more complex than LLMs and we can already use LLMs to improve anomaly detection rates for cancer.

theGnuMe 2y ago
> improve anomaly detection rates for cancer. What do you mean by that?
- mensetmanusman 2y ago
  https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11003452/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11003452/ “We propose and demonstrate the use of these models to assist in the manual curation process which results in higher accuracy and F1 score with lesser time and cost, thus improving efforts of cancer research.”
  - theGnuMe 2y ago
    How does this relate to anomaly detection?
    - mensetmanusman 2y ago
      Anomalys are markers to keep track of when diagnosing.
      - theGnuMe 2y ago
        That paper doesn't even mention the word anomaly or detecting them.
        mensetmanusman 2y ago
        Cancer is anomalous cell behavior.