3 ms·

It seems like a semi-supervised classification framework can address each of those 9 issues. I agree btw that each is important, but would emphasize the importa

• jmmcd 11y ago
  Part of the problem in anomaly detection is that sometimes anomalies are of a "novel" type. It can be impossible to collect labelled data on all possible anomalies. So learning a model of normal behaviour turns out better. But if we're talking about a specific application, where there's only one (or a few) types of bad behaviour, then it's probably worthwhile to manually label some data and use binary classification as you suggested.
  • graycat 11y ago
    Yes, for your first point, you seem to be considering behavioral monitoring where we attempt to characterize the behavior of a healthy or normal system and decree and declare everything else a sick system or an anomaly. Yes, of your second, an example seems to be detecting malware in communications data streams or hard disk files by looking for specific, characterizing patterns of bits, that is, signatures extracted from earlier detections of such malware. In behavioral monitoring, we might have challenges with false alarm rate.
• graycat 11y ago
  > It seems like a semi-supervised classification framework can address each of those 9 issues. Well, in the terminology of machine learning, semi-supervised needs a start with supervised training and labeled data, those are not possible with the challenge I mentioned with zero-day. Also, in that "framework", without more detail, it's not very clear how to select and obtain a false alarm rate. There is a lot on classification. Maybe what you mean is close to L. Breiman's book Classification and Regression Trees (CART and related software) and/or his more recent random forests. > My thinking ... nearest neighbors or logistic regression would surely be fine. So, it sounds like you have some historical or training data that, for the set of real numbers R and positive integers m and n, consists of m points in R^n. Then you want to assume that some regions in R^n (I omit an attempt at a careful definition of a region) are for a sick system and the other regions are for a healthy system (more generally, in some regions the conditional probability of a sick system is relatively high and in other regions, low). Well, if the number of variables n and the dimension of R^n is much over, say, 10, and for data from a server farm that's easy to have, then could need number of historical point m to be huge or could have some quite sparse data. It may be that logistic regression trees could fit such data, that is, the regions, with good accuracy. Then for false alarm rate, apply the trees, the detector, to the old data or some reserved old data, and assume that new data is appropriately similar. Then that application could give estimates of both false alarm rate and detection rate. First cut, I would suggest using regression (with or without the logistic version) trees and not just one regression expression. So, first cut, how to build a regression tree? Do ordinary regression, see where there is a poor fit, split off that data, and fit it separately. Continue splitting and get a tree. For more, read Breiman! Breiman was a darned good mathematician and statistician. So there the role of classification is that the detector, from the logistic regression trees, would return 0 for a healthy system and 1 for a sick one. If you get a low rate of false alarms and a high rate of detections, that is, nearly a perfect detector, on, say, your reserved historical data, and are confident you didn't get hurt with over fitting and are okay on other needed assumptions, then just smile on the way to the bank. > With that understanding, I don't follow what you mean by 'modest number of discrete cases' for classification. For your problem, as I understand it, you don't have "a modest number of discrete cases" but the post of mine I linked to for the thread on fraud detection did. > it's quite interesting how that eventually ends up intimately connected with the hypo test math. Again, my view is that in anomaly detection we can't be far from the framework and/or content of classic statistical hypothesis testing. What I gave in that post was a cheap and quick and dirty derivation of the classic, IIRC, 1947, Neyman-Pearson result on best possible detection. So, in part for best possible, we have to solve a maximization problem. My proof for the general case also makes use of Lagrange multipliers but also uses the Hahn decomposition from the Radon-Nikodym theorem. > doesn't appear at face value to deal with your point #7 about continuous data) Right, that fraud detection problem in my link was for a quite special case of anomaly detection. For those nine challenges I listed, I was not suggesting that we could attack them all with just one approach to anomaly detection. Instead, I was just saying that those nine challenges get encountered in some cases of anomaly detection. I was assuming implicitly that we would be taking different approaches for different cases of anomaly detection and addressing those of the nine challenges that applied in each of the cases. As a response to the OP here, I was suggesting that there is a lot to anomaly detection and, via the nine challenges, tried to describe some of how much there is to consider. In particular I never used Bayes and never mentioned programming languages or software, right, because I believe that there are challenges that need solutions that don't have much to do with Bayes or programming languages. E.g., in describing the role of the historical (training) data, I would consider conditional probabilities and conditional expectation. > but would emphasize the importance of understanding collinearity/correlation with multidimensional data. When I was considering graduate school, one day on my job at FedEx, I was at GE in Boston so took some time off and drove down to the Division of Applied Mathematics at Brown and talked with several of the professors including Ulf Grenander, one of the world's best statisticians. At the time he had a lot of log data from an IBM mainframe and was analyzing it. I noticed that he was surprised at, say, the complexity of the data. He said it was nothing like the data he was used to in statistics from, say, biostatistics. He was correct. And that complexity remains the case. In particular, the usual experience with probability distributions go into the trash -- e.g., in any very direct sense, usually Gaussian becomes nearly irrelevant. Sure, for the case I mentioned of data from a second order stationary stochastic process, covariance is crucial. But, suppose the data is from transactions, say, one data point from each transaction. Then point by point in time, we can't expect anything important from covariance. For such a transaction now? Sure: The data from one visitor to a Web site. Sure, maybe in the past on a computer system, establish some thresholds and use those to detect anomalies, that is, a sick system. Then, have some rules that may do better in some more complicated cases. Or maybe on multi-user computer, watch how busy the CPU is and the rate of page faults and try to detect virtual memory thrashing that goes on for, say, several seconds, and then try to diagnose that and correct it. Okay. But, the world of monitoring server farms and networks has gotten more challenging now! E.g., in some cases, now we want to be able to detect problems we have never seen before -- zero-day detection!