6 ms·

They gave a lot of IAAI talks! WOW! I gave only one, from our joint work with GM. It looks like they were able to make expert systems useful. Our work on Kn

They gave a lot of IAAI talks! WOW! I gave only one, from our joint work with GM. It looks like they were able to make expert systems useful. Our work on KnowledgeTool might be equally useful! After all, it appears that their work was stuck with C, and our work on KnowledgeTool was based on, was a pre-processor to -- sorry to mention this -- PL/I. But, gee, they based on C, so mentioning PL/I should not be so bad! Our use of KnowledgeTool was to monitor server farms and networks. There in particular we were trying to do problem detection. So, in expert systems, When I see A and B and 5 or more cases of C without a D, then it looks really bad -- raise an alarm. and, thus, have staff investigate. And this statement of a symptom, test, for something wrong, was supposed to come from operator experience. So, broadly the idea of KnowledgeTool for that work was not to find better ways to monitor but just to encode what operators had already learned, mostly just from experience. Hmm .... In such monitoring, we have two ways to be wrong, (1) a false alarm where we say that the system is sick when it is well and (2) a missed detection where we say that a system is well when it is sick. So, we want the false alarm rate, to be called the probability of Type I error, and the missed detection rate, to be called the probability of Type II error. Quite broadly there is a trade-off between these two rates: If we accept a higher false alarm rate, then we stand to get a lower missed detection rate. Not all means of detection are equivalent: Some poor detectors have really high rates for both false alarms and missed detections, and some good detectors have really low rates on both of the errors. A problem with expert systems was that we had no idea what rates we were getting or if our detectors were poor or good. Commonly at least in principle we can adjust the rate of false alarms. That is good because (1) a false alarm sends the bridge staff on wild goose chasing and (2) false alarm rate too high is a standard complaint. Looking at the problem in a little more detail, we easily had wide, deep rapidly flowing oceans of data. So, we could easily have data on 1000 variables with the data for each variable arriving at 100 times a second. So, write some expert system rules? No thanks! So, I dreamed up a solution, basically, right, as guess from Type I/II, a statistical hypothesis test but one that is both distribution free and multi-dimensional, maybe the first such. False alarm rate can be set in advance of collecting data, and is in a useful sense for each selected false alarm rate has the detection rate the highest possible (for that false alarm rate). I did this work to improve on expert systems for monitoring. Yes, I published the work, in Information Sciences. Ah, beating expert systems! What's in the paper is some applied math based on some advanced pure math prerequisites, especially an idea from ergodic theory. What's in the paper really SHOULD be deployed. I suspect that more could be done. Well, the OP starts with > Developed at NASA’s Johnson Space Center from 1985 to 1996, the C Language Integrated Production System (CLIPS) is a rule-based programming language useful for creating expert systems and other programs where a heuristic solution is easier to implement and maintain than an algorithmic solution. Gee, I thought that my "algorithmeic", really original mathematical statistics, was easier than doing as well with heuristics.

smee0815 7y ago
What is the title of your publication? Would love to give it a read.
- graycat 7y ago
  See my answer to your question just above! From you, too, we want to hear what utility you see in the paper!
HerrMonnezza 7y ago
This sounds interesting! Can you give any reference to your publication or URL?
- graycat 7y ago
  Okay, I'll give in: Nearly any good academic research library should have and be able to find the paper: "A Real-Time System-Adapted Anomaly Detector", Information Sciences, volume 115, pages 221-259. The core math is some measure preserving transformations of the data -- get to move data around without changing probability distribution -- for any distribution. The usual place see measure preserving is in ergodic theory. Recall, my work is distribution free which means need make no assumptions at all about probability distributions. In the context of computer and network performance data, distribution-free is nearly essential. After moving the data around, get to do some counting that leads to calculating false alarm rate. So the paper is some applied probability, that is, does some probability calculations in the service of some, call it, mathematical statistics. The simplest view is that it's all just nearest neighbors detection -- if get a point too far from the old points, then raise an alarm. Since I wrote the paper, that idea has become common. But apparently what has not become common is how to adjust and calculate false alarm rate, and in practice that is really important. Indeed, for any detection, my work can report the lowest false alarm rate for which the (real time) observation would still be a detection -- so get to have not just a detection but, intuitively, a <i>seriousness</i> measure. "Joe, this is a detection, and it will remain a detection with a false alarm rate of one in a billion. Put down your pizza, and let's investigate that." So, right, could display a strip chart with such data. For calculating false alarm rate there is a simple approach that gives the right answer, but the simple approach makes an impossible assumption of independence. Well, the right answer is still true; it's just that an actual proof of false alarm rate, where don't assume independence, is a bit tricky mathematically. This math is a close cousin of not independence but a weaker assumption exchangeability where often can get the same results as if had independence. My math may be something of a rationalization of part of resampling as pursued by P. Diaconis and B. Efron at Stanford. In an intuitive sense, we almost have independence. Hmm -- approximate independence is not a big field. There is a paper by M. Talagrand, A New Look at Independence that may be able to shed some light on my paper, find another way to do the calculations and show that the same calculations hold approximately, maybe with some bounds, in other situations. Talagrand is one heck of a good mathematician, a student of Choquet, a member of Bourbaki. It would be nice to have the most powerful statistical hypothesis tests as from the classic result of Neyman-Pearson; alas, in the context we don't have enough data to use Neyman-Pearson. But I did come up with a weaker but still useful sense in which my techniques do yield the most powerful test. In practice, the test will likely be seen as nicely powerful, that is, even when false alarm rate has been selected to something small, the detection rate is still, likely in practice, about the highest can hope for for that selected false alarm rate. There is an issue of how to make the computations fast -- so, need something out of computational geometry. I worked up a technique that should be reasonably fast. The solid state drives of today should do wonders for this technique. The computational geometry and the most powerful test derivation are not in the paper. There's more that can be done. Might also use the work for other anomaly detection problems. One nice point is, although might be using lots of variables, don't encounter "over fitting". What we were doing with expert systems was mostly just thresholds on one variable at a time. Well, in some cases, we know so much about the variable that that is okay. But there can be thousands of variables, collected by HP, Microsoft, etc., and we can't know just from personal knowledge or experience just what thresholds make good sense. And we have nothing solid on false alarm rate or how to adjust it -- yes, we could get an approximation by going back through the data. And, then we have still less on detection rate. If want a server farm that is reliable and secure, about the best you can hope for, then maybe my work would be one of the important tools and means. This stuff about "multi-dimensional" is serious and important. E.g., suppose we have three threshold detectors. Then whether we wanted to or not, we just assumed that the 3D region of normal, healthy, not an anomaly or sick, performance is a 3D box. The box does not promise to fit reality well. Make the box too small, and get too many false alarms. Make the box bigger and can reduce the false alarm rate, but since we still have a box that does not fit reality well we are stuck with detection rate too low and a "poor" detector. Well, can think a little and intuitively convince yourself that the region of normal (healthy) performance could be fractal, e.g., the Mandelbrot set, and still the math should work. To me, for monitoring server farms and networks, my little paper totally blew out of the water with the doors blown off nearly everything we were doing with expert systems. Let us all know what utility you find in the paper!

smee0815 7y ago

What is the title of your publication? Would love to give it a read.

graycat 7y ago
See my answer to your question just above! From you, too, we want to hear what utility you see in the paper!

HerrMonnezza 7y ago

This sounds interesting! Can you give any reference to your publication or URL?

graycat 7y ago
Okay, I'll give in: Nearly any good academic research library should have and be able to find the paper: "A Real-Time System-Adapted Anomaly Detector", Information Sciences, volume 115, pages 221-259. The core math is some measure preserving transformations of the data -- get to move data around without changing probability distribution -- for any distribution. The usual place see measure preserving is in ergodic theory. Recall, my work is distribution free which means need make no assumptions at all about probability distributions. In the context of computer and network performance data, distribution-free is nearly essential. After moving the data around, get to do some counting that leads to calculating false alarm rate. So the paper is some applied probability, that is, does some probability calculations in the service of some, call it, mathematical statistics. The simplest view is that it's all just nearest neighbors detection -- if get a point too far from the old points, then raise an alarm. Since I wrote the paper, that idea has become common. But apparently what has not become common is how to adjust and calculate false alarm rate, and in practice that is really important. Indeed, for any detection, my work can report the lowest false alarm rate for which the (real time) observation would still be a detection -- so get to have not just a detection but, intuitively, a <i>seriousness</i> measure. "Joe, this is a detection, and it will remain a detection with a false alarm rate of one in a billion. Put down your pizza, and let's investigate that." So, right, could display a strip chart with such data. For calculating false alarm rate there is a simple approach that gives the right answer, but the simple approach makes an impossible assumption of independence. Well, the right answer is still true; it's just that an actual proof of false alarm rate, where don't assume independence, is a bit tricky mathematically. This math is a close cousin of not independence but a weaker assumption exchangeability where often can get the same results as if had independence. My math may be something of a rationalization of part of resampling as pursued by P. Diaconis and B. Efron at Stanford. In an intuitive sense, we almost have independence. Hmm -- approximate independence is not a big field. There is a paper by M. Talagrand, A New Look at Independence that may be able to shed some light on my paper, find another way to do the calculations and show that the same calculations hold approximately, maybe with some bounds, in other situations. Talagrand is one heck of a good mathematician, a student of Choquet, a member of Bourbaki. It would be nice to have the most powerful statistical hypothesis tests as from the classic result of Neyman-Pearson; alas, in the context we don't have enough data to use Neyman-Pearson. But I did come up with a weaker but still useful sense in which my techniques do yield the most powerful test. In practice, the test will likely be seen as nicely powerful, that is, even when false alarm rate has been selected to something small, the detection rate is still, likely in practice, about the highest can hope for for that selected false alarm rate. There is an issue of how to make the computations fast -- so, need something out of computational geometry. I worked up a technique that should be reasonably fast. The solid state drives of today should do wonders for this technique. The computational geometry and the most powerful test derivation are not in the paper. There's more that can be done. Might also use the work for other anomaly detection problems. One nice point is, although might be using lots of variables, don't encounter "over fitting". What we were doing with expert systems was mostly just thresholds on one variable at a time. Well, in some cases, we know so much about the variable that that is okay. But there can be thousands of variables, collected by HP, Microsoft, etc., and we can't know just from personal knowledge or experience just what thresholds make good sense. And we have nothing solid on false alarm rate or how to adjust it -- yes, we could get an approximation by going back through the data. And, then we have still less on detection rate. If want a server farm that is reliable and secure, about the best you can hope for, then maybe my work would be one of the important tools and means. This stuff about "multi-dimensional" is serious and important. E.g., suppose we have three threshold detectors. Then whether we wanted to or not, we just assumed that the 3D region of normal, healthy, not an anomaly or sick, performance is a 3D box. The box does not promise to fit reality well. Make the box too small, and get too many false alarms. Make the box bigger and can reduce the false alarm rate, but since we still have a box that does not fit reality well we are stuck with detection rate too low and a "poor" detector. Well, can think a little and intuitively convince yourself that the region of normal (healthy) performance could be fractal, e.g., the Mandelbrot set, and still the math should work. To me, for monitoring server farms and networks, my little paper totally blew out of the water with the doors blown off nearly everything we were doing with expert systems. Let us all know what utility you find in the paper!