4 ms·

> Snort is the foremost Open Source Intrusion Prevention System (IPS) in the world. Snort IPS uses a series of rules that help define malicious network activity

> Snort is the foremost Open Source Intrusion Prevention System (IPS) in the world. Snort IPS uses a series of rules that help define malicious network activity and uses those rules to find packets that match against them and generates alerts for users. At IBM's Watson lab, I tried rules and was not thrilled. E.g., there was no way to know what the false alarm rate was or to adjust it or know what change in the false alarm rate an particular adjustment would make. And the rate of missed detections was also a problem. For that, for the highest possible detection rate, there is the Neyman-Pearson result, and we should at least try to do something similar in practice! And to write the rules, it appeared needed an expert in the system being monitored. So I worked up some solutions, responses to these issues, with some math, and published. But the people using rules are correct! Rules are what the market wants!

tptacek 4y ago
This is one of the oldest problems in network security. Rule-based detection systems converge on antivirus's effectiveness (or lack thereof). But anomaly systems have almost universally failed in practice, no matter what the anomaly model is, and people have come up with lots of them. I can rattle off reasons why model-based systems are hard to operationalize.
- graycat 4y ago
  Yes, monitoring, anomaly detection, stopping computer viruses, intrusion detection, etc. are old problems. And rules have long been the relatively sophisticated solution technique, i.e., more than just thresholds! And false alarm rate and detection rate have not played much of a role in the discussions. And attacking the problems assuming knowing some relatively fine details about the system, e.g., some Web server, that is the target of the work and the problems want to detect and correct has been common. E.g., when have an instance of some bad stuff that didn't detect soon enough but want to, analyze that problem or class of similar problems and build a detector just for that. And, yup, for that particular problem, might have a nicely low false alarm rate and relatively high detection rate -- on this class of problems seen before, analyzed, and understood. So, here maybe should say that have built a model. But I was not impressed with rules, didn't want to need a lot of fine expertise in the target system, and wanted to detect also problems never seen before, with some good progress on rates of false alarms and miss detections. Yes, what I did is not easily called "model-based" and was supposed to be "new, correct, and significant". Maybe it was at least partly new; maybe it still is. I started with the data being random variables -- that's not assuming much! Then would like to have the assumptions of independent and identically distributed (i.i.d.) random variables, but for practice that is asking a bit too much! And for some positive integer n, having i.i.d. random n-tuples seemed a bit much! Well, I used something weaker, but still strong enough, with some math, to get some results, called exchangability. So, right, my techniques are distribution free, that is, assume nothing about probability distributions and make no attempt to estimate such. Gee, I certainly didn't assume a Gaussian distribution! I was able to get false alarm rate adjustable and known exactly in advance. Looked like progress to me! I had some real data, fed it into my software, and, presto, bingo, the false alarm rate was as adjusted and desired! And using a classic result of S. Ulam, I was able to show that for detection rate the math was not trivial, that is, was better than merely trivial. More progress? To use the work, don't have to be an expert in the system being monitored, don't have anything like a model, and get to detect problems never seen before, sometimes called zero day problems. So, just have some random variables, exchangability, and the Ulam result. But, you are correct: Rules, models, thresholds, etc. are what is popular; the Ulam result is not very popular! My paper has been in the literature for many years now, and I suspect that so far the number of readers is 0.00. Not really popular! Not likely to compete with the new Tom Cruise movie! Ah, I forgot: I also worked up a relatively fast algorithm for the core calculations! I learned my lesson and am doing things quite different now! I started with a problem some hundreds of millions of people would like solved! At Easter, I talked to a guy who works in computer and network security. I asked him what would be the security risks if a computer with an up to date instance of Windows Server as a Web server just openly faced the Internet? His reaction was that the results would be multiple disasters from attackers around the world. He mentioned "SQL injection" -- gads, I thought I was quite careful when I took the data from the user and constructed my SQL queries! So, near the top of my TODO list is just to write a little .NET code, listen on some IP ports, scribble what comes in to a file, and see what I get! Anyone want to predict what I'm likely to get?

tptacek 4y ago

This is one of the oldest problems in network security. Rule-based detection systems converge on antivirus's effectiveness (or lack thereof). But anomaly systems have almost universally failed in practice, no matter what the anomaly model is, and people have come up with lots of them. I can rattle off reasons why model-based systems are hard to operationalize.

graycat 4y ago
Yes, monitoring, anomaly detection, stopping computer viruses, intrusion detection, etc. are old problems. And rules have long been the relatively sophisticated solution technique, i.e., more than just thresholds! And false alarm rate and detection rate have not played much of a role in the discussions. And attacking the problems assuming knowing some relatively fine details about the system, e.g., some Web server, that is the target of the work and the problems want to detect and correct has been common. E.g., when have an instance of some bad stuff that didn't detect soon enough but want to, analyze that problem or class of similar problems and build a detector just for that. And, yup, for that particular problem, might have a nicely low false alarm rate and relatively high detection rate -- on this class of problems seen before, analyzed, and understood. So, here maybe should say that have built a model. But I was not impressed with rules, didn't want to need a lot of fine expertise in the target system, and wanted to detect also problems never seen before, with some good progress on rates of false alarms and miss detections. Yes, what I did is not easily called "model-based" and was supposed to be "new, correct, and significant". Maybe it was at least partly new; maybe it still is. I started with the data being random variables -- that's not assuming much! Then would like to have the assumptions of independent and identically distributed (i.i.d.) random variables, but for practice that is asking a bit too much! And for some positive integer n, having i.i.d. random n-tuples seemed a bit much! Well, I used something weaker, but still strong enough, with some math, to get some results, called exchangability. So, right, my techniques are distribution free, that is, assume nothing about probability distributions and make no attempt to estimate such. Gee, I certainly didn't assume a Gaussian distribution! I was able to get false alarm rate adjustable and known exactly in advance. Looked like progress to me! I had some real data, fed it into my software, and, presto, bingo, the false alarm rate was as adjusted and desired! And using a classic result of S. Ulam, I was able to show that for detection rate the math was not trivial, that is, was better than merely trivial. More progress? To use the work, don't have to be an expert in the system being monitored, don't have anything like a model, and get to detect problems never seen before, sometimes called zero day problems. So, just have some random variables, exchangability, and the Ulam result. But, you are correct: Rules, models, thresholds, etc. are what is popular; the Ulam result is not very popular! My paper has been in the literature for many years now, and I suspect that so far the number of readers is 0.00. Not really popular! Not likely to compete with the new Tom Cruise movie! Ah, I forgot: I also worked up a relatively fast algorithm for the core calculations! I learned my lesson and am doing things quite different now! I started with a problem some hundreds of millions of people would like solved! At Easter, I talked to a guy who works in computer and network security. I asked him what would be the security risks if a computer with an up to date instance of Windows Server as a Web server just openly faced the Internet? His reaction was that the results would be multiple disasters from attackers around the world. He mentioned "SQL injection" -- gads, I thought I was quite careful when I took the data from the user and constructed my SQL queries! So, near the top of my TODO list is just to write a little .NET code, listen on some IP ports, scribble what comes in to a file, and see what I get! Anyone want to predict what I'm likely to get?