3 ms·
A story of my last use of Standard Deviation. My day to day job deals with credit card processing. A lot of my job deals with ensuring that transactions occur
by jasonlotito 16y ago
A story of my last use of Standard Deviation.
My day to day job deals with credit card processing. A lot of my job deals with ensuring that transactions occur securely and reliably. It's a tedious job, it's not exciting, and involves a lot of testing, but I love the sense of knowing that actual money is flowing through a system I built.
Anyways, one of the things I wanted to do was build an automate alert system that would notify me of problems with processing transactions. Looked at it from high up, the system is fairly stable, and while it would be easy to notice if transactions suddenly stopped on the entire system, this rarely, if ever, actually happens (and hasn't happened except for planned < 1 minute outages).
However, consider the system runs through it many small individual sites, looking at all transactions is fairly useless. Instead, I wanted a warning to notify me when any specific account was suffering. Each account is different, and accounts for a variable number of transactions each day. Some accounts do more, some less. Their are other variables: some accounts do well at different types of the day because of where they are promoted in the world. Weekends generally see an uptick, but this again is variable.
So, I developed a system (using standard deviation) that essentially looked at an accounts history for the past X time period for a certain period of time throughout the day. Some accounts are inspected by looking at the numbers in the past hour (accounts with steady transactions), others are looked at over the last few hours, others are looked at over the day.
Obviously, we don't alert ourselves to certain cases that fall outside standard deviation, and we've adjusted the numbers to look at other areas, but the result of using standard deviation in this way suddenly opened up a new way at looking at our numbers and evaluating the current status of our system, as well as the accounts using it. Even if a problem doesn't exist on our end, we can alert the people that handle these accounts that a problem might exist, allowing them to take necessary action.
Understanding standard deviation, understanding how it can be used (along with other associated tools of math) makes for some really interesting things that you can do to improve your system as a whole.
- tricky 16y agoA similar system was put in place by a pharmacy system to detect employee narcotics abuse. They'd throw an alarm when a pharmacist started dispensing pills at a rate outside their SD. The system worked great except in the case where a pharmacist slowly ramped up usage as his addiction worsened. Seems like that's a feature that works well in your case. As a business sells more stuff, their transactions will ramp up and your model will still hold. nice.
- jasonlotito 16y agoThat's one of the cases we don't alert ourselves too: getting a lot of transactions is a good thing. =) In the reverse, however, is another issue entirely. A site that is slowly getting less traffic won't be alerted. However, this is where we have to draw the line. We aren't looking for long term trends. Rather, immediate issues. Basically, if a site that normally has 20 attempted transactions during the last hour suddenly has 0, is this a problem? There is still a lot of tweaking that needs to go into the system. We're also exploring how to use this in other areas as well. Since all these accounts are internal, it's easy for me to get feedback. The end goal, however, is to be notified as soon as possible when something occurs that will negatively affect our bottom line.
- NY_Entrepreneur 16y agoThere's a much more mature way to approach your problem: What you have is a problem in statistical hypothesis testing. So, you want to start by stating and assuming a null hypothesis. This hypothesis is for a particular account and assumes that the account is normal, that is, that there is nothing wrong with it, that is, that any problems are null. With this assumption, you will get a lot of additional information that will let you calculate some probabilities and test the hypothesis. In this process, there are two ways to be wrong: (1) False Alarm. You conclude that there is something wrong with the account when it is normal. A false alarm is also called Type I error. Call the conditional probability of a false alarm given that the account is normal the false alarm rate. Commonly the numerical value is denoted by alpha. (2) Missed Detection. You conclude that the account is normal when there is a problem with it. A missed detection is also called Type II error. Call the conditional probability of a missed detection given that there is a problem with the account the missed detection rate. The numerical value is commonly denoted by beta. Typically you can design a hypothesis test so that alpha is adjustable and known in advance. Here's how the test calculation goes: Given the data on one user and assuming the null hypothesis, calculate the probability of getting data such as observed or worse. If this probability is really small, then we argue either (1) the account is normal but we observed something very rare or (2) the account has something wrong with it. If the calculated probability is very small, then we reject (2), conclude (1), and raise an alarm. In more detail, before we do the test, we select a geometric region, the critical region, for the data we will observe. We know that for a normal account, the probability of observing data in the critical region is just alpha. So, we take the data on the user and see if it falls in the critical region. If so, then we raise an alarm. Else we don't. This way we do get false alarm rate alpha. What beta we have for this test, however, is not yet determined; tests vary. For each false alarm rate alpha, you want the best or most powerful test you can get, that is, the smallest rate of missed detections beta. How to construct such a test is given in general but somewhat abstract terms by the famous Neyman-Pearson lemma. There are elementary proofs in most intermediate texts on mathematical statistics. The best proof I know I wrote out in terms of the Hahn decomposition in measure theory (it is an easy result of the Radon-Nikodym theorem with a famous proof by von Neumann). In practice, given essentially any calculation at all of the available data, you can estimate the false alarm rate alpha easily enough: Just take some old data on accounts that time, etc., have shown are good, feed that data into your calculation, and see what fraction of the calculations yield alarms. It is possible to test the detection rate similarly, but the number of bad accounts is typically much smaller meaning that getting a good estimate of the detection rate beta is harder. In the classic hypothesis tests, one important approach is to assume that we know the probability distribution of our data for accounts that are normal. Then we get to assume this distribution in our null hypothesis. When we do not assume such a distribution, then our test is distribution-free and, thus, more general. Another issue is the number of variables in our test: The classic tests commonly used just one variable so that the critical region was, typically, just one or two intervals on the line. But clearly using several variables and, for the critical region, something complicated, even a fractal, in several dimensions can give a more powerful test. Tests that are both distribution-free and that use several variables are rare. I developed a large family of such tests. Your problem is a special case of monitoring in near real time for problems, many or most of which have never been seen before. So, such monitoring can be important for server farms, networks, the financial system, and parts of national security. Once I gave a talk on my techniques at the Institute for Defense Analysis, and several persons in the audience thought of applications to problems such as yours. There is an intuitive approach: Suppose from the data on a user we get two variables. Suppose the probability density of these two variables for normal accounts looks something like the Shenandoah Mountains. Take the lowest point in the mountains and, then, the volume of the mountains above this level, and call that volume 1. Now pour water into the mountains and have it reach a level. Let this water cover the fraction alpha of the mountains. Let the wet area be our critical region. So, we get the largest critical region we can for the volume alpha we have covered. There is a crude optimality property here; this property can be useful for detecting zero day problem where we have no information on non-normal accounts. As we know from Mandelbrot, lake boundaries are typically fractals. So, our critical region can be a fractal. Does this mean that the mountains are rough? Actually no: It is possible for the mountains to be infinitely differentiable and the lake boundary still a fractal, e.g., the Mandelbrot set. That is, for any closed subset C of R^n, there exists a real value function f: R^n --> [0, infinity) so that f(x) = 0 for x in C and f(x) > 0 otherwise. But in your case, you have a lot of data on both normal accounts and bad ones. So, you want a means of separating such data. Again, the Neyman-Person lemma is first-cut guidance. In all of this, standard deviation will likely play little or no role unless somewhere you can justify a Gaussian assumption. However, if you can justify a Gaussian assumption, be aware that sample mean and sample standard deviation are sufficient statistics (more from the Radon-Nikodym theorem) for the Gaussian distribution so that, put simply, your test need use only sample mean and standard deviation.