4 ms·
The materials I listed never or nearly never mention 'Bayesian statistics', 'subjective probabilities', or 'prior probabilities'. For what has been done in sta
by HilbertSpace 17y ago
The materials I listed never or nearly never mention 'Bayesian statistics', 'subjective probabilities', or 'prior probabilities'.
For what has been done in statistics over the past 100 years or so, each research library has a large section of books and journals. Here my interest was to respond to the question about how to get started in statistics and to outline a future for statistics, especially for exploiting Moore's law for more in economic productivity.
For what I have done in statistics, my interests are in business, and there a good application mostly means starting a new business. I'm doing that, but I'm not supposed to describe the 'secret sauce' in public.
I can give an introduction that might be of some interest in computer science and practical computing.
Given a 9th grade math teacher and one class with 20 boys and another class with 18 girls, do the boys and girls do the same or is there a difference? Uh, maybe as in the URL I gave, some 'feminists' will be very picky about any claims of a difference!
Broadly what we do is make a 'hypothesis' that gives us enough in mathematical assumptions to do some probability calculations. This hypothesis is called the 'null' hypothesis apparently because we intend to reject it, that is, find it to be 'null'. Our intention is to conclude that the hypothesis leads to something of very low probability, so low we reject the hypothesis. Then we know something that appears to be false. Yes, this is less good and complete knowledge than we could want, but maybe this result is good considering how little in data and assumptions we used!
So, put all 20 + 18 scores in a pot, stir the pot, pull out 18 scores and average, average the remaining 20, take the difference in the averages, do this maybe 1 million times (thanking Moore's law), get the empirical distribution of the differences, pick a small number, say, 1%, for the really angry feminists, 0.1%, pick the region in the 'tails' with this fraction of the differences, get the difference for the real data before stirring, and see where it is. If that real difference is in the region, then have some bad news for the angry ones: Either boys and girls are not the same or they are the same and we have obsereved something rare, too rare to be believed. Amazing that anyone would suspect that by the 9th grade boys and girls were not the same!
Now, make some mathematics out of this!
When the hypothesis is true, we reject it, and make an error, 1% (or 0.1% or whatever) of the time. So, we have rejected the null hypothesis when it is true; this is called Type I error. Yup, the other possible error is to accept the null hypothesis when it is false, and that is Type II error. Semi-, pseudo-, quasi-amazing.
Hmm .... Here we have an introduction to distribution-free, that is, 'non-parametric', hypothesis testing based on ranks or permutations.
The stirring of the pot is called 're-sampling'. Actually, when we do the mathematics, we will likely want all the combinations of 38 things taken 18 at a time, and that might be 33,578,000,610. So, instead of straining Moore's law with all 33 billion, we just 'sample', in this case, 're-sample'.
So, we see that we get to select the probability of Type I error, that is, the 1%, in advance and get what we select. Progress.
Now suppose the class of 20 boys also takes English, general science, and history. Similarly for the class of 18 girls. So, now on each student we have 4 scores instead of just 1. How how to do the test? Hmm ...!
Or, suppose we are given a server farm and a network. We select a 'system' we want to monitor in real-time for health and wellness. Suppose that this system can report data on each of 12 relevant variables 100 times a second.
Our null hypothesis is that the system is healthy. Then, an instance of Type I error is a 'false alarm'. Suppose we want the false alarm rate to be, say, 1 a month.
Then an instance of Type II error is a missed detection of a real problem.
Then for tolerating that rate of false alarms, we want the lowest rate of missed detections we can get.
So, how do we construct our monitoring system?
We would like to use the classic Neyman-Pearson result. Here, however, we are asked for complete information on when our system is 'sick', and likely we don't have that.
Still we can select our rate of false alarms and do something smart with the 12 variables on problems never seen before, i.e., 'zero-day' problems.
So, we have obtained some automation of system monitoring with adjustable, known false alarm rate and, if we look a little, with some nice guarantees on detection rate. Progress in 'computer science'!
- ced 17y agoFirst, thanks for the long answer. So, put all 20 + 18 scores in a pot, stir the pot, pull out 18 scores and average, average the remaining 20, take the difference in the averages, do this maybe 1 million times (thanking Moore's law) I can't believe there isn't a well-defined analytical expression for that... What you described sounds like the kind of inference I would tackle through Bayesian hypothesis testing, while you use... Monte Carlo? I suspect I'm missing something. Anyway, it sounds like a very interesting problem and if I were around, I'd ask for an interview. Good luck.
- HilbertSpace 17y ago"I can't believe there isn't a well-defined analytical expression for that..." Well, yes, there is some solid mathematics behind the pot stirring! Writing out some appropriate mathematics, will likely want all 33 billion combinations of 38 things taken 18 at a time. With this math, there is no use of 'prior probabilities', and the Monte Carlo is just a fast way to replace finding all 33 billion combinations. For more, can see (with TeX markup): E.\ L.\ Lehmann, {\it Nonparametrics: Statistical Methods Based on Ranks,\/} ISBN 0-8162-4994-6, Holden-Day, San Francisco, 1975.\ \ Jaroslav H\'ajek and Zbyn\v ek \v Sid\'ak, {\it Theory of Rank Tests,\/} Academia, Prague, 1967.\ \ Sidney Siegel, {\it Nonparametric Statistics for the Behavioral Sciences,\/} McGraw-Hill, New York, 1956.\ \ So, it's old material. There are many such hypothesis tests. But the old material essentially always has, for the student case, only one number on each student. The part about what to do when each student has 4 scores can take us into the journals and maybe start some more research. Similarly for the 12 numbers from the 'system' to be monitored.
- ced 17y agoWith this math, there is no use of 'prior probabilities' But the full hypothesis is "Given the data, are girls better than boys at this exam?" and clearly, the prior probability is relevant. Maybe in this case one might want to use a 50-50% prior, but in general, if the hypothesis was instead "Given the [same] data, can we conclude that this paranormal event really happened?" then a healthy skeptical prior would be in order. Anyway, regardless of the "prior" issue, I've thought some more about your original problem, and I'm not so sure about your methodology. From my perspective, if you want to reach a "girls better than boys on this test in this class - true or false" conclusion, then individual variance is a crucial issue. Assuming that all girls and boys would always get the exact same result were they to take the same test over and over again, then you have a variance of 0. And thus, one could simply check if (average of boys) < (average of girls) and conclude accordingly. At the other extreme, if students show huge individual variance (eg.: their score depends on whether they had breakfast that morning), then the test results are almost meaningless... So the outcome is crucially dependent on this variance, which your problem description makes no mention of, and which Monte Carlo methods really do nothing to recover. One would have to make assumptions about it. Maybe a better example (closer to what your other "system health" problem) would be: a boy takes an exam 20 times, a girl takes the exam 20 times, they get such and such results (assuming they don't improve inbetween). Is the girl better than the boy? Then one could assume a Gaussian distribution of test results for both, estimate their average and variance, then check for overlap between the 2 gaussians and conclude accordingly. Maybe your MC really does boil down to something similar, but I don't see it. And sadly, I can't quite construct an argument about what I think is problematic with it. It just doesn't feel right. Thanks for the references, I might check them eventually, though they seem too specialized for my needs. Someone recently posted this book on HN, and people had good things to say about it: The Elements of Statistical Learning - Data Mining, Inference, and Prediction http://www-stat.stanford.edu/~tibs/ElemStatLearn// http://www-stat.stanford.edu/~tibs/ElemStatLearn// It's next on my reading list, but was absent from yours. It's available as a PDF --- the determining factor for someone with no library access.