3 ms·

Ah, we're getting a C in communications. > "does this bag of data have any properties?" WOW! That's one heck of a hard question! There can be a LOT of proper

by graycat 11y ago

Ah, we're getting a C in communications. > "does this bag of data have any properties?" WOW! That's one heck of a hard question! There can be a LOT of properties! No properties might mean random. So, one case is old random number testing! People settled on the Fourier test -- essentially hypothesis testing. You seem to be saying that hypothesis testing requires knowing something too specific about what is being looked for. Well, in some of the elementary stuff. But there's the old distribution free two sample tests: Joe teaches calculus to 30 students, and Mary teaches the same subject to 25 students. Is their different performances significant? Specific? We are not saying much about what is different, but we do get estimates of Type I error, that is, saying that there is a big difference when there is not. So, how? Put all 30 + 25 student scores (right, something specific, unidimensional, but see below) in a pot, stir briskly, pull out 30 and 25, that the difference in their averages, do this 1000 times, get an empirical distribution of the difference under the hypothesis that there is no difference, then see where the actual difference is in the empirical distribution and estimate false alarm rate. For measuring several quantities not just one, can look for a significant difference even if have no idea what the heck the cause is. So, are not being specific about a cause. Generally in making AI decisions, what to have some sense of being correct. A first cut approach is hypothesis testing. A second cut would be, say, looking for a diagnosis. So, have some data and 20 candidate diagnoses. So, do hypothesis tests on the appropriate data for each of the 20, maybe different parts of the data for each of the 20, and pick the diagnosis based on the Type I estimates. Hypothesis testing has a cute and powerful idea: Make an hypothesis of no effect and use that to calculate some probabilities. There's got to be a good role for that in AI.

cgearhart 11y ago
> That's one heck of a hard question! Exactly. That's the problem. It's not just a question of deciding whether the data is random or has some particular property, but trying to answer the question of whether there is _any_ signal buried in the data -- and then deciding whether that signal is meaningful. For the most part, unsupervised learning is more like data pre-processing than an alternative to hypothesis testing. General frequentist statistical techniques (including hypothesis testing, goodness of fit, etc.) are an integral part of evaluating the results of unsupervised learning.
- graycat 11y ago
  From a paper I published, go to a server farm and for three months collect data on each of 10 variables 100 times a second. Now, in real time, given values on each of those 10 variables, ask if the farm is sick or healthy? Get known, adjustable false alarm rate and a relatively high detection rate. So, it's a multi-dimensional, distribution-free hypothesis test. Don't know what patterns or properties are looking for and don't get a diagnosis, but still, if get a detection at a low false alarm rate, then have good evidence the farm is sick. So, this hypothesis test is not the same as you have described. So, is this unsupervised learning?

by graycat 11y ago

cgearhart 11y ago

> That's one heck of a hard question! Exactly. That's the problem. It's not just a question of deciding whether the data is random or has some particular property, but trying to answer the question of whether there is _any_ signal buried in the data -- and then deciding whether that signal is meaningful. For the most part, unsupervised learning is more like data pre-processing than an alternative to hypothesis testing. General frequentist statistical techniques (including hypothesis testing, goodness of fit, etc.) are an integral part of evaluating the results of unsupervised learning.

graycat 11y ago
From a paper I published, go to a server farm and for three months collect data on each of 10 variables 100 times a second. Now, in real time, given values on each of those 10 variables, ask if the farm is sick or healthy? Get known, adjustable false alarm rate and a relatively high detection rate. So, it's a multi-dimensional, distribution-free hypothesis test. Don't know what patterns or properties are looking for and don't get a diagnosis, but still, if get a detection at a low false alarm rate, then have good evidence the farm is sick. So, this hypothesis test is not the same as you have described. So, is this unsupervised learning?