5 ms·

No this does NOT have to do with sample size. It's about picking a theory, then testing that theory only, rather than trying to seek a theory in data that alre

No this does NOT have to do with sample size. It's about picking a theory, then testing that theory only, rather than trying to seek a theory in data that already exists. At the end of the article it explicitly says this and gives three specific ways to solve the problem.

Xurinos 16y ago
You're not wrong. I get the initial thrust. The problem is in the solutions. The measuring of those things at the end could very well fall into the realm of "streaks", too, unless we have good consideration of sample size. To use one of your examples: Let's say that I do 10 coin flips to determine bias, and I see those 10 heads in a row. Now I want to test this situation further. Is my coin biased? I run the test 10 more times. What happens if I get a "streak" of 10 heads again? Have I verified my claim of bias? No! I just have not flipped my coin a sufficient number of times. From this I see two things: One, an interesting anomaly arose that suggested I need to look into things a bit more. My theory is that the coin is biased towards heads. Two, to solve the problem, I need to narrow down variables where I can (you mention) and perform the test enough times (you did not mention). We know that 10 is not enough. How do we know this? 1000 is arbitrary. What is the right number? In other words, I agree with the fallacy you pointed out. The solutions are just insufficient. The fallacy can continue with the right probability of occurrences, even if someone tried to narrow down the variables.
- metellus 16y ago
  What you're talking about is called hypothesis testing. Basically, you can figure out exactly how unlikely a given event is under a certain set of assumptions (in this case, how unlikely it would be for a fair coin to show heads 10 times in a row). If it is sufficiently unlikely, where you and people reading your work decide what "sufficiently unlikely" means, you can claim that your initial assumptions were incorrect. I've been away from statistics for too long to go into specifics, but you can mathematically determine how large a sample size you need to be X percent confident that something weird is happening. I'd suggest reading http://en.wikipedia.org/wiki/Hypothesis_test http://en.wikipedia.org/wiki/Hypothesis_test for more info.
jessriedel 16y ago
There's not really an important distinction between seeking a theory in data which already exists and choosing a theory and then collecting data except that the later generally implies a large amount of data per hypothesis. In particular, there's nothing wrong with the OP's example of picking 10 variants of an ad (i.e, testing 10 hypotheses) and finding out which one is better so long as you have enough data. It's really a matter of sample size.
tel 16y ago
No, sample size is intimately tied to this fallacy. Each observation can be loosely thought to lend power to whatever analysis you choose to run, even those including very large numbers of possible explanations. For instance: You know, Rodriguez is 7 for 8 against left-handed pitchers in asymmetric ballparks when the tide is going out during El Niño. is a valid statement if the model you're using includes factors for pitcher-handedness, ballpark design, and weather patterns. This is an extraordinarily broad model though almost every observation is at least a little bit novel at first, so you need a great deal of power to distinguish truly interesting things. If you sample sufficiently (exercise left to the reader) and randomly across all those effects you'd serve a chance of learning something about how they correlate in a statistically profound fashion. The barb of the fallacy is forgetting to watch your possibility space grow as you reach for new explanations. The moment you mention the weather you become beholden to divide your certainty by the total number of weather patterns. Unless you're very careful, that number is usually very, very large.
moultano 16y ago
Well, that is what the article says, but that's flagrantly incorrect.
btilly 16y ago
You missed the standard solution used in machine learning. Randomly divide your data into two halves. Train your model on one half. Then test it on the other. This eliminates most of the randomly discovered patterns. It also leads to a much better sense of exactly how many different ideas you tried, and therefore how high the level of statistical evidence needs to be on your successes.

Xurinos 16y ago

You're not wrong. I get the initial thrust. The problem is in the solutions. The measuring of those things at the end could very well fall into the realm of "streaks", too, unless we have good consideration of sample size. To use one of your examples: Let's say that I do 10 coin flips to determine bias, and I see those 10 heads in a row. Now I want to test this situation further. Is my coin biased? I run the test 10 more times. What happens if I get a "streak" of 10 heads again? Have I verified my claim of bias? No! I just have not flipped my coin a sufficient number of times. From this I see two things: One, an interesting anomaly arose that suggested I need to look into things a bit more. My theory is that the coin is biased towards heads. Two, to solve the problem, I need to narrow down variables where I can (you mention) and perform the test enough times (you did not mention). We know that 10 is not enough. How do we know this? 1000 is arbitrary. What is the right number? In other words, I agree with the fallacy you pointed out. The solutions are just insufficient. The fallacy can continue with the right probability of occurrences, even if someone tried to narrow down the variables.

metellus 16y ago
What you're talking about is called hypothesis testing. Basically, you can figure out exactly how unlikely a given event is under a certain set of assumptions (in this case, how unlikely it would be for a fair coin to show heads 10 times in a row). If it is sufficiently unlikely, where you and people reading your work decide what "sufficiently unlikely" means, you can claim that your initial assumptions were incorrect. I've been away from statistics for too long to go into specifics, but you can mathematically determine how large a sample size you need to be X percent confident that something weird is happening. I'd suggest reading http://en.wikipedia.org/wiki/Hypothesis_test http://en.wikipedia.org/wiki/Hypothesis_test for more info.

jessriedel 16y ago

There's not really an important distinction between seeking a theory in data which already exists and choosing a theory and then collecting data except that the later generally implies a large amount of data per hypothesis. In particular, there's nothing wrong with the OP's example of picking 10 variants of an ad (i.e, testing 10 hypotheses) and finding out which one is better so long as you have enough data. It's really a matter of sample size.

tel 16y ago

No, sample size is intimately tied to this fallacy. Each observation can be loosely thought to lend power to whatever analysis you choose to run, even those including very large numbers of possible explanations. For instance: You know, Rodriguez is 7 for 8 against left-handed pitchers in asymmetric ballparks when the tide is going out during El Niño. is a valid statement if the model you're using includes factors for pitcher-handedness, ballpark design, and weather patterns. This is an extraordinarily broad model though almost every observation is at least a little bit novel at first, so you need a great deal of power to distinguish truly interesting things. If you sample sufficiently (exercise left to the reader) and randomly across all those effects you'd serve a chance of learning something about how they correlate in a statistically profound fashion. The barb of the fallacy is forgetting to watch your possibility space grow as you reach for new explanations. The moment you mention the weather you become beholden to divide your certainty by the total number of weather patterns. Unless you're very careful, that number is usually very, very large.

moultano 16y ago

Well, that is what the article says, but that's flagrantly incorrect.

btilly 16y ago

You missed the standard solution used in machine learning. Randomly divide your data into two halves. Train your model on one half. Then test it on the other. This eliminates most of the randomly discovered patterns. It also leads to a much better sense of exactly how many different ideas you tried, and therefore how high the level of statistical evidence needs to be on your successes.