3 ms·

So at least two people reading this seem to think it's about using science in the context of their pet peeves. It's not. It's about using a statistical test fo

So at least two people reading this seem to think it's about using science in the context of their pet peeves. It's not. It's about using a statistical test for a data dependent hypothesis and interpreting the test as if it were used for a data-independent hypothesis. That's all. It's not about using statistics in politics or finance. It's about first looking at the data, then formulating a hypothesis, then running a standard test which is based on the idea that you chose the hypothesis independently of the data. This is a problem in any field.

lrei 12y ago
I upvoted your comment but a problem in "any field" seems too strong a statement. AFAIK this isn't much of a problem in CS (my field) and never heard math or physics people complaining... Seems to only be problem in Social "Sciences" & Bio/Med where many (most?) results are statistical significance tests.
- anon4 12y ago
  I think yummifajitas means that if done in any field, it would be a problem, not that it is necessarily observed in every field.
  - lrei 12y ago
    Yes probably what he was saying. Also, my understanding - not expert, likely wrong - is that it is sometimes done in physics (e.g analyzing data from sensors). But haven't heard any complaints (which does not mean they don't exist). Someone else would need to weight in as this is just supposition and hearsay :)
- tveita 12y ago
  I bet you could find a fair number of statistics errors in machine learning and other empirical branches of CS.
anon4 12y ago
Was about to post this. Thanks.
noahl 12y ago
So actually, I think the idea that data-dependent hypotheses are bad is fundamentally wrong, and is based on a misunderstanding of probability. The reason you'd avoid data-dependent hypotheses is simple: if the data comes from a process with some sort of randomness in it, then there will usually be things that appear to be interesting patterns but are in fact random artifacts. If you look at your data, you may be tempted to formulate a hypothesis based on these random artifacts. It may pass statistical tests (because the data you have does contain the pattern), but it is not, in fact, causal. To avoid this, you maintain the discipline of only making hypotheses before you look at your data, which means that you can't see a random effect and then guess that it's real. The problem is, this doesn't mean that if your hypothesis passes a statistical test, the result must be causal. It only lowers the probability - there is still a chance that your hypothesis was wrong, but your data happens to contain a random fluctuation that makes it look right. The only way to protect against this danger is to continuously gather data and re-evaluate your hypotheses, while understanding that there is always some probability that the effect you think you see is really random noise. And once you're doing this continuous monitoring anyway, then there's no reason to reject data-dependent hypotheses. By definition, if the effect you're seeing is a random occurrence, then it should go away with more data. If it doesn't go away, then maybe you've found something that you wouldn't have been able to guess in advance, which is good! And if you see a random effect, form a hypothesis that passes some test, and then assume that your hypothesis must be true, then the problem is not your data, but rather that you misunderstand how probability works. In short, avoiding data-dependent hypotheses is a hack that only reduces the probability of an error that you should be avoiding entirely anyway. Once you accept this and start avoiding the error, there's no reason to avoid data-dependent hypotheses, and they can be quite useful.
- cja23 12y ago
  Yes! I think any good statistician would agree with all of that and emphasize the importance of how you present your conclusions. Forming "data-dependent hypotheses" is part of what most statistician's call "exploratory data analysis" (EDA). When presenting EDA findings, we should use terms like "association", "relationship", "correlation", "possible", and not words like "cause", "effect", "p-value", "test", etc.
- 12y ago
tansey 12y ago
Indeed! I have a particularly relevant horror story. For one of my graduate classes, I built a game where two people (a liar and a truth teller) would try to convince a third person (the judge) they had experienced something. The goal was to see if people used different language when lying than they did when telling the truth. It was a fun project and we were focused more on the NLP/ML side of things. Turns out to be reasonably possible to separate our ~500 examples via a linear SVM, with some interesting separating words. That's where our claims stopped. Then I took the results to a psychology prof. They loaded the data into SAS or something and it proceeded to perform HUNDREDS of independent t-tests. The results came out in a few seconds and the professor exclaimed "Oh look! Pronouns are statistically significant! Oh and possessive nouns too!" -- I cringed. The flip side of this is that as statisticians, we do know how to handle different kinds of testing. If you're going to be looking at all these different outcomes, that's fine, but you just need to correct for it. We've had Bonferroni correction since the 1960s; Benjamini-Hochberg and related false discovery rate methods have been around for almost 20 years now. In fact, there are even situations where the data can help define a prior for your hypothesis testing [1]. Lastly, there is this quote from the article: > There is no statistical quality board that could enforce such larger analyses—nor would we believe such coercion to be appropriate I'm not sure if that's such a bad idea. In theory, that should be the job of the journal reviewers and editor. In practice, it's often the blind leading the blind, with confirmation bias thrown in to boot. Maybe we need an SRB as a companion to IRB. [1] http://arxiv.org/abs/1411.6144 http://arxiv.org/abs/1411.6144
- mcguire 12y ago
  "Then I took the results to a psychology prof. They loaded the data into SAS or something and it proceeded to perform HUNDREDS of independent t-tests. The results came out in a few seconds and the professor exclaimed 'Oh look! Pronouns are statistically significant! Oh and possessive nouns too!' -- I cringed." This wasn't, by any chance, James W. Pennebaker from UT Austin? (Or somebody related to him?) I was just reading The Secret Life of Pronouns and his chapter on lying does point to pronouns as a key factor for distinguishing lying. On the other hand, he does take some pains to note that getting to 65% correctness is reasonably easy but getting much more has been impossible so far, outside of restricted environments such as laboratory conditions.

lrei 12y ago

I upvoted your comment but a problem in "any field" seems too strong a statement. AFAIK this isn't much of a problem in CS (my field) and never heard math or physics people complaining... Seems to only be problem in Social "Sciences" & Bio/Med where many (most?) results are statistical significance tests.

anon4 12y ago
I think yummifajitas means that if done in any field, it would be a problem, not that it is necessarily observed in every field.
- lrei 12y ago
  Yes probably what he was saying. Also, my understanding - not expert, likely wrong - is that it is sometimes done in physics (e.g analyzing data from sensors). But haven't heard any complaints (which does not mean they don't exist). Someone else would need to weight in as this is just supposition and hearsay :)
tveita 12y ago
I bet you could find a fair number of statistics errors in machine learning and other empirical branches of CS.

anon4 12y ago

Was about to post this. Thanks.

noahl 12y ago

So actually, I think the idea that data-dependent hypotheses are bad is fundamentally wrong, and is based on a misunderstanding of probability. The reason you'd avoid data-dependent hypotheses is simple: if the data comes from a process with some sort of randomness in it, then there will usually be things that appear to be interesting patterns but are in fact random artifacts. If you look at your data, you may be tempted to formulate a hypothesis based on these random artifacts. It may pass statistical tests (because the data you have does contain the pattern), but it is not, in fact, causal. To avoid this, you maintain the discipline of only making hypotheses before you look at your data, which means that you can't see a random effect and then guess that it's real. The problem is, this doesn't mean that if your hypothesis passes a statistical test, the result must be causal. It only lowers the probability - there is still a chance that your hypothesis was wrong, but your data happens to contain a random fluctuation that makes it look right. The only way to protect against this danger is to continuously gather data and re-evaluate your hypotheses, while understanding that there is always some probability that the effect you think you see is really random noise. And once you're doing this continuous monitoring anyway, then there's no reason to reject data-dependent hypotheses. By definition, if the effect you're seeing is a random occurrence, then it should go away with more data. If it doesn't go away, then maybe you've found something that you wouldn't have been able to guess in advance, which is good! And if you see a random effect, form a hypothesis that passes some test, and then assume that your hypothesis must be true, then the problem is not your data, but rather that you misunderstand how probability works. In short, avoiding data-dependent hypotheses is a hack that only reduces the probability of an error that you should be avoiding entirely anyway. Once you accept this and start avoiding the error, there's no reason to avoid data-dependent hypotheses, and they can be quite useful.

cja23 12y ago
Yes! I think any good statistician would agree with all of that and emphasize the importance of how you present your conclusions. Forming "data-dependent hypotheses" is part of what most statistician's call "exploratory data analysis" (EDA). When presenting EDA findings, we should use terms like "association", "relationship", "correlation", "possible", and not words like "cause", "effect", "p-value", "test", etc.
12y ago

tansey 12y ago

Indeed! I have a particularly relevant horror story. For one of my graduate classes, I built a game where two people (a liar and a truth teller) would try to convince a third person (the judge) they had experienced something. The goal was to see if people used different language when lying than they did when telling the truth. It was a fun project and we were focused more on the NLP/ML side of things. Turns out to be reasonably possible to separate our ~500 examples via a linear SVM, with some interesting separating words. That's where our claims stopped. Then I took the results to a psychology prof. They loaded the data into SAS or something and it proceeded to perform HUNDREDS of independent t-tests. The results came out in a few seconds and the professor exclaimed "Oh look! Pronouns are statistically significant! Oh and possessive nouns too!" -- I cringed. The flip side of this is that as statisticians, we do know how to handle different kinds of testing. If you're going to be looking at all these different outcomes, that's fine, but you just need to correct for it. We've had Bonferroni correction since the 1960s; Benjamini-Hochberg and related false discovery rate methods have been around for almost 20 years now. In fact, there are even situations where the data can help define a prior for your hypothesis testing [1]. Lastly, there is this quote from the article: > There is no statistical quality board that could enforce such larger analyses—nor would we believe such coercion to be appropriate I'm not sure if that's such a bad idea. In theory, that should be the job of the journal reviewers and editor. In practice, it's often the blind leading the blind, with confirmation bias thrown in to boot. Maybe we need an SRB as a companion to IRB. [1] http://arxiv.org/abs/1411.6144 http://arxiv.org/abs/1411.6144

mcguire 12y ago
"Then I took the results to a psychology prof. They loaded the data into SAS or something and it proceeded to perform HUNDREDS of independent t-tests. The results came out in a few seconds and the professor exclaimed 'Oh look! Pronouns are statistically significant! Oh and possessive nouns too!' -- I cringed." This wasn't, by any chance, James W. Pennebaker from UT Austin? (Or somebody related to him?) I was just reading The Secret Life of Pronouns and his chapter on lying does point to pronouns as a key factor for distinguishing lying. On the other hand, he does take some pains to note that getting to 65% correctness is reasonably easy but getting much more has been impossible so far, outside of restricted environments such as laboratory conditions.