3 ms·

You don't have to view it as "throwing out the data". You can just think of it as an alternative explanation for the data. Original hypothesis: Old people are

You don't have to view it as "throwing out the data". You can just think of it as an alternative explanation for the data. Original hypothesis: Old people are worse at giving random responses. Alternative hypothesis: Old people are more likely give bad faith responses. This review is suggesting the AH is equally good at explaining the data as the OH.

native_samples 4y ago
Although technically, this would be P-hacking. You aren't meant to change your hypothesis post-facto to fit the data. You'd have to conclude no effect, and then design a separate study to determine if age differences correlate with bad faith answers.
- alex_smart 4y ago
  It would be p-hacking if we just took the same data to conclude that old people are more likely to give bad faith responses. That is just a possible explanation for the data being offered to reject the original hypothesis. At the very least, it is an interesting observation that the entire trend line disappears on removing data points where people guess all same coin toss results.
foerbert 4y ago
I probably should have clarified that I was responding to the content of the parent comment rather than the submission itself. I think this is just the slop of language, but in this case it's obscuring all the important details so excuse me for being a bit pedantic. Forming and accepting a hypothesis are very different things. You can't just come up with a new hypothesis after looking at some data and then immediately accept it because the data supports it. It would absolutely be incorrect to look at the original data, form an alternate hypothesis, and then immediately go on to suggest it is an equally good explanation as the original hypothesis. You don't have to accept the original hypothesis if you think the experiment is flawed, and you're free to propose any hypothesis you want, but that's the limit without new data.
- alex_smart 4y ago
  Since two people have come to the same misunderstanding, I must have worded my argument inadequately. Of course review is not the time to accept or form new hypotheses. Neither I nor the author of this article is suggesting that we should accept this new hypothesis "old people are more likely to give bad faith responses" from the data collected for this study. But review is the perfect time to look for interesting features in the data that challenge the original hypothesis. In this case, it is very difficult for the original hypothesis to explain why older people are only worse at giving random responses in a very specific way: giving answers that are all 0s or 1s.

native_samples 4y ago

Although technically, this would be P-hacking. You aren't meant to change your hypothesis post-facto to fit the data. You'd have to conclude no effect, and then design a separate study to determine if age differences correlate with bad faith answers.

alex_smart 4y ago
It would be p-hacking if we just took the same data to conclude that old people are more likely to give bad faith responses. That is just a possible explanation for the data being offered to reject the original hypothesis. At the very least, it is an interesting observation that the entire trend line disappears on removing data points where people guess all same coin toss results.

foerbert 4y ago

I probably should have clarified that I was responding to the content of the parent comment rather than the submission itself. I think this is just the slop of language, but in this case it's obscuring all the important details so excuse me for being a bit pedantic. Forming and accepting a hypothesis are very different things. You can't just come up with a new hypothesis after looking at some data and then immediately accept it because the data supports it. It would absolutely be incorrect to look at the original data, form an alternate hypothesis, and then immediately go on to suggest it is an equally good explanation as the original hypothesis. You don't have to accept the original hypothesis if you think the experiment is flawed, and you're free to propose any hypothesis you want, but that's the limit without new data.

alex_smart 4y ago
Since two people have come to the same misunderstanding, I must have worded my argument inadequately. Of course review is not the time to accept or form new hypotheses. Neither I nor the author of this article is suggesting that we should accept this new hypothesis "old people are more likely to give bad faith responses" from the data collected for this study. But review is the perfect time to look for interesting features in the data that challenge the original hypothesis. In this case, it is very difficult for the original hypothesis to explain why older people are only worse at giving random responses in a very specific way: giving answers that are all 0s or 1s.