4 ms·

Beautifully crafted web page. For any experimental science, the integrity of experiment (thus reliability of data) is important. For experiments with human su

by quanto 4y ago

Beautifully crafted web page. For any experimental science, the integrity of experiment (thus reliability of data) is important. For experiments with human subjects, the question is whether the subjects answered the questions in good faith. A sequence like 'HHHHHHHHHH' for the coin toss experiment looks like an answer in bad faith; it is mechanically easy to keep pressing the same button, and a subject is unlikely to think that such a sequence is a likely random sequence. Therefore, the replicating authors are fully justified to eliminate those poor faith answers. The original paper authors' claim that 'HHHHHHHHHH' is as equally probabilistic as any other sequence is irrelevant. Edited for clarity.

foerbert 4y ago
This is not a sound approach. You're declaring what humans think random is first, and then throwing out any data that doesn't match your declaration. There is no way to learn anything from this. I also think 'HHHHHHHHHH' is unlikely to be a good faith response, but if the goal is to actually learn anything instead of merely reinforcing my prior beliefs, it doesn't matter. You need to find a way to design the experiment that discourages bad faith answers or let's you judge them objectively. Alternatively if you have some outside knowledge about the 'shape' of bad faith answers for your kind of experiment, you may be able to use that to properly adjust your data. But 'nah I don't think so' isn't an acceptable reason to throw out data. It's especially egregious to do so when the data is answers that are, at a bare minimum, technically correct.
- lapsis_beeftech 4y ago
  It seems to me that if we reject a subset of experimental samples because they look like bad data (e.g. extreme outlier caused by sensor malfunction) we are still keeping all the bad data we are unable to recognize as such (e.g. sensor malfunctions producing less extreme data), which introduces a bias.
- alex_smart 4y ago
  You don't have to view it as "throwing out the data". You can just think of it as an alternative explanation for the data. Original hypothesis: Old people are worse at giving random responses. Alternative hypothesis: Old people are more likely give bad faith responses. This review is suggesting the AH is equally good at explaining the data as the OH.
  - native_samples 4y ago
    Although technically, this would be P-hacking. You aren't meant to change your hypothesis post-facto to fit the data. You'd have to conclude no effect, and then design a separate study to determine if age differences correlate with bad faith answers.
    - alex_smart 4y ago
      It would be p-hacking if we just took the same data to conclude that old people are more likely to give bad faith responses. That is just a possible explanation for the data being offered to reject the original hypothesis. At the very least, it is an interesting observation that the entire trend line disappears on removing data points where people guess all same coin toss results.
  - foerbert 4y ago
    I probably should have clarified that I was responding to the content of the parent comment rather than the submission itself. I think this is just the slop of language, but in this case it's obscuring all the important details so excuse me for being a bit pedantic. Forming and accepting a hypothesis are very different things. You can't just come up with a new hypothesis after looking at some data and then immediately accept it because the data supports it. It would absolutely be incorrect to look at the original data, form an alternate hypothesis, and then immediately go on to suggest it is an equally good explanation as the original hypothesis. You don't have to accept the original hypothesis if you think the experiment is flawed, and you're free to propose any hypothesis you want, but that's the limit without new data.
    - alex_smart 4y ago
      Since two people have come to the same misunderstanding, I must have worded my argument inadequately. Of course review is not the time to accept or form new hypotheses. Neither I nor the author of this article is suggesting that we should accept this new hypothesis "old people are more likely to give bad faith responses" from the data collected for this study. But review is the perfect time to look for interesting features in the data that challenge the original hypothesis. In this case, it is very difficult for the original hypothesis to explain why older people are only worse at giving random responses in a very specific way: giving answers that are all 0s or 1s.
Closi 4y ago
> The original paper authors' claim that 'HHHHHHHHHH' is as equally probabilistic as any other sequence is irrelevant. Agree with your post, but don't believe that the authors paper made this claim unless I am missing something?
roelschroeven 4y ago
If you're going to exclude bad faith answers, I think you should exclude all of them. But I don't think you can do that. Is HTHTHTHTHT a bad faith answer? Always or only sometimes? We're trying to infer the test subject's intent from their answer, and that's fundamentally impossible I think. I think including all answers is a solid approach. If test subjects have bad faith, I think that can be filed under 'less random'. If old test subjects show more bad faith, I think it's not really wrong to say older people are less random. And it does have predictive power. Arbitrarily (because they is no way to do it subjectively) excluding some answers and not others has, I think, a greater risk of skewing the results.
- mach1ne 4y ago
  However, the study concluded that person's ability to produce randomness peaks at 25. An increase in showing bad faith doesn't tell us anything about the ability to produce randomness if desired. Thus, if we accept the bad faith answers as part of the data, the conclusion of the study becomes incorrect, at least in wording.
jeremyjh 4y ago
It’s hard to believe the original authors made their argument in good faith. They probably ran the numbers with the filters and saw they wouldn’t have a paper that way.

by quanto 4y ago

foerbert 4y ago

This is not a sound approach. You're declaring what humans think random is first, and then throwing out any data that doesn't match your declaration. There is no way to learn anything from this. I also think 'HHHHHHHHHH' is unlikely to be a good faith response, but if the goal is to actually learn anything instead of merely reinforcing my prior beliefs, it doesn't matter. You need to find a way to design the experiment that discourages bad faith answers or let's you judge them objectively. Alternatively if you have some outside knowledge about the 'shape' of bad faith answers for your kind of experiment, you may be able to use that to properly adjust your data. But 'nah I don't think so' isn't an acceptable reason to throw out data. It's especially egregious to do so when the data is answers that are, at a bare minimum, technically correct.

lapsis_beeftech 4y ago
It seems to me that if we reject a subset of experimental samples because they look like bad data (e.g. extreme outlier caused by sensor malfunction) we are still keeping all the bad data we are unable to recognize as such (e.g. sensor malfunctions producing less extreme data), which introduces a bias.
alex_smart 4y ago
You don't have to view it as "throwing out the data". You can just think of it as an alternative explanation for the data. Original hypothesis: Old people are worse at giving random responses. Alternative hypothesis: Old people are more likely give bad faith responses. This review is suggesting the AH is equally good at explaining the data as the OH.
- native_samples 4y ago
  Although technically, this would be P-hacking. You aren't meant to change your hypothesis post-facto to fit the data. You'd have to conclude no effect, and then design a separate study to determine if age differences correlate with bad faith answers.
  - alex_smart 4y ago
    It would be p-hacking if we just took the same data to conclude that old people are more likely to give bad faith responses. That is just a possible explanation for the data being offered to reject the original hypothesis. At the very least, it is an interesting observation that the entire trend line disappears on removing data points where people guess all same coin toss results.
- foerbert 4y ago
  I probably should have clarified that I was responding to the content of the parent comment rather than the submission itself. I think this is just the slop of language, but in this case it's obscuring all the important details so excuse me for being a bit pedantic. Forming and accepting a hypothesis are very different things. You can't just come up with a new hypothesis after looking at some data and then immediately accept it because the data supports it. It would absolutely be incorrect to look at the original data, form an alternate hypothesis, and then immediately go on to suggest it is an equally good explanation as the original hypothesis. You don't have to accept the original hypothesis if you think the experiment is flawed, and you're free to propose any hypothesis you want, but that's the limit without new data.
  - alex_smart 4y ago
    Since two people have come to the same misunderstanding, I must have worded my argument inadequately. Of course review is not the time to accept or form new hypotheses. Neither I nor the author of this article is suggesting that we should accept this new hypothesis "old people are more likely to give bad faith responses" from the data collected for this study. But review is the perfect time to look for interesting features in the data that challenge the original hypothesis. In this case, it is very difficult for the original hypothesis to explain why older people are only worse at giving random responses in a very specific way: giving answers that are all 0s or 1s.

Closi 4y ago

> The original paper authors' claim that 'HHHHHHHHHH' is as equally probabilistic as any other sequence is irrelevant. Agree with your post, but don't believe that the authors paper made this claim unless I am missing something?

roelschroeven 4y ago

If you're going to exclude bad faith answers, I think you should exclude all of them. But I don't think you can do that. Is HTHTHTHTHT a bad faith answer? Always or only sometimes? We're trying to infer the test subject's intent from their answer, and that's fundamentally impossible I think. I think including all answers is a solid approach. If test subjects have bad faith, I think that can be filed under 'less random'. If old test subjects show more bad faith, I think it's not really wrong to say older people are less random. And it does have predictive power. Arbitrarily (because they is no way to do it subjectively) excluding some answers and not others has, I think, a greater risk of skewing the results.

mach1ne 4y ago
However, the study concluded that person's ability to produce randomness peaks at 25. An increase in showing bad faith doesn't tell us anything about the ability to produce randomness if desired. Thus, if we accept the bad faith answers as part of the data, the conclusion of the study becomes incorrect, at least in wording.

jeremyjh 4y ago

It’s hard to believe the original authors made their argument in good faith. They probably ran the numbers with the filters and saw they wouldn’t have a paper that way.