4 ms·

>it often seems that studies like this are looking for a predetermined outcome that will reinforce the preconceived ideas of the authors. Isn't this the defini

by pedrocr 11y ago

>it often seems that studies like this are looking for a predetermined outcome that will reinforce the preconceived ideas of the authors. Isn't this the definition of how science should work. 1) Formulate an hypothesis, 2) design an experiment to prove it 3) and then run it rigorously and accept whatever is the result. This case (and many others) failed 3) but everything else seems more than natural. Physicists smashing particles at the LHC were also trying to "reinforce their preconceived ideas" of how particles work. They have the standard model and they want to validate it where possible.

remarkEon 11y ago
This is my sentiment exactly, and why I posted the original article and read the refuting paper [0]. There's been a flurry of posts on HN recently about hypothesis testing and people not understanding what p-values are and what these statistics really mean, and I feel like that sits within the same critique about this paper. People who manipulate their model spec to get a convenient p-value are just as guilty as those who completely fabricate data. [0] http://web.stanford.edu/~dbroock/broockman_kalla_aronow_lg_irregularities.pdf http://web.stanford.edu/~dbroock/broockman_kalla_aronow_lg_i...
datenwolf 11y ago
> Isn't this the definition of how science should work. 1) Formulate an hypothesis, 2) design an experiment to prove it 3) and then run it rigorously and accept whatever is the result. No. The critical difference lies in steps 2 and 3. You can never prove something in science, because you'll never get all the corner cases. What you're really after is designing an experiment that is able to falsify (=disprove) your hypothesis and run that. Because if you find one single instance that contradicts the original hypothesis, you know that your hypothesis in its current form is wrong, because it can't be applied universally.
- pedrocr 11y ago
  You're mixing up concepts. You want an experiment that can falsify the hypothesis, because if it can't it's worthless. That doesn't mean you don't run the experiment to give evidence to the hypothesis. So in the LHC case the hypothesis is "the Higgs exists and it's energy is X". Then they run the experiment and get evidence that says "there is a very high probability that there is a particle with energy X" and so their initial hypothesis is considered proven. If you don't like the word proven, then they conclude it's extremely unlikely to be false.
barry-cotter 11y ago
You need to actually publish your findings too. Robert Putnam sat on the findings that diversity reduces trust for years trying to disprove it.
Lawtonfogle 11y ago
>Isn't this the definition of how science should work. There are two problems. First is the easy problem which is that #3 doesn't always happen. The data gets slightly cooked, the study gets redone, explanations are given as to explain away the results. The way it is done can be from outright dishonest to quite acceptable. But the bigger problem is that there are some hypotheses that are not thought up or tested, or are done so in low numbers. Namely ones that would be political/career suicide, but even those that could hurt funding or that would have biased a person from ever becoming a researcher. For example, say you wanted to test an experiment that gay women were more violent to their domestic partners that straight men, straight women, or gay men. Finally, even when results may be found, they will be attacked as being untrustable because the author must've been biased to even go looking for such results.
- pedrocr 11y ago
  Most of what you mention, while probably true is not the issue I was replying to. >Finally, even when results may be found, they will be attacked as being untrustable because the author must've been biased to even go looking for such results. Sure, and that may be the problem of looking for a predetermined outcome that haberman was referring to. If your testing methods are poor enough that spurious results are likely, then the fact that all your hypothesis are somehow biased is a problem. Though I'd say the problem is more with the testing methods than with the hypothesis. Using the LHC example again no one could attack Higgs for being biased as an argument for the boson not existing after the LHC results were in. The provenance of the hypothesis should be irrelevant if the testing methods are good enough to provide strong evidence for it.
CWuestefeld 11y ago
2) design an experiment to prove it I think this is incorrect. A good experiment is designed to disprove the hypothesis. And so if the experiment fails to disprove, we've got one more vote of confidence in the hypothesis. It's never possible to out-and-out prove any hypothesis. There's always going to be variables that we didn't think were relevant. All we can do is keep adding to the heap of evidence in its favor, by falsifying more and more objections to it.
- pedrocr 11y ago
  >A good experiment is designed to disprove the hypothesis. Statistical tests are often designed around a null hypothesis that is defined as the opposite of the hypothesis we are testing. Once we can reject that null hypothesis we have proven the hypothesis. So for the LHC you'd disprove something like "there is no particle of energy X" (the null hypothesis) by finding a definite signal at energy X, thus proving that the Higgs Boson exists (the initial hypothesis). Prove/Disprove are relative terms though as you can only have a degree of confidence X on the validity of the hypothesis.
sukilot 11y ago
No. Science is about answering interesting questions, not justifomying desired answers. Scientists should be invested in he question, not a preferred answer. Leave the answer preferences to engineers and inventors and businesses.
- pedrocr 11y ago
  This custom of starting comments with "No." is quite frustrating. Vehemently stating that someone else is wrong does not an argument make. Nor is the rest of the comment much of an argument. Scientists have questions they want answered and to solve them they formulate hypothesis and test them. If you don't formulate hypothesis you have nothing to test and can't advance.
haberman 11y ago
I think the problem is that it takes diligence to design an experiment that is not biased towards the desired result. I also think it's a systemic problem of positive results being far more rewarded (in terms of funding, prestige, etc) than negative results, which gives even more incentive to bias the result (intentionally or unintentionally) towards the positive, satisfying outcome. Imagine you have two social scientists, A and B. Scientist A publishes a positive result that resonates with people; it fits the social and political climate of the day. They give TED talks about it, and the result is reported in newspapers. Later experiments are unable to replicate the result, maybe because the original experiment was a little biased somehow, but nobody can say for sure and there is no scandal because there was no fraud. Scientist B is more rigorous in their experimentation and their work fails to demonstrate a hypothesis. They either publish the negative result, or don't publish at all. Which scientist will have advanced their career more? I think that is the problem.
- pedrocr 11y ago
  Yeah, but then the problem is that the experiments are crappy not that the hypothesis generation is at fault. You used social scientists and I used physicists and that is telling. The social "sciences" are often far from scientific. If a physicist suddenly goes bonkers and claims that space is filled with ether he can't go very far because he can't generate a spurious positive result in favor of his crazy hypothesis.
  - haberman 11y ago
    I used social scientists because that is what this article was about and what my comment was about. I agree that in physics (the hardest of hard sciences) it is harder to fall prey to this kind of false positive result.

by pedrocr 11y ago

remarkEon 11y ago

This is my sentiment exactly, and why I posted the original article and read the refuting paper [0]. There's been a flurry of posts on HN recently about hypothesis testing and people not understanding what p-values are and what these statistics really mean, and I feel like that sits within the same critique about this paper. People who manipulate their model spec to get a convenient p-value are just as guilty as those who completely fabricate data. [0] http://web.stanford.edu/~dbroock/broockman_kalla_aronow_lg_irregularities.pdf http://web.stanford.edu/~dbroock/broockman_kalla_aronow_lg_i...

datenwolf 11y ago

> Isn't this the definition of how science should work. 1) Formulate an hypothesis, 2) design an experiment to prove it 3) and then run it rigorously and accept whatever is the result. No. The critical difference lies in steps 2 and 3. You can never prove something in science, because you'll never get all the corner cases. What you're really after is designing an experiment that is able to falsify (=disprove) your hypothesis and run that. Because if you find one single instance that contradicts the original hypothesis, you know that your hypothesis in its current form is wrong, because it can't be applied universally.

pedrocr 11y ago
You're mixing up concepts. You want an experiment that can falsify the hypothesis, because if it can't it's worthless. That doesn't mean you don't run the experiment to give evidence to the hypothesis. So in the LHC case the hypothesis is "the Higgs exists and it's energy is X". Then they run the experiment and get evidence that says "there is a very high probability that there is a particle with energy X" and so their initial hypothesis is considered proven. If you don't like the word proven, then they conclude it's extremely unlikely to be false.

barry-cotter 11y ago

You need to actually publish your findings too. Robert Putnam sat on the findings that diversity reduces trust for years trying to disprove it.

Lawtonfogle 11y ago

>Isn't this the definition of how science should work. There are two problems. First is the easy problem which is that #3 doesn't always happen. The data gets slightly cooked, the study gets redone, explanations are given as to explain away the results. The way it is done can be from outright dishonest to quite acceptable. But the bigger problem is that there are some hypotheses that are not thought up or tested, or are done so in low numbers. Namely ones that would be political/career suicide, but even those that could hurt funding or that would have biased a person from ever becoming a researcher. For example, say you wanted to test an experiment that gay women were more violent to their domestic partners that straight men, straight women, or gay men. Finally, even when results may be found, they will be attacked as being untrustable because the author must've been biased to even go looking for such results.

pedrocr 11y ago
Most of what you mention, while probably true is not the issue I was replying to. >Finally, even when results may be found, they will be attacked as being untrustable because the author must've been biased to even go looking for such results. Sure, and that may be the problem of looking for a predetermined outcome that haberman was referring to. If your testing methods are poor enough that spurious results are likely, then the fact that all your hypothesis are somehow biased is a problem. Though I'd say the problem is more with the testing methods than with the hypothesis. Using the LHC example again no one could attack Higgs for being biased as an argument for the boson not existing after the LHC results were in. The provenance of the hypothesis should be irrelevant if the testing methods are good enough to provide strong evidence for it.

CWuestefeld 11y ago

2) design an experiment to prove it I think this is incorrect. A good experiment is designed to disprove the hypothesis. And so if the experiment fails to disprove, we've got one more vote of confidence in the hypothesis. It's never possible to out-and-out prove any hypothesis. There's always going to be variables that we didn't think were relevant. All we can do is keep adding to the heap of evidence in its favor, by falsifying more and more objections to it.

pedrocr 11y ago
>A good experiment is designed to disprove the hypothesis. Statistical tests are often designed around a null hypothesis that is defined as the opposite of the hypothesis we are testing. Once we can reject that null hypothesis we have proven the hypothesis. So for the LHC you'd disprove something like "there is no particle of energy X" (the null hypothesis) by finding a definite signal at energy X, thus proving that the Higgs Boson exists (the initial hypothesis). Prove/Disprove are relative terms though as you can only have a degree of confidence X on the validity of the hypothesis.

sukilot 11y ago

No. Science is about answering interesting questions, not justifomying desired answers. Scientists should be invested in he question, not a preferred answer. Leave the answer preferences to engineers and inventors and businesses.

pedrocr 11y ago
This custom of starting comments with "No." is quite frustrating. Vehemently stating that someone else is wrong does not an argument make. Nor is the rest of the comment much of an argument. Scientists have questions they want answered and to solve them they formulate hypothesis and test them. If you don't formulate hypothesis you have nothing to test and can't advance.

haberman 11y ago

I think the problem is that it takes diligence to design an experiment that is not biased towards the desired result. I also think it's a systemic problem of positive results being far more rewarded (in terms of funding, prestige, etc) than negative results, which gives even more incentive to bias the result (intentionally or unintentionally) towards the positive, satisfying outcome. Imagine you have two social scientists, A and B. Scientist A publishes a positive result that resonates with people; it fits the social and political climate of the day. They give TED talks about it, and the result is reported in newspapers. Later experiments are unable to replicate the result, maybe because the original experiment was a little biased somehow, but nobody can say for sure and there is no scandal because there was no fraud. Scientist B is more rigorous in their experimentation and their work fails to demonstrate a hypothesis. They either publish the negative result, or don't publish at all. Which scientist will have advanced their career more? I think that is the problem.

pedrocr 11y ago
Yeah, but then the problem is that the experiments are crappy not that the hypothesis generation is at fault. You used social scientists and I used physicists and that is telling. The social "sciences" are often far from scientific. If a physicist suddenly goes bonkers and claims that space is filled with ether he can't go very far because he can't generate a spurious positive result in favor of his crazy hypothesis.
- haberman 11y ago
  I used social scientists because that is what this article was about and what my comment was about. I agree that in physics (the hardest of hard sciences) it is harder to fall prey to this kind of false positive result.