5 ms·

My graduate advisor told me that I should be designing experiments that had counterintuitive results that were solidly significant. I don't see how you can loo

My graduate advisor told me that I should be designing experiments that had counterintuitive results that were solidly significant. I don't see how you can look for a specific kind of outcome from an experiment without fooling yourself, or cheating.

cryoshon 9y ago
>My graduate advisor told me that I should be designing experiments that had counterintuitive results that were solidly significant. This is the advice of a true scientist. I'm refreshed to see that they're still out there.
- Filligree 9y ago
  > This is the advice of a true scientist. I'm refreshed to see that they're still out there. It encourages students to cheat, because most of the time you won't get either. Not a significant result, and usually not a counterintuitive one either. You're pretty likely to get a significant result that agrees with existing theory, though.
  - richardwhiuk 9y ago
    By significant, you mean statistically, not to people in general.
    - vorotato 9y ago
      Kind of bizzare that we don't want to report where we weren't able to find a statistically significant relationship.
    - Filligree 9y ago
      Of course.
PeterisP 9y ago
Before doing an experiment, you can often see that with your settings, if you'd get the counterintuitive result, then it wouldn't be statistically significant, so you'd need to plan a larger experiment or not do it at all. You can look at any experiment as having essentially two possible outcomes - you need to ensure that at least one of those outcomes is publishable, because there are many experiments where any possible outcome will be weak.
- pbhjpbhj 9y ago
  You "need" a tentatively supporting result so you can say "more work is necessary", if you get conclusive results it's harder to get funded to work on a new topic I imagine?
carlmr 9y ago
My advisor wanted me to build something on his theory on a particular system with a particular problem. I showed him it wasn't possible. He even made a simple problem more complex just to fit his base theory. It's only about getting citations.
- noam87 9y ago
  What field?
nostrademons 9y ago
You can if you admit the possibility of failure. Design an experiment that if it gets a significant result, that result will be solidly significant and counterintuitive. The flip side of that is that most of your results will not be significant, i.e. they will support the accepted scientific consensus. (And these should be published too - lack of interest in reproduced results is what triggered the reproduction crisis.) It's much like with tech startups - you can design a startup such that it will either be worth billions or zero by tackling a problem that most people believe to be unsolvable but will affect lots of people, and then trying multiple counterintuitive ways of solving it. The vast majority of them will fail because that's what it means to be counterintuitive, but if it succeeds, you have a world-changing company. Financial instruments as well - an option is a financial contract whose value will either be zero or a lot, depending on whether an unlikely-but-possible event occurs.
- freshhawk 9y ago
  Sooner or later everyone will get off their collective asses and start registering studies before they are performed so that this is enforced and tracked. Hopefully.
  - tripzilch 9y ago
    Funnily enough, some research into parapsychology (and similar)--being, the serious research that goes into stuff barely at the edge of pseudoscience, does exactly this. Preregistration is an important part to even begin having any experiments into "remote sensing" or "extrasensory perception" taken seriously by the community. The "serious" part of it, any way. There's a bunch of statisticians who love analysing this kind of thing, and there's parapsychology researchers that love drafting up experiments that challenge and skirt the edges of experimental science. Not nearly enough of them of course. But still, it's kind of funny how certain fields in pseudoscience actually drive progress in statistics and scientific methodology :)
claytonjy 9y ago
It's easy to assume the worst of researchers, but I think Gelman's 2013 "Garden of Forking Paths"[1] does a good job of showing how it doesn't have to be intentional misconduct. It more likely or more often comes down to implicitly selecting your analysis plan given the data, and not correcting for the (hidden) multiple comparison problem that arises when doing frequentist statistical tests in that reality: the theory behind null-hypothesis testing and p-values dictates you account for all the other analytical choices you could have made for that data, or would have made in the face of other datasets. Gelman's other point is this is all made much worse in inherently low-power fields like psychology and the social sciences. A field like physics can "get away with" it more because making the appropriate corrections is less likely to wildly change the conclusions. [1]: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf http://www.stat.columbia.edu/~gelman/research/unpublished/p_...
Fomite 9y ago
One piece of advice I often give to my graduate students is "Pick problems where 'Yes' and 'No' are equally interesting answers."

cryoshon 9y ago

>My graduate advisor told me that I should be designing experiments that had counterintuitive results that were solidly significant. This is the advice of a true scientist. I'm refreshed to see that they're still out there.

Filligree 9y ago
> This is the advice of a true scientist. I'm refreshed to see that they're still out there. It encourages students to cheat, because most of the time you won't get either. Not a significant result, and usually not a counterintuitive one either. You're pretty likely to get a significant result that agrees with existing theory, though.
- richardwhiuk 9y ago
  By significant, you mean statistically, not to people in general.
  - vorotato 9y ago
    Kind of bizzare that we don't want to report where we weren't able to find a statistically significant relationship.
  - Filligree 9y ago
    Of course.

PeterisP 9y ago

Before doing an experiment, you can often see that with your settings, if you'd get the counterintuitive result, then it wouldn't be statistically significant, so you'd need to plan a larger experiment or not do it at all. You can look at any experiment as having essentially two possible outcomes - you need to ensure that at least one of those outcomes is publishable, because there are many experiments where any possible outcome will be weak.

pbhjpbhj 9y ago
You "need" a tentatively supporting result so you can say "more work is necessary", if you get conclusive results it's harder to get funded to work on a new topic I imagine?

carlmr 9y ago

My advisor wanted me to build something on his theory on a particular system with a particular problem. I showed him it wasn't possible. He even made a simple problem more complex just to fit his base theory. It's only about getting citations.

noam87 9y ago
What field?

nostrademons 9y ago

You can if you admit the possibility of failure. Design an experiment that if it gets a significant result, that result will be solidly significant and counterintuitive. The flip side of that is that most of your results will not be significant, i.e. they will support the accepted scientific consensus. (And these should be published too - lack of interest in reproduced results is what triggered the reproduction crisis.) It's much like with tech startups - you can design a startup such that it will either be worth billions or zero by tackling a problem that most people believe to be unsolvable but will affect lots of people, and then trying multiple counterintuitive ways of solving it. The vast majority of them will fail because that's what it means to be counterintuitive, but if it succeeds, you have a world-changing company. Financial instruments as well - an option is a financial contract whose value will either be zero or a lot, depending on whether an unlikely-but-possible event occurs.

freshhawk 9y ago
Sooner or later everyone will get off their collective asses and start registering studies before they are performed so that this is enforced and tracked. Hopefully.
- tripzilch 9y ago
  Funnily enough, some research into parapsychology (and similar)--being, the serious research that goes into stuff barely at the edge of pseudoscience, does exactly this. Preregistration is an important part to even begin having any experiments into "remote sensing" or "extrasensory perception" taken seriously by the community. The "serious" part of it, any way. There's a bunch of statisticians who love analysing this kind of thing, and there's parapsychology researchers that love drafting up experiments that challenge and skirt the edges of experimental science. Not nearly enough of them of course. But still, it's kind of funny how certain fields in pseudoscience actually drive progress in statistics and scientific methodology :)

claytonjy 9y ago

It's easy to assume the worst of researchers, but I think Gelman's 2013 "Garden of Forking Paths"[1] does a good job of showing how it doesn't have to be intentional misconduct. It more likely or more often comes down to implicitly selecting your analysis plan given the data, and not correcting for the (hidden) multiple comparison problem that arises when doing frequentist statistical tests in that reality: the theory behind null-hypothesis testing and p-values dictates you account for all the other analytical choices you could have made for that data, or would have made in the face of other datasets. Gelman's other point is this is all made much worse in inherently low-power fields like psychology and the social sciences. A field like physics can "get away with" it more because making the appropriate corrections is less likely to wildly change the conclusions. [1]: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf http://www.stat.columbia.edu/~gelman/research/unpublished/p_...

Fomite 9y ago

One piece of advice I often give to my graduate students is "Pick problems where 'Yes' and 'No' are equally interesting answers."