4 ms·

> Maybe there should be a way to summarise null findings, to stop people replicating unsuccessful or unfounded work. But repeating failed experiments has the s

by lutusp 12y ago

> Maybe there should be a way to summarise null findings, to stop people replicating unsuccessful or unfounded work. But repeating failed experiments has the same scientific value as repeating successful ones -- it solidifies the result, increase our certainly about it. In real science, null results, and their replication, are extremely important. My favorite example is the Michelson-Morley experiment that expected to find differences in the speed of light due to the motion of the earth through the hypothetical ether. The failure of this experiment, its publication, and efforts to replicate the result, also dutifully published, finally created a revolution in physics when Einstein explained why there was a null result, and coincidentally gave birth to the modern world.

iamwithnail 12y ago
'Real science' - thanks for that. ;-) I think the issue I'm trying to get at - I suspect there is less value in null proving a lot of social science hypotheses - the impacts are smaller than physical constants of the universe (or finding say, that the Higgs Boson didn't exist) - if I prove, because my empirical work says so, that discourse can be meaningfully (for a given value of meaningful...) analysed in a new way, that's great, and will advance some other work. But proving (even repeatedly) that it can't be won't really change anything, imo. I guess though, the main issue is: I'm sure as hell not going to get published saying it, which ties back nicely to the point made in Nature, really.
- lutusp 12y ago
  > I suspect there is less value in null proving a lot of social science hypotheses Yes, but that applies equally to proving positive social science results. My point wasn't to compare social science to other sciences, it was only to point out the value of replicating null results in general -- that it stands equal to replicating positive results. > if I prove, because my empirical work says so, that discourse can be meaningfully (for a given value of meaningful...) analysed in a new way, that's great, and will advance some other work. But proving (even repeatedly) that it can't be won't really change anything, imo. Both findings have equal weight, therefore both findings deserve equal attention from replicators. By contrast, ignoring null results is a disaster for whatever science they occur in -- it may mean that other people might spend a lot of time spinning their wheels studying things that have already been found wanting.
  - JoeAltmaier 12y ago
    When you can identify a single null point to replicate, then sure. But there are lots of endeavors (what rate of positive feedback maximized reinforcing a behavior?) where there isn't a single null that defines a theory or disproves a reasonable conjecture. I'd guess in physics this might be possible, and math, but maybe nowhere else in science.
- marcosdumay 12y ago
  I don't understand your objection. Most research leads to no result. Most of the remaining research leads to only marginal results. Most of the research that leads to important results isn't still ground breaking. Only a tiny minority of the rare class of research that leads to important results is ground breaking. That's true in any field, in any period.
return0 12y ago
I heard a radical proposal from an academic recently: require every experiment to be reproduced by a second team before being published.
- p4bl0 12y ago
  That would be nice but it doesn't scale. For instance in particle physics they only have one LHC.
  - oldmanjay 12y ago
    So do it where it does scale. It seems anecdotally there's much less fakery happening in particle physics anyway.
  - lutusp 12y ago
    Yes, true, but experiments can be repeated there also, perhaps using different methods to give more insight.
  - dainucae 12y ago
    They only have one LHC, but they do have two detectors (ATLAS and CMS) which are operated by seperate groups.
  - marcosdumay 12y ago
    They have two main detectors on the LHC, with different charateristics, and several teams using different procedures to analyse the data. The finding of the Higgs, for example, was published by two teams.
  - return0 12y ago
    The teams working in separate detectors are supposed to work independently (so that they do not bias each other)
- lutusp 12y ago
  Yes, that resembles a now-famous suggestion by a Nobel Prizewinner with respect to psychological research: http://www.nature.com/news/nobel-laureate-challenges-psychologists-to-clean-up-their-act-1.11535 http://www.nature.com/news/nobel-laureate-challenges-psychol... Title: "Nobel laureate challenges psychologists to clean up their act" Quote: "For all these reasons, right or wrong, your field is now the poster child for doubts about the integrity of psychological research. Your problem is not with the few people who have actively challenged the validity of some priming results. It is with the much larger population of colleagues who in the past accepted your surprising results as facts when they were published. ... My reason for writing this letter is that I see a train wreck looming."

by lutusp 12y ago

iamwithnail 12y ago

'Real science' - thanks for that. ;-) I think the issue I'm trying to get at - I suspect there is less value in null proving a lot of social science hypotheses - the impacts are smaller than physical constants of the universe (or finding say, that the Higgs Boson didn't exist) - if I prove, because my empirical work says so, that discourse can be meaningfully (for a given value of meaningful...) analysed in a new way, that's great, and will advance some other work. But proving (even repeatedly) that it can't be won't really change anything, imo. I guess though, the main issue is: I'm sure as hell not going to get published saying it, which ties back nicely to the point made in Nature, really.

lutusp 12y ago
> I suspect there is less value in null proving a lot of social science hypotheses Yes, but that applies equally to proving positive social science results. My point wasn't to compare social science to other sciences, it was only to point out the value of replicating null results in general -- that it stands equal to replicating positive results. > if I prove, because my empirical work says so, that discourse can be meaningfully (for a given value of meaningful...) analysed in a new way, that's great, and will advance some other work. But proving (even repeatedly) that it can't be won't really change anything, imo. Both findings have equal weight, therefore both findings deserve equal attention from replicators. By contrast, ignoring null results is a disaster for whatever science they occur in -- it may mean that other people might spend a lot of time spinning their wheels studying things that have already been found wanting.
- JoeAltmaier 12y ago
  When you can identify a single null point to replicate, then sure. But there are lots of endeavors (what rate of positive feedback maximized reinforcing a behavior?) where there isn't a single null that defines a theory or disproves a reasonable conjecture. I'd guess in physics this might be possible, and math, but maybe nowhere else in science.
marcosdumay 12y ago
I don't understand your objection. Most research leads to no result. Most of the remaining research leads to only marginal results. Most of the research that leads to important results isn't still ground breaking. Only a tiny minority of the rare class of research that leads to important results is ground breaking. That's true in any field, in any period.

return0 12y ago

I heard a radical proposal from an academic recently: require every experiment to be reproduced by a second team before being published.

p4bl0 12y ago
That would be nice but it doesn't scale. For instance in particle physics they only have one LHC.
- oldmanjay 12y ago
  So do it where it does scale. It seems anecdotally there's much less fakery happening in particle physics anyway.
- lutusp 12y ago
  Yes, true, but experiments can be repeated there also, perhaps using different methods to give more insight.
- dainucae 12y ago
  They only have one LHC, but they do have two detectors (ATLAS and CMS) which are operated by seperate groups.
- marcosdumay 12y ago
  They have two main detectors on the LHC, with different charateristics, and several teams using different procedures to analyse the data. The finding of the Higgs, for example, was published by two teams.
- return0 12y ago
  The teams working in separate detectors are supposed to work independently (so that they do not bias each other)
lutusp 12y ago
Yes, that resembles a now-famous suggestion by a Nobel Prizewinner with respect to psychological research: http://www.nature.com/news/nobel-laureate-challenges-psychologists-to-clean-up-their-act-1.11535 http://www.nature.com/news/nobel-laureate-challenges-psychol... Title: "Nobel laureate challenges psychologists to clean up their act" Quote: "For all these reasons, right or wrong, your field is now the poster child for doubts about the integrity of psychological research. Your problem is not with the few people who have actively challenged the validity of some priming results. It is with the much larger population of colleagues who in the past accepted your surprising results as facts when they were published. ... My reason for writing this letter is that I see a train wreck looming."