10 ms·

This would be a huge leap forward. I can't blame researchers for tailoring their research to help their own careers but this should help prevent that. A null fi

by xupybd 8y ago

This would be a huge leap forward. I can't blame researchers for tailoring their research to help their own careers but this should help prevent that. A null finding might not be great for the researcher but it's a win for science.

hn_throwaway_99 8y ago
Agreed - this should help "level the playing field" so scientists can at least be somewhat more evaluated on the quality of their hypotheses and scientific protocols as opposed to the (largely governed by luck) outcome of their studies.
- forapurpose 8y ago
  > the (largely governed by luck) outcome of their studies I've never heard that. Is there some evidence that it's largely luck? Does the theory imply that Einstein and Newton were extremely lucky?
  - mettamage 8y ago
    Even if it doesn't, there's a whole world outside of theoretical physics.
    - forapurpose 8y ago
      So Darwin was very lucky? The questions remain for any field: Is there evidence, and is the implication that the leading scientists are consider very lucky?
      - drb91 8y ago
        Nah, I think you’re missing the point that you can google “bullshit psych studies” and get pages of results. This isn’t an indictment of science, this is an indictment of journals and academia today. Had this registration been in effect 300 years ago, Newton/Darwin would still be fine.
      - pjc50 8y ago
        Yes, he was the lucky person chosen to be the field geologist on the Beagle
      - goto11 8y ago
        Darwin was lucky that Wallace came to the same theory independently but contacted Darwin for feedback. This forced Darwin to publish his own results. If Wallace hadn't contacted Darwin before publishing, Darwin would probably be an unknown today.
  - drb91 8y ago
    They have the advantage of being in fairly trivially replicatable fields, so we can say with confidence they found something true. As to luck, who can say?
  - xupybd 8y ago
    Not the best examples. They both came up with analytical models and tested them. They were also amazing outliers. In that that achieved so much. For your everyday empirical research there is a lot of luck. You have to disprove a lot of possibilities to get to the true data. And as we only reward the positive findings your lucky if you’re test that one truth. If you’re sifting through drugs to find there possible other uses you’re going to have a lot of null results. But those are good they’re still extra knowledge.
  - buboard 8y ago
    You re almost creating a straw man. There have probably been scientists as intelligent or more intelligent than einstein, but worked on fields that proved fruitless, or didn't manage to finish their work within a lifetime. There is a large element of luck in any research, which has to align with capability. The matter at hand is how to keep those promising people in a science career and not the just the ones that publish a lot. Publishing negative but honest results is a step. Though i hope it doesn't end up becoming gamed as well.
    - whatshisface 8y ago
      Einstien did his Nobel prize winning work (the photoelectric effect) very early in life, and then went on to continually make big successes. If he had started out spinning his wheels he would have had plenty of time to switch.
      - buboard 8y ago
        that doesnt mean he was immune to unsuccessful investigations, in which he spent a large part of his later life https://en.wikipedia.org/wiki/Einstein%27s_unsuccessful_investigations https://en.wikipedia.org/wiki/Einstein%27s_unsuccessful_inve...
  - hn_throwaway_99 8y ago
    I'm not 100% sure about Newton but Einstein didn't do any empirical research, which is the topic of this discussion.
wallace_f 8y ago
>I can't blame researchers for tailoring their research to help their own careers What about a lowly restaurant-owner who skimps on food quality or sanitary costs to "help their career?" A psychopath or "can't blame them?" Erroneous research can also have human costs. White collar and blue collar workers should both be scrutinized equally. I'm not sure where this acceptance of moral depravity among more-privileged peoples comes from, but I really don't like it.
- jfz 8y ago
  There is a massive difference between doing something that will clearly hurt someone and choosing to do only work that benefits you. Researchers are currently under no obligation to write up their null findings, and it would be hard to get null findings in prestigious journals. This should be fixed at an institutional level, like the pre-registration of studies in the article, not as a mandate to each individual researcher to write up and publish every study.
  - wallace_f 8y ago
    >There is a massive difference between doing something that will clearly hurt someone and choosing to do only work that benefits you. This is the most-downvoted comment I've ever made on HN, and I think your explanation is in summary what people disagree with. However, the problems of bias and integrity in scientific research can and do have costs in terms of harm to human life. It's just that the connection between just following incentives and bad scientific research is much more abstracted, and therefore is not clearly intentional negligence, as the case with something like food safety.
    - btrettel 8y ago
      There's a great recent blog post on the basic point you're making: http://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-incentives-its-you/ http://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-inc...
      - 8y ago
deleted 8y ago
[deleted]
cabalamat 8y ago
> A null finding might not be great for the researcher The fix for that would be to change science funding so a null finding doesn't hurt the researcher's career.
- dmichulke 8y ago
  It already doesn't, null findings are simply not published. Unless of course you want to reward null findings which might, depending on the circumstances, open another can of worms.
  - thiagotomei 8y ago
    At least in my field (high energy physics in the LHC), null findings are routinely published. Everybody who is searching for physics beyond the standard model publishes their null findings left and right.
    - whatshisface 8y ago
      Above, a commenter makes a distinction between negative and null result, saying that null results are those without any statistically significant conclusion (neither negative nor positive). Those don't get published, the BSM papers usually rule out parts of parameter space which makes them negative results.
- taurath 8y ago
  The interests of humanity and the interests of the organizations that have the capital to fund science projects are not frequently aligned.
  - justinjlynn 8y ago
    This must change.
  - 08-15 8y ago
    Exactly! Stop government funded research! Today, most science is funded by government agencies, like the NIH. This means, bureaucrats who know nothing about science allocate other people's money to research they don't care about. Since they don't understand the research, they need objective measures of "research quality", and they picked number of publications and impact factor of the journals as measures. What you measure becomes your objective, and we got "publish or perish" and all sorts of scientific misconduct. Science needs a free market, too.
hannob 8y ago
> I can't blame researchers for tailoring their research to help their own careers but this should help prevent that. It regularly astonishes me how easily people are willing to accept scientific malpractice with such excuses. While I agree that mechanisms like registered reports are the way to fix these things at the core, I do think that these mentalities ("it's just the system, I can't do anything but cheat with my scientific publications, my career!") are a big part of the problem. Good read on the issue: http://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-incentives-its-you/ http://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-inc...
- deleted 8y ago
  [deleted]
- Devils_Avocado 8y ago
  So, do 'scientists' not have dopamine pathways and an innate biological drive to maximise their personal longevity & reproductive success by securing control of as many resources as possible for as little risk as possible? "It's not the system; we only consistently get the same negative result that's different from what we want & expect because we have the wrong people staffing it. Don't change the status quo, change human neurological architecture".
- gaius 8y ago
  It regularly astonishes me how easily people are willing to accept scientific malpractice with such excuses. Likewise. The BBC reports that 2/3 of scientific research cannot be reproduced. Let’s be honest about what that means: 2/3 of scientists are fraudulent or incompetent. https://www.bbc.com/news/science-environment-39054778 https://www.bbc.com/news/science-environment-39054778
  - lancebeet 8y ago
    If we want to give people the benefit of the doubt, it could also mean that scientific research is difficult and even competent individuals often make mistakes.
  - SubiculumCode 8y ago
PeterisP 8y ago
One thing that this achieves is that it makes null findings publishable. At least in my field, I pretty much have to discard null results because no reasonable venue will publish them; it's not worth the time to write them up if they'll be rejected anyway - at best, no-improvement results can get a mention, a line in a table, or possibly a paragraph in a published paper about something that actually does make a difference.

by xupybd 8y ago

hn_throwaway_99 8y ago

Agreed - this should help "level the playing field" so scientists can at least be somewhat more evaluated on the quality of their hypotheses and scientific protocols as opposed to the (largely governed by luck) outcome of their studies.

forapurpose 8y ago
> the (largely governed by luck) outcome of their studies I've never heard that. Is there some evidence that it's largely luck? Does the theory imply that Einstein and Newton were extremely lucky?
- mettamage 8y ago
  Even if it doesn't, there's a whole world outside of theoretical physics.
  - forapurpose 8y ago
    So Darwin was very lucky? The questions remain for any field: Is there evidence, and is the implication that the leading scientists are consider very lucky?
    - drb91 8y ago
      Nah, I think you’re missing the point that you can google “bullshit psych studies” and get pages of results. This isn’t an indictment of science, this is an indictment of journals and academia today. Had this registration been in effect 300 years ago, Newton/Darwin would still be fine.
    - pjc50 8y ago
      Yes, he was the lucky person chosen to be the field geologist on the Beagle
    - goto11 8y ago
      Darwin was lucky that Wallace came to the same theory independently but contacted Darwin for feedback. This forced Darwin to publish his own results. If Wallace hadn't contacted Darwin before publishing, Darwin would probably be an unknown today.
- drb91 8y ago
  They have the advantage of being in fairly trivially replicatable fields, so we can say with confidence they found something true. As to luck, who can say?
- xupybd 8y ago
  Not the best examples. They both came up with analytical models and tested them. They were also amazing outliers. In that that achieved so much. For your everyday empirical research there is a lot of luck. You have to disprove a lot of possibilities to get to the true data. And as we only reward the positive findings your lucky if you’re test that one truth. If you’re sifting through drugs to find there possible other uses you’re going to have a lot of null results. But those are good they’re still extra knowledge.
- buboard 8y ago
  You re almost creating a straw man. There have probably been scientists as intelligent or more intelligent than einstein, but worked on fields that proved fruitless, or didn't manage to finish their work within a lifetime. There is a large element of luck in any research, which has to align with capability. The matter at hand is how to keep those promising people in a science career and not the just the ones that publish a lot. Publishing negative but honest results is a step. Though i hope it doesn't end up becoming gamed as well.
  - whatshisface 8y ago
    Einstien did his Nobel prize winning work (the photoelectric effect) very early in life, and then went on to continually make big successes. If he had started out spinning his wheels he would have had plenty of time to switch.
    - buboard 8y ago
      that doesnt mean he was immune to unsuccessful investigations, in which he spent a large part of his later life https://en.wikipedia.org/wiki/Einstein%27s_unsuccessful_investigations https://en.wikipedia.org/wiki/Einstein%27s_unsuccessful_inve...
- hn_throwaway_99 8y ago
  I'm not 100% sure about Newton but Einstein didn't do any empirical research, which is the topic of this discussion.

wallace_f 8y ago

>I can't blame researchers for tailoring their research to help their own careers What about a lowly restaurant-owner who skimps on food quality or sanitary costs to "help their career?" A psychopath or "can't blame them?" Erroneous research can also have human costs. White collar and blue collar workers should both be scrutinized equally. I'm not sure where this acceptance of moral depravity among more-privileged peoples comes from, but I really don't like it.

jfz 8y ago
There is a massive difference between doing something that will clearly hurt someone and choosing to do only work that benefits you. Researchers are currently under no obligation to write up their null findings, and it would be hard to get null findings in prestigious journals. This should be fixed at an institutional level, like the pre-registration of studies in the article, not as a mandate to each individual researcher to write up and publish every study.
- wallace_f 8y ago
  >There is a massive difference between doing something that will clearly hurt someone and choosing to do only work that benefits you. This is the most-downvoted comment I've ever made on HN, and I think your explanation is in summary what people disagree with. However, the problems of bias and integrity in scientific research can and do have costs in terms of harm to human life. It's just that the connection between just following incentives and bad scientific research is much more abstracted, and therefore is not clearly intentional negligence, as the case with something like food safety.
  - btrettel 8y ago
    There's a great recent blog post on the basic point you're making: http://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-incentives-its-you/ http://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-inc...
    - 8y ago

deleted 8y ago

[deleted]

cabalamat 8y ago

> A null finding might not be great for the researcher The fix for that would be to change science funding so a null finding doesn't hurt the researcher's career.

dmichulke 8y ago
It already doesn't, null findings are simply not published. Unless of course you want to reward null findings which might, depending on the circumstances, open another can of worms.
- thiagotomei 8y ago
  At least in my field (high energy physics in the LHC), null findings are routinely published. Everybody who is searching for physics beyond the standard model publishes their null findings left and right.
  - whatshisface 8y ago
    Above, a commenter makes a distinction between negative and null result, saying that null results are those without any statistically significant conclusion (neither negative nor positive). Those don't get published, the BSM papers usually rule out parts of parameter space which makes them negative results.
taurath 8y ago
The interests of humanity and the interests of the organizations that have the capital to fund science projects are not frequently aligned.
- justinjlynn 8y ago
  This must change.
- 08-15 8y ago
  Exactly! Stop government funded research! Today, most science is funded by government agencies, like the NIH. This means, bureaucrats who know nothing about science allocate other people's money to research they don't care about. Since they don't understand the research, they need objective measures of "research quality", and they picked number of publications and impact factor of the journals as measures. What you measure becomes your objective, and we got "publish or perish" and all sorts of scientific misconduct. Science needs a free market, too.

hannob 8y ago

> I can't blame researchers for tailoring their research to help their own careers but this should help prevent that. It regularly astonishes me how easily people are willing to accept scientific malpractice with such excuses. While I agree that mechanisms like registered reports are the way to fix these things at the core, I do think that these mentalities ("it's just the system, I can't do anything but cheat with my scientific publications, my career!") are a big part of the problem. Good read on the issue: http://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-incentives-its-you/ http://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-inc...

deleted 8y ago
[deleted]
Devils_Avocado 8y ago
So, do 'scientists' not have dopamine pathways and an innate biological drive to maximise their personal longevity & reproductive success by securing control of as many resources as possible for as little risk as possible? "It's not the system; we only consistently get the same negative result that's different from what we want & expect because we have the wrong people staffing it. Don't change the status quo, change human neurological architecture".
gaius 8y ago
It regularly astonishes me how easily people are willing to accept scientific malpractice with such excuses. Likewise. The BBC reports that 2/3 of scientific research cannot be reproduced. Let’s be honest about what that means: 2/3 of scientists are fraudulent or incompetent. https://www.bbc.com/news/science-environment-39054778 https://www.bbc.com/news/science-environment-39054778
- lancebeet 8y ago
  If we want to give people the benefit of the doubt, it could also mean that scientific research is difficult and even competent individuals often make mistakes.
- SubiculumCode 8y ago

PeterisP 8y ago

One thing that this achieves is that it makes null findings publishable. At least in my field, I pretty much have to discard null results because no reasonable venue will publish them; it's not worth the time to write them up if they'll be rejected anyway - at best, no-improvement results can get a mention, a line in a table, or possibly a paragraph in a published paper about something that actually does make a difference.