15 ms·

Millions of animals missing from scientific studies

129 points by spekcular 6y ago

bootcampwhere 6y ago
At it's very base this makes no sense. It comes from a "moral" sentiment that for some reason animals should be rationed, as if life is limited. All science is cursed by unnecessary repetition but there's nothing special about animal studies that makes this problem any worse.
SubiculumCode 6y ago
"The most common reasons they gave were that the studies didn’t achieve statistical significance, a controversial but commonly used threshold for publication; that the data were part of a pilot project; and that there were technical issues with the animal models. But none of these is a valid excuse to not publish your findings in the scientific record, says study co-author Kimberley Wever, a metascientist at Radboud University Medical Center. “All animal studies should be published, and all studies are valuable for the research community.” I hard disagree that pilot projects and results from data with technical issues should be published. There is a real cost to write-ups (time). Pilot projects usually are done just to prove feasibility of a protocol, and data retrieved from experiments with technical issues is likely noise. This could include inaccurate measurements, cross-contamination, etc. Taking the time to write and publish all the junk is, frankly prohibitive. I do think null-results should be publishable, however. The title is a bit of click bait also: The animals are not missing. Make a request of the relevant Institutional Review Board. Are any of you voting my comment down actually scientists?
- georgebarnett 6y ago
  If you have time to do the work to get a pilot project off the ground then you absolutely have time to do a short write up on how it failed.
  - SubiculumCode 6y ago
    Pilot projects are usually a small sample that demonstrated that a protocol can be achieved, not a meaningful set of data to test a hypothesis. So no, it is probably a waste of time.
    - georgebarnett 6y ago
      Whether or not a protocol can be achieved using a certain method is a valuable outcome. Write it up so your peers know what didn’t work.
      - SubiculumCode 6y ago
        Yes. The details of the used protocols are supposed to be included (and usually are) in the primary publication of the research.
        DangitBobby 6y ago
        But not the results of the pilot unless it was promising?
        SubiculumCode 6y ago
        pilot data is not what you think it is. Pilot data is about finding parameters, testing procedures. It is neither a consistent nor a full replica of an actual experiment.
      - ckemere 6y ago
        I think the thing that non-experimentalists don't realize is how many experiments fail for reasons that are not understood. For example - "the neurons in the part of the brain failed to express the transgene I wanted to use to study them." To actually write a paper about that null result requires substantially more experiments to form hypotheses and test them. Moreover, my lab is far from being an expert in the esoteric ways that this can happen, so when I try to submit the paper, the experts ask for even 10x more experiments. So I've gone farther and farther from the questions that are "important" (reflect my interests, expertise, and funding). But this explains the value of scientific meetings and poster sessions where I can just tell this random tidbit to interested people without the burden of peer review. An interesting recent development I've observed once was a tweet describing the fact that a dye (Texas red) labels brain vasculature even when injected subcutaneously. This random fact is not in the literature but is quite helpful from a procedural perspective. I think that science twitter has a potentially super valuable role to play in reporting unexpected or otherwise difficult to publish findings.
- 1_over_n 6y ago
  Actual scientist. I believe Record keeping should be kept, i'm not saying it justifies a full publication but these records should be kept. If you go on to publish then these records should be acknowledged in the publication. It can be included in appendices / extra information even if it's not in the main article.
crazygringo 6y ago
> But none of these is a valid excuse to not publish your findings in the scientific record Are journals even going to publish most no-result studies? (I.e. a result of no significant correlation found.) Journals pick and choose the best articles to peer-review and publish out of all submissions. It feels like there ought to be some other system for tracking no-result and unpublished studies, something easy and low-friction -- not the trouble of an entire paper, but maybe something more like the length of an abstract? Then if you think about running a study, you can run a search to see if others have already done something similar and found nothing... and reach out to them personally if you need more info. Is there nothing like this?
- SubiculumCode 6y ago
  Why not publish them in an arXiv?
  - dddddaviddddd 6y ago
    It can be lots of work to present the data and describe it, graduate students may give priority to incentived 'high-impact' publication instead.
    - harimau777 6y ago
      Unless you can get a PhD or funding from failed results I don't see graduate students giving any priority to publishing them.
      - refurb 6y ago
        Exactly. I could see a PI saying "well, you'll finish your PhD in 5 years if you're lucky and most of your experiments are successful. In the meantime, could I get you to spend 6 months writing up all these unsuccessful experiments? You'll get zero credit for it and it will delay your PhD. Thanks!"
- question000 6y ago
mensetmanusman 6y ago
Based on the %success chance of typical research, I’m surprised such a high percentage of the animals end up in publications. That probably means the review board is doing a pretty good job, because they must be denying studies they believe wont answer any interesting conclusive question?
lvs 6y ago
This is really a non-issue. There are numerous preliminary experiments and trivially failed experiments done in the course of figuring out a research direction and optimizing protocols. This work rarely makes it into publications in any field, and it would muddy the science to do so. I'm not a fan of most animal work in general, but the idea that there are "missing" animals is just a misapprehension about the process.
- lamename 6y ago
  This is the most spot on comment. This is field- and technique- specific, but animals approved in a grant and animals that end up in the paper are, in my experience, extremely unlikely to be the same number. Even in an experienced lab with well-oiled technical expertise, there's nearly always a fraction of animals that go to pilot experiments (i.e. didnt work), have some confound, or just didn't work. And by didn't work I mean didn't even get far enough to count as negative data, e.g. if you're trying something very technically demanding or new. Jumping to the conclusion that negative results aren't being reported (which is indeed its own big problem) from a difference in animal number suggests lack of understanding of the process, but perhaps formal accounting of research animals differs by country.
abeppu 6y ago
The general principle seems to be that any effort which doesn't yield a result (it failed for methodological reasons, there was no effect, there was not a significant effect, they were developing a protocol, etc) should be published so that other researchers can learn from the effort. But that principle doesn't sound specifically related to animals (or mammals, as this article mentions mice, rats and rabbits, but not fruit flies etc). Should the same apply bacterial or fungal cultures? What about outside of biology? Should someone studying psychology of personality who gets a non-result from a survey of psych department undergrads publish it? Should a materials scientist who builds a bad battery describe it in detail? Should I create a PR for every branch which I don't want to merge? At some point, any new endeavor can only start by choosing to not spend 5 years reviewing all the one-off failures.
- MLtoTheRescue 6y ago
  Notate for ML, dump in machine, ping when incongruities occur. Repeat.
projektfu 6y ago
In the US, almost all animals should be accounted for by the IACUC at the research organization. In practice that includes rats, mice and fish even though the animal welfare act excludes them, because the Public Health Service does not. It does not surprise me that they do not always appear in papers. How often does an experiment end without any result at all? Pretty often.
pvaldes 6y ago
> The researchers believe the pattern seen on a university in Holland could be similar at institutions around the world Suddenly a wildly extrapolated clickbaitykachu appears Non-sequitur, I choose you! Why to say millions when we could say thousands?
ggm 6y ago
Many scientific studies fail to publish the number of animals in total, used in their statistics. This would represent millions of animal-subjects, over time. Many scientific studies obscure how many input subjects are rejected from final assessment. This would represent millions of experimental subjects, over time.
jonnycomputer 6y ago
Actually, it may not be necessary for journals to publish null results. The important things is that the data be made available. Funding agencies like the NIH are making this mandatory. I think the NIH is, unfortunately, taking the wrong approach in implementing its data archives; it places an interest in data harmonization above common sense and data intelligibility, in my opinion. Still, I think its a move in the right direction.
dj_mc_merlin 6y ago
> The researchers also surveyed the scientists involved to find out why so many animals were missing. The most common reasons they gave were that the studies didn’t achieve statistical significance, a controversial but commonly used threshold for publication I do not come from academia so I may be mistaken, but why would you publish a non statistically significant result? There seems to me there's a very distinct difference between "this is most likely true" and "this might be true, we can't say yet". The first is useful, the second adds to the chatter needlessly. Some may say that it promotes further research, but what if a false weakly proven result misleads instead?
- GirkovArpa 6y ago
  Imagine if a drug company ran a study 100 times before finding a "significant" result that supported the claims they wanted to make about their product. Without the other 99 studies, the single study is highly misleading.
  - pixl97 6y ago
    Depends what methodologies were changed between studies. If people with a particular gene were effected, but said expression was very rare then you might have a huge number of studies that show no effect, but were highly effective in that one case.
    - rebuilder 6y ago
      Exactly, so if you only published the one study where this rare gene was present in a participant, you'd get a wildly distorted picture of the general effectiveness of your intervention.
      - pixl97 6y ago
        The issue comes in, especially in future medicines that only target particular genes, that you only care about the effectiveness in that limited case.
tgbugs 6y ago
Full disclosure, Anita is a lab mate. One of the things that is latent in this article is that in the US you are supposed to have done a power analysis in order to justify the number of animals you are using in your study. Almost no one does this, and it is not surprising -- if you are doing cutting edge research you are in unknown unknown territory and any power analysis is likely to be no better than a guess. In a sense it is farcical that exploratory research needs to pretend that it is always successful. Countless animal experiments are uninterpretable simply because they were poorly executed by first year graduate students. There is no other way to learn. Write them off as animals used for training if we want full accounting, but researchers' time is scarce enough as it is so asking them to publish uninterpretable results is a non-starter. On the other hand I think it is important to publish as many results as possible to avoid file draw effects etc. Data publishing might be one way around this, but at the moment few labs anywhere in any field have the know-how to publish raw data for negative results, even if it is just sticking files in a git repo and getting a DOI from zenodo.
- 1_over_n 6y ago
  "Countless animal experiments are uninterpretable simply because they were poorly executed by first year graduate students. There is no other way to learn. Write them off as animals used for training if we want full accounting, but researchers' time is scarce enough as it is so asking them to publish uninterpretable results is a non-starter." > I'm sorry but this simply isn't true. During my masters i did in vivo surgery on 30 rodents and tested how many the procedure was succesful in (10). I had to record how many the procedure was sucessful in and write this up in my methods. I'm not sure how you see keeping a record of this as something that takes a significant length of time that you wouldn't bother to record it. In fact i'm pretty sure (at least for the UK) that keeping records like this is essential under the home office rules for animal safety in scientific procedures. These rules are there to help with the 3 R's (reduce, replace, refinement). This guidance aims to cut down on the use of animals in research or at least ensure quality experiments are being ran. If you don't measure - how can you improve? https://www.nc3rs.org.uk/the-3rs https://www.nc3rs.org.uk/the-3rs
  - lamename 6y ago
cycomanic 6y ago
I don't believe publication of non results or negative results would be helpful at least not published in the traditional sense. Apart from the work for writing this up it is already hard enough to follow the amount of publications coming out in most fields at the moment. Non result publications would just not be read and would make finding the right literature so much more difficult. What is really needed is publication of the data. The big problem with that is that there are no good systems for data publication that don't require significant extra work. Ideally the data would be made available directly from some electronic labbook system, with the relevant metadata attached to it. At the moment all systems are miles away from that (even if we don't consider how bad record keeping is in many research labs). If funding agencies would really want to do something about open data they would significantly invest into a good data system and make data entry mandatory and in particular not just at the end, but during the study (private, but then made available at the selection of the PI or after some time)
- tsimionescu 6y ago
  Wouldn't this open the door to even more p-hacking than today - unscrupulous researchers taking others' data and finding some kind of signal in it? My understanding was that the gold standard for research is to come up with a hypothesis, design an experiment to test that hypothesis, collecting data in such a way that you have statistical independence between the variables you wish to measure, and finally analyze the data and conclude check if you have confirmed your hypothesis or if it remains unconfirmed. But data collected correctly to test one hypothesis is not necessarily good for testing other hypotheses - e.g. perhaps it was irrelevant for your research if all the animals were female and might have had birth defects, but that doesn't mean your data is useful for someone studying the whole population. This would mean that not only do you have to publish the raw data, but also describe your data collection methods, your assumptions, things you didn't check etc. How far are you from publishing your whole paper at that point?
  - cycomanic 6y ago
    >Wouldn't this open the door to even more p-hacking than today - unscrupulous researchers taking others' data and finding some kind of signal in it? I would argue that the more data is out there, the less easy it is to "p-hack", i.e. it's much more difficult to find some weird correlation if you have a huge population. Also I would not call researchers who use others data "unscrupulous". In fact that is been done in meta studies already and is exactly the point of the exercise. Increase the amount of data published so if some "unscrupulous" or "erroneous" researcher publishes a study with some some spurious correlation, we can look at a lot more data to see if that also exists there. >My understanding was that the gold standard for research is to come up with a hypothesis, design an experiment to test that hypothesis, collecting data in such a way that you have statistical independence between the variables you wish to measure, and finally analyze the data and conclude check if you have confirmed your hypothesis or if it remains unconfirmed. >But data collected correctly to test one hypothesis is not necessarily good for testing other hypotheses - e.g. perhaps it was irrelevant for your research if all the animals were female and might have had birth defects, but that doesn't mean your data is useful for someone studying the whole population. It is true that sometimes data collected for one purpose is not necessarily good for another purpose, however very often it is and many discoveries were made by looking for something new in data collected for a completely different purpose. >This would mean that not only do you have to publish the raw data, but also describe your data collection methods, your assumptions, things you didn't check etc. How far are you from publishing your whole paper at that point? Good reproducible science should do that anyway. You should always keep a labbook that clearly documents what you are doing so that someone else (or your future self), can reproduce the results. So in the requirement I'm asking for is forcing people to do good science. I can tell from experience this documentation is still very far away from a whole paper.

129 points by spekcular 6y ago

At it's very base this makes no sense. It comes from a "moral" sentiment that for some reason animals should be rationed, as if life is limited. All science is cursed by unnecessary repetition but there's nothing special about animal studies that makes this problem any worse.

SubiculumCode 6y ago

"The most common reasons they gave were that the studies didn’t achieve statistical significance, a controversial but commonly used threshold for publication; that the data were part of a pilot project; and that there were technical issues with the animal models. But none of these is a valid excuse to not publish your findings in the scientific record, says study co-author Kimberley Wever, a metascientist at Radboud University Medical Center. “All animal studies should be published, and all studies are valuable for the research community.” I hard disagree that pilot projects and results from data with technical issues should be published. There is a real cost to write-ups (time). Pilot projects usually are done just to prove feasibility of a protocol, and data retrieved from experiments with technical issues is likely noise. This could include inaccurate measurements, cross-contamination, etc. Taking the time to write and publish all the junk is, frankly prohibitive. I do think null-results should be publishable, however. The title is a bit of click bait also: The animals are not missing. Make a request of the relevant Institutional Review Board. Are any of you voting my comment down actually scientists?

georgebarnett 6y ago
If you have time to do the work to get a pilot project off the ground then you absolutely have time to do a short write up on how it failed.
- SubiculumCode 6y ago
  Pilot projects are usually a small sample that demonstrated that a protocol can be achieved, not a meaningful set of data to test a hypothesis. So no, it is probably a waste of time.
  - georgebarnett 6y ago
    Whether or not a protocol can be achieved using a certain method is a valuable outcome. Write it up so your peers know what didn’t work.
    - SubiculumCode 6y ago
      Yes. The details of the used protocols are supposed to be included (and usually are) in the primary publication of the research.
      - DangitBobby 6y ago
        But not the results of the pilot unless it was promising?
        SubiculumCode 6y ago
        pilot data is not what you think it is. Pilot data is about finding parameters, testing procedures. It is neither a consistent nor a full replica of an actual experiment.
    - ckemere 6y ago
      I think the thing that non-experimentalists don't realize is how many experiments fail for reasons that are not understood. For example - "the neurons in the part of the brain failed to express the transgene I wanted to use to study them." To actually write a paper about that null result requires substantially more experiments to form hypotheses and test them. Moreover, my lab is far from being an expert in the esoteric ways that this can happen, so when I try to submit the paper, the experts ask for even 10x more experiments. So I've gone farther and farther from the questions that are "important" (reflect my interests, expertise, and funding). But this explains the value of scientific meetings and poster sessions where I can just tell this random tidbit to interested people without the burden of peer review. An interesting recent development I've observed once was a tweet describing the fact that a dye (Texas red) labels brain vasculature even when injected subcutaneously. This random fact is not in the literature but is quite helpful from a procedural perspective. I think that science twitter has a potentially super valuable role to play in reporting unexpected or otherwise difficult to publish findings.
1_over_n 6y ago
Actual scientist. I believe Record keeping should be kept, i'm not saying it justifies a full publication but these records should be kept. If you go on to publish then these records should be acknowledged in the publication. It can be included in appendices / extra information even if it's not in the main article.

crazygringo 6y ago

> But none of these is a valid excuse to not publish your findings in the scientific record Are journals even going to publish most no-result studies? (I.e. a result of no significant correlation found.) Journals pick and choose the best articles to peer-review and publish out of all submissions. It feels like there ought to be some other system for tracking no-result and unpublished studies, something easy and low-friction -- not the trouble of an entire paper, but maybe something more like the length of an abstract? Then if you think about running a study, you can run a search to see if others have already done something similar and found nothing... and reach out to them personally if you need more info. Is there nothing like this?

SubiculumCode 6y ago
Why not publish them in an arXiv?
- dddddaviddddd 6y ago
  It can be lots of work to present the data and describe it, graduate students may give priority to incentived 'high-impact' publication instead.
  - harimau777 6y ago
    Unless you can get a PhD or funding from failed results I don't see graduate students giving any priority to publishing them.
    - refurb 6y ago
      Exactly. I could see a PI saying "well, you'll finish your PhD in 5 years if you're lucky and most of your experiments are successful. In the meantime, could I get you to spend 6 months writing up all these unsuccessful experiments? You'll get zero credit for it and it will delay your PhD. Thanks!"
question000 6y ago

mensetmanusman 6y ago

Based on the %success chance of typical research, I’m surprised such a high percentage of the animals end up in publications. That probably means the review board is doing a pretty good job, because they must be denying studies they believe wont answer any interesting conclusive question?

lvs 6y ago

This is really a non-issue. There are numerous preliminary experiments and trivially failed experiments done in the course of figuring out a research direction and optimizing protocols. This work rarely makes it into publications in any field, and it would muddy the science to do so. I'm not a fan of most animal work in general, but the idea that there are "missing" animals is just a misapprehension about the process.

lamename 6y ago
This is the most spot on comment. This is field- and technique- specific, but animals approved in a grant and animals that end up in the paper are, in my experience, extremely unlikely to be the same number. Even in an experienced lab with well-oiled technical expertise, there's nearly always a fraction of animals that go to pilot experiments (i.e. didnt work), have some confound, or just didn't work. And by didn't work I mean didn't even get far enough to count as negative data, e.g. if you're trying something very technically demanding or new. Jumping to the conclusion that negative results aren't being reported (which is indeed its own big problem) from a difference in animal number suggests lack of understanding of the process, but perhaps formal accounting of research animals differs by country.

abeppu 6y ago

The general principle seems to be that any effort which doesn't yield a result (it failed for methodological reasons, there was no effect, there was not a significant effect, they were developing a protocol, etc) should be published so that other researchers can learn from the effort. But that principle doesn't sound specifically related to animals (or mammals, as this article mentions mice, rats and rabbits, but not fruit flies etc). Should the same apply bacterial or fungal cultures? What about outside of biology? Should someone studying psychology of personality who gets a non-result from a survey of psych department undergrads publish it? Should a materials scientist who builds a bad battery describe it in detail? Should I create a PR for every branch which I don't want to merge? At some point, any new endeavor can only start by choosing to not spend 5 years reviewing all the one-off failures.

MLtoTheRescue 6y ago
Notate for ML, dump in machine, ping when incongruities occur. Repeat.

projektfu 6y ago

In the US, almost all animals should be accounted for by the IACUC at the research organization. In practice that includes rats, mice and fish even though the animal welfare act excludes them, because the Public Health Service does not. It does not surprise me that they do not always appear in papers. How often does an experiment end without any result at all? Pretty often.

pvaldes 6y ago

> The researchers believe the pattern seen on a university in Holland could be similar at institutions around the world Suddenly a wildly extrapolated clickbaitykachu appears Non-sequitur, I choose you! Why to say millions when we could say thousands?

ggm 6y ago

Many scientific studies fail to publish the number of animals in total, used in their statistics. This would represent millions of animal-subjects, over time. Many scientific studies obscure how many input subjects are rejected from final assessment. This would represent millions of experimental subjects, over time.

jonnycomputer 6y ago

Actually, it may not be necessary for journals to publish null results. The important things is that the data be made available. Funding agencies like the NIH are making this mandatory. I think the NIH is, unfortunately, taking the wrong approach in implementing its data archives; it places an interest in data harmonization above common sense and data intelligibility, in my opinion. Still, I think its a move in the right direction.

dj_mc_merlin 6y ago

> The researchers also surveyed the scientists involved to find out why so many animals were missing. The most common reasons they gave were that the studies didn’t achieve statistical significance, a controversial but commonly used threshold for publication I do not come from academia so I may be mistaken, but why would you publish a non statistically significant result? There seems to me there's a very distinct difference between "this is most likely true" and "this might be true, we can't say yet". The first is useful, the second adds to the chatter needlessly. Some may say that it promotes further research, but what if a false weakly proven result misleads instead?

GirkovArpa 6y ago
Imagine if a drug company ran a study 100 times before finding a "significant" result that supported the claims they wanted to make about their product. Without the other 99 studies, the single study is highly misleading.
- pixl97 6y ago
  Depends what methodologies were changed between studies. If people with a particular gene were effected, but said expression was very rare then you might have a huge number of studies that show no effect, but were highly effective in that one case.
  - rebuilder 6y ago
    Exactly, so if you only published the one study where this rare gene was present in a participant, you'd get a wildly distorted picture of the general effectiveness of your intervention.
    - pixl97 6y ago
      The issue comes in, especially in future medicines that only target particular genes, that you only care about the effectiveness in that limited case.

tgbugs 6y ago

Full disclosure, Anita is a lab mate. One of the things that is latent in this article is that in the US you are supposed to have done a power analysis in order to justify the number of animals you are using in your study. Almost no one does this, and it is not surprising -- if you are doing cutting edge research you are in unknown unknown territory and any power analysis is likely to be no better than a guess. In a sense it is farcical that exploratory research needs to pretend that it is always successful. Countless animal experiments are uninterpretable simply because they were poorly executed by first year graduate students. There is no other way to learn. Write them off as animals used for training if we want full accounting, but researchers' time is scarce enough as it is so asking them to publish uninterpretable results is a non-starter. On the other hand I think it is important to publish as many results as possible to avoid file draw effects etc. Data publishing might be one way around this, but at the moment few labs anywhere in any field have the know-how to publish raw data for negative results, even if it is just sticking files in a git repo and getting a DOI from zenodo.

1_over_n 6y ago
"Countless animal experiments are uninterpretable simply because they were poorly executed by first year graduate students. There is no other way to learn. Write them off as animals used for training if we want full accounting, but researchers' time is scarce enough as it is so asking them to publish uninterpretable results is a non-starter." > I'm sorry but this simply isn't true. During my masters i did in vivo surgery on 30 rodents and tested how many the procedure was succesful in (10). I had to record how many the procedure was sucessful in and write this up in my methods. I'm not sure how you see keeping a record of this as something that takes a significant length of time that you wouldn't bother to record it. In fact i'm pretty sure (at least for the UK) that keeping records like this is essential under the home office rules for animal safety in scientific procedures. These rules are there to help with the 3 R's (reduce, replace, refinement). This guidance aims to cut down on the use of animals in research or at least ensure quality experiments are being ran. If you don't measure - how can you improve? https://www.nc3rs.org.uk/the-3rs https://www.nc3rs.org.uk/the-3rs
- lamename 6y ago

cycomanic 6y ago

I don't believe publication of non results or negative results would be helpful at least not published in the traditional sense. Apart from the work for writing this up it is already hard enough to follow the amount of publications coming out in most fields at the moment. Non result publications would just not be read and would make finding the right literature so much more difficult. What is really needed is publication of the data. The big problem with that is that there are no good systems for data publication that don't require significant extra work. Ideally the data would be made available directly from some electronic labbook system, with the relevant metadata attached to it. At the moment all systems are miles away from that (even if we don't consider how bad record keeping is in many research labs). If funding agencies would really want to do something about open data they would significantly invest into a good data system and make data entry mandatory and in particular not just at the end, but during the study (private, but then made available at the selection of the PI or after some time)

tsimionescu 6y ago
Wouldn't this open the door to even more p-hacking than today - unscrupulous researchers taking others' data and finding some kind of signal in it? My understanding was that the gold standard for research is to come up with a hypothesis, design an experiment to test that hypothesis, collecting data in such a way that you have statistical independence between the variables you wish to measure, and finally analyze the data and conclude check if you have confirmed your hypothesis or if it remains unconfirmed. But data collected correctly to test one hypothesis is not necessarily good for testing other hypotheses - e.g. perhaps it was irrelevant for your research if all the animals were female and might have had birth defects, but that doesn't mean your data is useful for someone studying the whole population. This would mean that not only do you have to publish the raw data, but also describe your data collection methods, your assumptions, things you didn't check etc. How far are you from publishing your whole paper at that point?
- cycomanic 6y ago
  >Wouldn't this open the door to even more p-hacking than today - unscrupulous researchers taking others' data and finding some kind of signal in it? I would argue that the more data is out there, the less easy it is to "p-hack", i.e. it's much more difficult to find some weird correlation if you have a huge population. Also I would not call researchers who use others data "unscrupulous". In fact that is been done in meta studies already and is exactly the point of the exercise. Increase the amount of data published so if some "unscrupulous" or "erroneous" researcher publishes a study with some some spurious correlation, we can look at a lot more data to see if that also exists there. >My understanding was that the gold standard for research is to come up with a hypothesis, design an experiment to test that hypothesis, collecting data in such a way that you have statistical independence between the variables you wish to measure, and finally analyze the data and conclude check if you have confirmed your hypothesis or if it remains unconfirmed. >But data collected correctly to test one hypothesis is not necessarily good for testing other hypotheses - e.g. perhaps it was irrelevant for your research if all the animals were female and might have had birth defects, but that doesn't mean your data is useful for someone studying the whole population. It is true that sometimes data collected for one purpose is not necessarily good for another purpose, however very often it is and many discoveries were made by looking for something new in data collected for a completely different purpose. >This would mean that not only do you have to publish the raw data, but also describe your data collection methods, your assumptions, things you didn't check etc. How far are you from publishing your whole paper at that point? Good reproducible science should do that anyway. You should always keep a labbook that clearly documents what you are doing so that someone else (or your future self), can reproduce the results. So in the requirement I'm asking for is forcing people to do good science. I can tell from experience this documentation is still very far away from a whole paper.