16 ms·

"all you have to do is read the methods section in the paper and follow the instructions." I wish science was that simple. The methods section only contains va

• JumpCrisscross 10y ago
  > The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know Maybe the format needs to change. Perhaps journals should require video, audio commentary or automated note taking for publication.
  • sndean 10y ago
    This exists (https://www.jove.com/ https://www.jove.com/), but, in my experience, even having that doesn't necessarily allow you to replicate the experiment. Though I should say that I really like JOVE. You can learn a lot.
    • IndianAstronaut 10y ago
      JOVE is really nice. My old lab posted some videos on there of how to for experiments. It is a huge step up from just reading methods and materials.
  • graeham 10y ago
    Usually, if you want the full details you look/ ask for the thesis or full report that the journal article is the condensed version of. Most journals also allow supplementary sections for more detailed methods. If its too long, people simply won't read it - already most will just read the abstract and look at figures.
  • jamessb 10y ago
    > Maybe the format needs to change. Perhaps journals should require video, audio commentary or automated note taking for publication. A 'world view' column in Nature suggested the same things last week [1]; the author described a paper of theirs [2]: > Yes, visual evidence can be faked, but a few simple safeguards should be enough to prevent that. Take a typical experiment in my field: using a tank of flowing water to expose fish to environmental perturbations and looking for shifts in behaviour. It is trivial to set up a camera, and equally simple to begin each recorded exposure with a note that details, for example, the trial number and treatment history of the organism. (Think of how film directors use clapper boards to keep records of the sequence of numerous takes.) This simple measure would make it much more difficult to fabricate data and ‘assign’ animals to desired treatment groups after the results are known. [1]: http://www.nature.com/news/science-lies-and-video-taped-experiments-1.21432 http://www.nature.com/news/science-lies-and-video-taped-expe... [2]: http://onlinelibrary.wiley.com/doi/10.1111/2041-210X.12668/full http://onlinelibrary.wiley.com/doi/10.1111/2041-210X.12668/f...
    • djsumdog 10y ago
      I thought some journals were now allowed video archives. Some of the most famous experiments, such as the Stanley Milgram studies, have excellent video documentation (and in the case of Milgram, it's been replicated all around the world .. although no it's not ethical to do so).
    • mattkrause 10y ago
      I don't think this is particularly practical. Most experiments run for years (literally) and no one is going to record or archive, let alone watch, years of footage to confirm that one paper is legit. A brief experiment showing the apparatus and the collection of a few data points might be helpful for understanding the paper, but I can't see using it to verify a non-trivial experiment.
      • hackuser 10y ago
        > no one is going to record or archive, let alone watch, years of footage Recording and storing years of footage shouldn't be a significant problem with modern tech. Nobody has to watch years of it; they can watch the parts they are interested in. They also can watch at 4x and search, as needed.
  • ovis 10y ago
    I was chatting with a colleague last summer, and asking him for more details about a method he had published that I was having trouble reproducing. After pointing me to a few papers that didn't include the relevant details, hefinally told me that "if we told everyone about how it was done, anyone could do it." While at the time I was pretty upset with him, perhaps it's the competitive nature of science and funding that also gives people a mild incentive to be secretive. In this case, simply publishing code would have resolved the questions.
    • hannob 10y ago
      > I was chatting with a colleague last summer, and asking > him for more details about a method he had published that > I was having trouble reproducing. After pointing me to a > few papers that didn't include the relevant details, > hefinally told me that "if we told everyone about how it > was done, anyone could do it." That's not science, that's bullshit. Can you please expose that? Scientists shouldn't simply get away with such malicious behavior.
      • wycx 10y ago
        That is exactly how science works. Methods sections include just enough information to get a good sense of how the experiment was done, not enough information to replicate it exactly. Some disciplines are worse than others. People could also publish plain text data as supplementary material, but why do that when you can get away with a raster image of a plot...
        • michaelmrose 10y ago
          At that point its actually not even science and shouldn't be funded
          • toss1941 10y ago
            I remember my teachers in high school making such a big deal about the scientific method on how important it was, how experiments must be reproducible to be useful, but today you barely hear it mentioned one way or another.
  • bbctol 10y ago
    The journal I work for just rolled out a new methods format; the big change is requiring all resources to be listed, with their origin (a major problem is that Chemical A or lab rat subspecies B might be very different from one company or another, even if they're theoretically identical.) What's really needed is broader standardization of all practices; requiring video wouldn't solve the issue that different labs might not know what another would think needed to be videotaped. No one is going to document and record the entire course of an experiment; that would literally be months of footage. We'd like to make sure everyone has the same understanding of procedures and techniques, but that requires communication not just through journals, but between scientists and academic institutions.
    • jonlucc 10y ago
      Additionally, not everything appears on video. We have found major differences between identical studies run in rooms set at different temperatures. Also, many of the procedures I do would essentially require a camera operator to capture all of the movement. I take a cage out of a rack, take a mouse out of the cage, weigh it, dose it, etc.
      • hackuser 10y ago
        > not everything appears on video Sure, but a lot of information does appear. Not everything appears in papers; should we stop publishing them?
        • jonlucc 10y ago
          No, but the burden of videography is a lot higher than the burden of writing a paper. In fact, a paper isn't written for every study, so in order to write a paper, you'd have to take video of every study just in case one of them is used in a paper in the future. It's a much higher burden than people want to believe. Add to that the fact that most animal facilities won't add cameras unless you force them at gunpoint because historically, videos of that sort make for targeting by protest groups.
    • hackuser 10y ago
  • SilasX 10y ago
    Or require the author to only communicate to the technicians via the same means that will be published with the paper -- i.e. Make sure anyone reading the paper has the same spec for the experiment as those who performed the original. [1] That would help prevent such "hidden specs" from entering the experiment. [1] Note: this implies that authors cannot be part of the experiment or conduct it themselves, because they can't "pass on their identity" in a research paper, as would be necessary to put readers on par with the technicians.
    • jonlucc 10y ago
      In my job, I work with an outside contract research organization. The way this works is pretty similar to what you're describing, and its a living hell. For starters, it takes 6-10 drafts back and forth to get a protocol to start with, and the resulting document is never shorter than 12 pages. Then, once they start a study, if we get any aberrant data, the question becomes "did they mess that up, or is that data real?". Of course anything we want to be sure about, we run twice, but when you have a panel of 20 compounds, you can't duplicate all of the work, so some compounds get dropped even if the data were not "real". Also, and maybe this would be different with another CRO, there are often very stressful conversations in which they are trying to avoid being blamed (because if they screwed up, we aren't supposed to pay them the full price), but we just really want to know what happened. Lastly, you can tell a lot by being hands-on with a study; there's a lot you can miss if you aren't in the room with the study. Just my 2 cents.
      • SilasX 10y ago
        Well, the burden of a policy has to be judged relative to what you're trying to accomplish. If the policy is ensuring that you achieve your ostensible goals, then that burden is justified. Based on your description, it actually sounds like it's making you do things exactly the way science is supposed to work! You quickly identify issues of "real effect or experimenter error or undocumented protocol?" -- and you prevent any ambiguous case from "infecting" the literature. Those are the same objectives modern science is currently failing at with it's "publish but never replicate" incentives. > Lastly, you can tell a lot by being hands-on with a study; there's a lot you can miss if you aren't in the room with the study. I wasn't saying that you can't be there in the lab and do that kind of experimentation, just that scientists shouldn't represent this kind of ad hoc work as the repeatable part that merits a scientific conclusion. The process should be: if you find something interesting that way, see if you can identify a repeatable, articulable procedure by which others can see the same thing, and publish that.
  • Obi_Juan_Kenobi 10y ago
    The 'unknown unknowns' will never be eliminated. Certainly it's worthwhile to try to improve methods communication, but there are limits. You really have to try it yourself before you can understand the degree of troubleshooting that's required of a good experimentalist. You could have scientists live-streaming all their work, and you'd still have the same issues you do now. Even a simple experiment, something most wouldn't blink an eye at, has dozens and dozens of variables that could influence the result. The combinatoric complexity is staggering. The reality is that you try things, find some that work, and then convince yourself that it's a real result with some further work. The methods that endure are the ones that replicate well and work robustly. Molecular biology is still built on Sanger sequencing, electrophoresis, and blotting, all in the context of good crossing and genetics, because that's what works. Some of the genomic tools are starting to get there, I'd venture to say that RNAseq is reasonably standardized and robust at this point. Interpreting genomic data is another story...
  • Fomite 10y ago
    So there's two different problems here. What you're talking about is the authors not reporting variables they considered, and then didn't control for. These not being reported is not universal - for example, I very frequently report every variable considered, and make my code available, with comments about variable selection within. What the parent post is talking about is "unmeasured confounding", or a Rumsfeldian "Unknown Unknown". If there is something that matters for your estimate, but you're unaware it exists, by definition you can neither report it nor control for it.
• fsloth 10y ago
  So why aren't mouse studies done in acoustically isolated rooms? Too expensive or lack of forethought? I know metrological laboratories that are mechanically insulated with all sorts of contraptions making the space vibration free so it'd not as if buildings could not be adapted for their intended scientific use as a matter of principle.
  • angry-hacker 10y ago
    This sounds a reasonable answer. If you can't do science in that environment, then it's not scientific because you can't control all the variables.
    • rtkwe 10y ago
      The classic ideal of controlling a single variable at a time is nice but largely impossible. Especially in biological sciences but that's true even down in chemistry and physics the systems scientists are trying to control are complex and it's not always possible to modify one variable without affecting others.
      • toss1941 10y ago
        How can you possibly draw any useful conclusions when your experiment's parameters are so fluid as to be uncontrollable?
        • rtkwe 10y ago
          In some cases you use various statistical tools over many trials or in others you measure the effects of the variables in other experiments. Or you can control the important variables and allow those with small or no effect on what you're measuring to vary slightly.
        • Obi_Juan_Kenobi 10y ago
          Independent experiments and controls. AKA, literally the basis of experimental science. Let's say you want to know where a protein localizes in a cell, of a given tissue, in both mutant and wild-type organisms. * Immunolocalization. You develop antibodies to the protein of interest, fix and mount tissue, perfuse it with the antibody, and use a secondary antibody to make it detectable. * Fluorescent tagging. You make a construct with your protein fused to a fluorescent protein. Usually this involves trying a few different tagging strategies until you find one that expresses well. Then you can make a stable transgenic, or try a transient assay. If it works, then you see some nice glowy confocal images. Be careful, though, as the tagging can affect protein localization. * Fractionation. In some cases, you can get a rough idea by e.g. extracting nuclei and doing a simple Western blot to see where the protein shows up. In the real world, you might start with the FP-tag and see that your protein is absent from the nucleus in your mutant. Which would be cool, and interesting in terms of figuring out it's function. If that was presented as the only result, I'd reject the paper and think the authors are terrible investigators. I'd want to see, at least, some nuclei preps that detect the protein in the WT, and don't in the mutant. I'd love to see immunos, too, as FP often does mess up the localization. You can take it even further and start doing deletions. You take the protein and crop bits out to see what happens. You should see stuff like removing the NLS makes it stop going to the nucleus. That's a good sanity check and a sign your methods are working. You can also try to mess with active sites, protein-protein interaction domains, etc. etc. All within a theoretical model of what you think is going on in the cell. Ultimately, the difficulty of replication isn't that troublesome. An inability to do so is, but science has never been easy. That's what you sign up for, and that's why you need to read papers critically. You get a sense of distrust for data unless there's really solid evidence of something. And when you find that solid evidence, you get a big smile and warm feeling in nerd heart.
• sametmax 10y ago
  Even if you do know all that, it means reproducing the experiment is costly. Controlling properly an environment take times and resources, and it's not like it's easy to get those for anything, let alone for something like peer reviewing.
• danjoc 10y ago
  >most of the time, when you are doing an experiment you never done, you just fuck it up. "Experiments are supposed to be replicable. The authors should have done it themselves before publication" That seems to imply the experiments aren't even replicated by the people who are supposed to know how to get it right.
• carsongross 10y ago
  > The methods section only contains variables the authors think worth controlling, and in reality you never know, and the authors never know. Think about the epistemological ramifications of this. This is why trust is so important in social contexts, even in science.
  • cwyers 10y ago
    > Think about the epistemological ramifications of this. This this this this this a million times this. If your experiment with lab rats needs to be done this finely to be replicated, is it really telling you anything about the real world? Because of course, we don't really care about the behavior of laboratory rats, right? Not in proportion to how often we do studies with laboratory rats versus other species of animals. We care because of what it can tell us about biology in general. And if the finding can't even generalize to "laboratory rats whose cages have been cleaned recently," does the study really say what it seems to?
    • IanCal 10y ago
      The point, I think, is that you want to control for things that really alter the outcome. It's not that a clean cage means that the end-product no longer works, but you want to compare "didn't have drug" vs "did have drug" not "didn't have drug" vs "did have drug but the fire alarm went off this morning". Unless your effect size is much larger than all the noise, you could easily miss a true result. Worse, you could get a false result the other way. What if your control group had all been spooked before they were measured a few times? That could make the control group seem worse than they are!
      • cwyers 10y ago
        So, let's continue with your example of a drug being given to the rodents. Now, you want to control diet -- you don't want your control group eating differently than your treatment group. This ends up being relatively easy to do with laboratory rodents, they aren't popping down to McDonalds for lunch or lying in their food diaries, like human subjects might. You buy your rodent food from one rodent pellet supplier, and you don't change brands or SKU during the experiment, and voila, control. And this works fine if the drug you're studying works identically given all rodent diets. But you don't know that it does! You're not controlling for the variable of diet in the sense that you know the effect of the variable of interest across all possible diets. You know how the drug works conditional on one specific rodent diet. And if you're not putting what brand of rodent food you use, and that causes replication failures, that suggests that you don't understand the effect of the drug as well as you thought you did. And if you have an entire field of research that undergoes such replication failures often, it's fair to start wondering how much of what gets called "science" isn't the study of the natural world but the study of the very specific sets of conditions that happen in labs in American and European universities.
• aaronchall 10y ago
  > Make sure the computer don't start to fucking updating windows half way during experiment. Solution? Linux. Also, use proper tense and grammar when describing methods, else others may find the instructions ambiguous. I do appreciate this response's level of detail... +1.
  • glibgil 10y ago
    > use proper tense and grammar when describing methods What a nauseatingly unnecessary comment. Boo
    • hackuser 10y ago
      I disagree. Grammar rules are a very effective checklist for clarity and precision in communication, IME. Does the verb have a subject and object? Otherwise, it's not clear who should do which to what. The instructions say, 'turn it on'; does "it" have a clear antecedent? Otherwise, what are they turning on? I would guess that the practical benefits are a big reason that they are the 'rules'.
      • deleted 10y ago
        [deleted]
• LordKano 10y ago
  Don't forget to make sure that none of the people who will be near the area of the experiment live with and thereby smell like a cat.
  • jonlucc 10y ago
    You can add to this that if you are in a facility with both rats and mice, you should never house them in the same room, and you should probably not enter a mouse room after being in a rat room. In the wild, rats are predators of mice.
    • cwyers 10y ago
      But in the wild, there aren't separate rooms for predators and prey, either. If laboratory rodent experiments are so reliant on such synthetic conditions, what's the point?
      • jpfed 10y ago
        If some mice are housed near rats and others are not (or simply vary in how far they are from the nearest rat), that will introduce variation into your measurements. For experiments not specifically about mouse/rat interactions, this variation is irrelevant noise that makes it harder to detect or characterize the effects the experimenters actually care about.
        • cwyers 10y ago
          That's not my point, though. Like, if you conduct physics experiments in a vacuum because interactions between objects and atmosphere isn't part of what you're studying, that's fine. If everyone in physics is conducting experiments that way, then suddenly you're left with the question of, it turns out that in real life I encounter atmosphere all the time, and how much can these experimental results tell me about the world I actually interact with? And if it turns out that all the preconditions to physics experiments can't be published because there are too many of them to list, and you're just expected to know these things, doesn't that throw the credibility of the whole enterprise into doubt? Controlling for variables is fine. But if you aren't comprehensively listing all the variables you're controlling for, if the very idea of doing so is considered a fool's errand, then are you even really doing science?
• benmmurphy 10y ago
  a load of variables that are needed to be controlled in order to get the original experiment to work is kind of a red flag tho. like someone could just keep trying the experiment and then adding another variable they are controlling in order to get the experiment to work. but really they are just running a bunch of experiments and getting lucky at some point. assuming they stop trying to control variables or try and replicate themselves as soon as they get a positive result :)
  • aaronmck 10y ago
    Yes an no. Of course, in something like an animal model it can be really hard to control everything, and the result you find could just be 'luck'. On the other hand, figuring out what variables to control is huge part of science. Say the development of next generation DNA sequencing technologies. People tried a ton of different variables, conditions, reagents, flow cells, etc. And failed and failed. But eventually they controlled the right conditions, optimized the right things, and now the process is done in thousands of labs every day as a routine tool. This is a technology development example, but the same could be said of the conditions needed to make stem cells.
  • Obi_Juan_Kenobi 10y ago
    It's not a red flag, it's reality. Biology is complicated, and no matter what the synthetic biologists tell you :) You use independent methods and good controls to deal with this. This is nothing new, it's been the basis of experimental science for decades. If you're allowed to publish without doing this, the field has failed. Molecular biology, in particular, has flourished because of the effectiveness of genetic controls. Mutants are highly reproducible (you just send out seed, cultures, or live specimens), verifiable (sequencing), and combinatoric (crossing).
  • Fomite 10y ago
    Eh... If adding a variable changes the actually effect being estimated, then it may indeed have been worth adding. If it doesn't, they'll just eat away at their precision.
• Chris2048 10y ago
  I would have thought that skinner boxes would be more sophisticated by now: - sound proof/resistant, record audio levels. - be hermetically sealed, temperature controlled and opaque to outside. Given something as complex as a living animal, why aren't they treated as controlled as a chemical might be?
• Balgair 10y ago
  A lot of the commenters are trying to come up with technical solutions to the issue. It belies their incomplete study of the 'softer' sciences and the difficulty of biological studies. Tech solutions won't work, bureaucratic solutions wont work, more data wont work. Guys, the bio side of things is incredibly complicated and trying to set controls that are achievable in your time and budget is the heart of these fields. The thing you are trying to study in the bio side of science is actually alive and trying to study you right on back. If you are going to kill the thingys, they really do want to kill you too. Look at this diagram of mitochondrial phospholipid gene/protein interactions (https://www.frontiersin.org/files/Articles/128067/fgene-06-00003-r2/image_m/fgene-06-00003-g005.jpg https://www.frontiersin.org/files/Articles/128067/fgene-06-0...). That is very complicated and that is for one of the best studied organelles in the Animalia family. There are an uncountable and evolving set of other proteins that then interact with that diagram in different ways depending on the cell-type, species, and developmental history of the organelle (to start with). All of which are totally unknown to you and will likely forever remain unknown to you up until your death. Hell, we are still figuring out the shapes of organs in our own bodies. People that have studied areas for decades spending untold millions of dollars on some of the most central questions of life have essentially nothing to say for themselves and the money spent('We dream because we get sleepy'). Trying to tease out a system that has been evolving for 4.5 billion years and makes a new generation (on average) every 20 minutes is going to be just insanely difficult. This stuff is hard, the fields know it, and we don't believe anything that someone else says, let alone what our 'facts' and experiments tell us. But we solider on because we love it and because we want to help the world.
  • epistasis 10y ago
    I think there's also a lot of confusion about what the scientific literature should be. Outsiders think of journals as pre-packaged nuggets of science fact, with conclusions that you can read and trust. Scientists who publish in the journals view them as a way to communicate a body of work of "I did this, here is the result I saw. I think it might mean this." The difference between these two views can not be understated. Each group wants the literature to be useful to them, so that they can use it, and that's understandable. For most areas of science, especially in the early days of that science, it's absolutely essential that scientist's view be allowed to persist. Is it better to share early, or to wait to publish until you've tried all possible things that could potentially go wrong? I think the answer is obviously that you share the early data, and what you think it means, even if you may be wrong about it. If the goal is to advance knowledge as quickly as possible, I think that the scientist's view is probably a better idea of what a journal needs to be than what the outsiders' view is. In some fields, like fMRI studies, the field is realizing that they may need to go about things differently. And that means that a lot of the interpretations that were published earlier are incorrect. But that process of incorrect interpretation to corrected interpretation is an essential part of science.
• return0 10y ago
  A good way to avoid this is using robots wherever possible.
• TheSpiceIsLife 10y ago
  Could this be addressed by referencing a standard mouse handling document, something similar to: Pre-experiment mouse handling procedures as outlined in section M.1.2 and M.2.4 of the Mouse Handling Procedure Standards Body Publication of 2014 Which document could have any number of sections detailing the finer points variable control for the relevant experiment.
  • SubiculumCode 10y ago
    There is continual progress on standardization of methodology, but it takes a while. People have vested interests in using their protocols, and there often valid disagreements about what is actually the best protocol. Nevertheless, these kinds of standardization documents are important. But only as long as deviations from these standards are not discounted when justified theoretically.
• hyperion2010 10y ago
  My PhD work is trying to address this by developing better ways for scientists to record and communicate their methods/protocols. Methods sections are NOT about providing the information needed to reproduce the work, often even the supplemental information is insufficient. They are a best guess at what needs to be done, and the methods section is often massively condensed due to editor demands and cites other papers that also don't contain the relevant information because some aspect of the method changed between papers. (I could go on and on about this.) I spent 7 years doing experimental biology (from bacteria to monkeys) and trying to replicate someone else's techniques from their papers was always a complete nightmare. Every experimentalist I talk to about this relates the same experience -- sometimes unprompted. Senior faculty tell a slightly different story, that they can't interpret data of someone who has left the lab, but it is the same issue. We must address this, we have no choice, we cannot continue has we have for the past 70+ years, the apprenticeship system does not scale for producing communicable/replicable results (though it is still the best way to train someone to actually do something). EDIT: An addendum. This stuff is hard even when you assume that all science is done in good faith. That said, malicious or fraudulent behaviour is much harder to hide when you have symbolic documentation/specification of what you claim you are doing, especially if they are integrated with data acquisition systems that sign their outputs. Still many ways around this, but post hoc tampering is hard if you publish a stream of git commit hashes publicly.
  • wuschel 10y ago
    True, biology is a horror show full of surprises. Many experimental instructions are as reliable as astrologic forecasts. I guess that is the price of complexity and the human factor. I would love to hear more about your work, and the strategies you propose to improve the reproducibility of scientific experiments. My email can be found in my user description. Cheers!
  • petra 10y ago
    Wouldn't it be hard to achieve deep consistency between experiments, in so many labs around the world, with so different conditions/cultures/etc , and when the experimenters aren't experts in consistency , but in their science ? Wouldn't it be better to use something like a cloud biology model - where you define experiments via code, CRO's compete on consistency(and efficiency and automation) and since they probably do a much larger volume of experiments than the regular lab, they would have stronger incentives to develop better processes and technologies ?
• busyant 10y ago
  I can't tell you how many times I tried (unsuccessfully) to replicate a procedure using the methods section in a research article. Then I contact the author and they send me a protocol that is a 'distant cousin' to what I read in the published article. This caused me to assiduously report methods when I wrote papers.
  • SubiculumCode 10y ago
    That is frustrating. In my field of cognitive neuroscience, there is often little incentive for a researcher to hide parts of their methods from their 'competitors' since, for example, my memory study is not about reporting but protecting a new technology. Indeed, prestige is often a consequence of others adopting your methods, so researchers are motivated to share scripts, and report the methods fully. HOWEVER, certain journals can place limits on the length of the methods section, which is a damn shame.
    • hackuser 10y ago
      > there is often little incentive for a researcher to hide parts of their methods from their 'competitors' since, for example, my memory study is not about reporting but protecting a new technology If you are protecting, not reporting information (e.g., about a new technology), why would you be incentivized to share it?
      • SubiculumCode 10y ago
        yeah it seems funny, but it happens all the time.Perhaps there are reporting requirements from the funding?
• maverick_iceman 10y ago
  Wouldn't this allow you to explain away all negative results? Say you get a negative result and you don't like it. Then you start fishing for a cause and found out someone sneezed during the experiment, so you'll delete the null result. Isn't that problematic?
• tnecniv 10y ago
  Heck, I've tried to reproduce CS papers with incomplete methodologies. There's always enough that that I know it will work, but they normally don't include all the tricks you need to get it to work easily or efficiently. Stuff like "we used an SMT solver with this important feature" but no mention of which solver or "a key part of this algorithm is factoring the sparse matrix with this non-standard technique that you'll have to implement yourself" but don't tell you the number of tricks they used to make it as fast as you need.
• TuringNYC 10y ago
  I'm surprised no one mentioned the company Mousera (http://www.vium.com/ http://www.vium.com/). Granted, it is not a solution for all of science, but it essentially brings down many animal experiments to scripts, which can be run almost like I might run a docker container and script. The idea is incredible. You can even scale it up by running the script multiple times. Disclaimer: I did bench and animal work 21yrs ago during undergrad, but havent used this company. So I know the field and how difficult replication is, but i'm not sure what has happened in wet science since 1997.
• mtdewcmu 10y ago
  What if you took the information that doesn't fit in the paper and publish it on the web?
• facepalm 10y ago
  Shouldn't it be the responsibility of the original experimenter to provide the information necessary to repeat the experiment? I see nothing in your list that sounds like a good excuse for not being repeatable.
• pdkl95 10y ago
  http://calteches.library.caltech.edu/51/2/CargoCult.htm http://calteches.library.caltech.edu/51/2/CargoCult.htm "Cargo Cult Science" > I looked into the subsequent history of this research. The subsequent experiment, and the one after that, never referred to Mr. Young. They never used any of his criteria of putting the corridor on sand, or being very careful. They just went right on running rats in the same old way, and paid no attention to the great discoveries of Mr. Young, and his papers are not referred to, because he didn’t discover anything about the rats. In fact, he discovered all the things you have to do to discover something about rats. But not paying attention to experiments like that is a characteristic of Cargo Cult Science.
• ktamiola 10y ago
  Nothing to add!