4 ms·
> http://pages.cs.wisc.edu/~kovar/hall.html http://pages.cs.wisc.edu/~kovar/hall.html This reminds me of my bioinformatics class. The final project was to rep
by fpgaminer 9y ago
> http://pages.cs.wisc.edu/~kovar/hall.html http://pages.cs.wisc.edu/~kovar/hall.html
This reminds me of my bioinformatics class. The final project was to reproduce the results of a famous paper in the field.
All of us spent _weeks_ trying to do it. Nobody succeeded. The more we dug into the paper, the more holes appeared. There were variables missing in the paper, assumptions not covered, datasets not properly specified, etc. It made reproduction nearly impossible; like winning the lottery. Imagine trying to recreate the results of a deep learning paper without the paper specifying _any_ information about the layers used, their sizes, or any hyperparameters.
The professor was equally mystified.
Years later I learned this kind of pseudo-science is rife in the field of bioinformatics. I felt both a sense of relief in knowing we weren't crazy, and disappointment. I actually really liked that class; the field of bioinformatics fascinated me. But realizing what a cesspool it was, left me disappointed.
I'm glad machine learning as a field has taken proactive steps to avoid these exact kinds of issues. It's now common practice in ML to publish code and models alongside your papers, and most ML libraries allow deterministic training. This makes reproduction of results easy. It's a breath of fresh air. That doesn't obviate all problems. Methodologies and conclusions are still up for debate in any given paper. But at least the experiments themselves are reproducible. And if you question the methodology or some aspect of the experiment, you can go in and augment the experiment yourself.
- mulmen 9y agoI would say this kind of lesson is the most valuable you could learn. Having people with this kind of experience enhances any field they participate in. Dieselgate started with a team of students attempting to reproduce VW's claimed emission numbers.
- sevenfive 9y agoCurious, can you link the paper?
- fpgaminer 9y agoThe paper was Neuronal Transcriptome of Aplysia: Neuronal Compartments and Circuitry by Moroz This was a decade ago. Looking at the paper again I believe we only tried to reproduce a small portion of it; the phylogeny tree from the paper and its supplemental material.
- deepnotderp 9y ago> I'm glad machine learning as a field has taken proactive steps to avoid these exact kinds of issues. Yeah about that, I've got some bad news...
- twic 9y agoIt's not just bioinformatics. Wet-lab biology is often similar. A friend of mine spent a good chunk of his PhD trying to reproduce an experiment involving growing primary cells in serum-free medium (the idea was to use that experiment as a starting point, and explore more aspects of it). The protocol was: 1. Grow some regular immortal cells in serum-based media in a dish, so they coat the dish with extracellular matrix 2. Use trypsin to detach the cells from the dish and remove them 3. Wash the dish carefully to remove all traces of serum, but leaving the extracellular matrix 4. Plate the primary cells onto the dish and grow them in serum-free medium He tried for months and couldn't get the cells to grow. Then he got sloppy, didn't wash the dishes as carefully as he should, and bingo, the primary cells grew fine, as described in the original paper. After some subtle digging, the inescapable conclusion was that the original authors had not washed their plates all that carefully either, and the serum-free medium was not exactly that. The whole premise of the experiment was flawed.