2 ms·
Some papers take more than a day of work to reproduce A large part of my job is this. Generally reproducing a paper with no code is months of work. Doing GPT-
by nl 7y ago
Some papers take more than a day of work to reproduce
A large part of my job is this. Generally reproducing a paper with no code is months of work.
Doing GPT-2 from scratch is around $50k or compute time and data collection.
It's extraordinarily rare you need to do this though. I know a few who have (mostly for foreign languages) and all have been able to access TPU grants from Google or multi-node GPU clusters (which are pretty easy to find if you work in the field - plenty of vendors want someone to test their "supercomputer" for a a week)
Even a simple deep learning paper will require you to have at least a NVIDIA 1080ti if you are lucky and for NLP I needed to buy a RTX Titan ($2500 graphics card
Most of my work is in NLP and I do a large amount of it on a 1070.
- zitterbewegung 7y agoYea GPT2 requires a bit more ram than on the 10 or 20 aeries TI cards . My point I think was unclear was the issue of reproducing the paper from scratch (I guess the model is enough ?) not sure how those papers are peer reviewed unless they also send the dataset to the reviewers ? I also was unclear was when I said reproducing I mean to just get a model that has already been pretrained. I agree with your points .
- nl 7y agoPeer review almost never means reproducing the model, but that isn't because of the dataset (which is usually available to the reviewer) but because that's not what a peer review is! A peer review isn't an adversarial process where you think that the person has done something wrong. Instead, it's extra eyes on it to make sure they have thought of everything, and to say what additional tests might be needed and why.