4 ms·

To add to this, there are many not-so-great side effects due to all of the complexity even if the code is all correct and perfectly reproduced. For instance, I

To add to this, there are many not-so-great side effects due to all of the complexity even if the code is all correct and perfectly reproduced. For instance, I suspect a large fraction of claimed "better than baseline" results in papers are actually a result of well-tuned proposed model and not-very-well-tuned baseline. Papers usually say something along the lines of "we tune these hyperaparameters with crossvalidation", but you can be very thorough or not very thorough in this process and achieve very different results. Or if the author is (intentionally or unintentionally) devious, they could adjust the hyperparameter search ranges in a way that doesn't cover the optimal region, or is way too broad. All of these problems combined are the reason that I pay much less attention to numbers in the tables of papers. Papers are, unfortunately, more of a source of cool ideas and inspiration for stuff to put into your toolbox and maybe try yourself in your next project.

Smerity 9y ago
+1. The "well-tuned proposed model and not-very-well-tuned baseline" is something I feel nearly every researcher is guilty of, including myself :) It's especially pronounced however when people compare to a baseline from paper X (usually by copying and pasting the number) which may be a year or more old. A year or two may not sound like much but for deep learning but you might as well be comparing Heron's engine[1] to a modern combustible engine. The most cited attention paper, Bahdanau et al. 2014, was only two and a half years ago - and their results weren't all that strong initially anyway. There are so many minor auxiliary improvements / newly found domain expertise that can add up to a substantially improved result (initializations, regularization techniques, hyper parameter choices that work well for that given task / dataset, ...) even for almost the exact same model. As an example of this, I implemented a Keras SNLI model[2] as a teaching aid for friends only to find my simplest baseline outperformed a number of fairly recent papers. This is not necessarily an issue at all with those papers - many of the ideas and techniques they propose have gone on to be used in other tasks and models - but it does indicate how these additional best practices / more time on the baseline can really accumulate over only months. Keras is a brilliant example of that as it's "best practices included" - when you use an LSTM it'll default to strong weight initializations etc. I'm vaguely hoping that in the future there can be workshops at conferences titled "Pushing the baselines" where we take some time to see what new techniques can apply to old baselines and/or tune baselines with the explicit goal of getting them to regain their edge. There's at least one at ICML on reproducibility[3] which I'm looking forward to :) [1]: https://en.wikipedia.org/wiki/Aeolipile https://en.wikipedia.org/wiki/Aeolipile [2]: https://github.com/Smerity/keras_snli https://github.com/Smerity/keras_snli [3]: https://sites.google.com/view/icml-reproducibility-workshop/home https://sites.google.com/view/icml-reproducibility-workshop/...

Smerity 9y ago

+1. The "well-tuned proposed model and not-very-well-tuned baseline" is something I feel nearly every researcher is guilty of, including myself :) It's especially pronounced however when people compare to a baseline from paper X (usually by copying and pasting the number) which may be a year or more old. A year or two may not sound like much but for deep learning but you might as well be comparing Heron's engine[1] to a modern combustible engine. The most cited attention paper, Bahdanau et al. 2014, was only two and a half years ago - and their results weren't all that strong initially anyway. There are so many minor auxiliary improvements / newly found domain expertise that can add up to a substantially improved result (initializations, regularization techniques, hyper parameter choices that work well for that given task / dataset, ...) even for almost the exact same model. As an example of this, I implemented a Keras SNLI model[2] as a teaching aid for friends only to find my simplest baseline outperformed a number of fairly recent papers. This is not necessarily an issue at all with those papers - many of the ideas and techniques they propose have gone on to be used in other tasks and models - but it does indicate how these additional best practices / more time on the baseline can really accumulate over only months. Keras is a brilliant example of that as it's "best practices included" - when you use an LSTM it'll default to strong weight initializations etc. I'm vaguely hoping that in the future there can be workshops at conferences titled "Pushing the baselines" where we take some time to see what new techniques can apply to old baselines and/or tune baselines with the explicit goal of getting them to regain their edge. There's at least one at ICML on reproducibility[3] which I'm looking forward to :) [1]: https://en.wikipedia.org/wiki/Aeolipile https://en.wikipedia.org/wiki/Aeolipile [2]: https://github.com/Smerity/keras_snli https://github.com/Smerity/keras_snli [3]: https://sites.google.com/view/icml-reproducibility-workshop/home https://sites.google.com/view/icml-reproducibility-workshop/...