+1. The "well-tuned proposed model and not-very-well-tuned baseline" is something I feel nearly every researcher is guilty of, including myself :) It's especial

+1. The "well-tuned proposed model and not-very-well-tuned baseline" is something I feel nearly every researcher is guilty of, including myself :) It's especially pronounced however when people compare to a baseline from paper X (usually by copying and pasting the number) which may be a year or more old. A year or two may not sound like much but for deep learning but you might as well be comparing Heron's engine[1] to a modern combustible engine. The most cited attention paper, Bahdanau et al. 2014, was only two and a half years ago - and their results weren't all that strong initially anyway. There are so many minor auxiliary improvements / newly found domain expertise that can add up to a substantially improved result (initializations, regularization techniques, hyper parameter choices that work well for that given task / dataset, ...) even for almost the exact same model. As an example of this, I implemented a Keras SNLI model[2] as a teaching aid for friends only to find my simplest baseline outperformed a number of fairly recent papers. This is not necessarily an issue at all with those papers - many of the ideas and techniques they propose have gone on to be used in other tasks and models - but it does indicate how these additional best practices / more time on the baseline can really accumulate over only months. Keras is a brilliant example of that as it's "best practices included" - when you use an LSTM it'll default to strong weight initializations etc. I'm vaguely hoping that in the future there can be workshops at conferences titled "Pushing the baselines" where we take some time to see what new techniques can apply to old baselines and/or tune baselines with the explicit goal of getting them to regain their edge. There's at least one at ICML on reproducibility[3] which I'm looking forward to :) [1]: https://en.wikipedia.org/wiki/Aeolipile https://en.wikipedia.org/wiki/Aeolipile [2]: https://github.com/Smerity/keras_snli https://github.com/Smerity/keras_snli [3]: https://sites.google.com/view/icml-reproducibility-workshop/home https://sites.google.com/view/icml-reproducibility-workshop/...