> “This year, we’re getting Z% correct, whereas last year we could only get (Z-ε)%” does sound like progress. But is it meaningful? This is one thing that both

> “This year, we’re getting Z% correct, whereas last year we could only get (Z-ε)%” does sound like progress. But is it meaningful? This is one thing that bother's me a lot when I read published work. I have a feeling this is a result of everyone using the same benchmark datasets, so it inevitably becomes more of an _engineering_ exercise rather than scientific progress. In NLP the difference between a publishable result and one that is not is often to squeeze out a few extra (Z-ε)% by throwing in an attention mechanism and ensembles to your new super duper improved SOTA architecture. This is the problem of "replicability", which really requires more than just the same benchmark dataset used over and over again. The author seems to touch on this point later on. Then there's the issue of "reproducibility". Very few researchers seem to publish their code with instructions how to build and re-create their results. What an awful lot of time is wasted trying to reproduce results. Here's a good example: https://groups.google.com/forum/#!topic/word2vec-toolkit/Q49FIrNOQRo https://groups.google.com/forum/#!topic/word2vec-toolkit/Q49...