I appreciate the authors calling out #2. ImageNet and CIFAR are both "solved" benchmarks, to the extent that state of the art algorithms by now are likely overf

I appreciate the authors calling out #2. ImageNet and CIFAR are both "solved" benchmarks, to the extent that state of the art algorithms by now are likely overfitting to the specific dataset details. In particular, objects in ImageNet are nearly always in canonical pose, with no occlusion, few confounding objects, illuminated, and in a semantically-obvious configuration. For some applications this is acceptable but as an industry benchmark ImageNet is not informative (user generated photo distributions are never this clean). Even more damning is the recent BagNet paper (nice summary here: https://blog.evjang.com/2019/02/bagnet.html https://blog.evjang.com/2019/02/bagnet.html), which indicates that ImageNet can likely be solved with no global features (i.e. model doesn't have to learn anything truly abstract, just configurations of textures, shapes, colors). I thought the author of that blog post put it nicely: "As someone who is deeply interested in AGI, I find ImageNet much less interesting now, precisely because it can be solved with models that have little global understanding of images."