Not sure if I follow your JFT argument, but there's a large body of work on both (a) studying whether chasing ImageNet accuracy yields models that generalize we

Not sure if I follow your JFT argument, but there's a large body of work on both (a) studying whether chasing ImageNet accuracy yields models that generalize well to out of distribution data [1, 2, 3] and (b) contextualizing progress on ImageNet (i.e., what does high accuracy on ImageNet really mean?) [4, 5, 6]. For (a), maybe surprisingly the answer is mostly yes! Better ImageNet accuracy generally corresponds to better out of distribution accuracy. For (b), it turns out that the ImageNet dataset is full of contradictions---many images have multiple ImageNet-relevant objects, and often are ambiguously or mis-labeled, etc---so it's hard to disentangle progress in identifying objects vs. models overfitting to the quirks of the benchmark. [1] ObjectNet: https://objectnet.dev https://objectnet.dev / associated paper [2] ImageNet-v2: https://arxiv.org/abs/1902.10811 https://arxiv.org/abs/1902.10811 [3] An Unbiased Lookat Dataset Bias: https://people.csail.mit.edu/torralba/publications/datasets_cvpr11.pdf https://people.csail.mit.edu/torralba/publications/datasets_... (pre-AlexNet!) [4] From ImageNet to Image Classification: https://arxiv.org/abs/2005.11295 https://arxiv.org/abs/2005.11295 [5] Are we done with ImageNet? https://arxiv.org/abs/2006.07159 https://arxiv.org/abs/2006.07159 [6] Evaluating Machine Accuracy on ImageNet: http://proceedings.mlr.press/v119/shankar20c.html http://proceedings.mlr.press/v119/shankar20c.html