On the other hand, some of their correct images have questionable captions as well. For example, they labeled the geyser picture correctly, but their top label

On the other hand, some of their correct images have questionable captions as well. For example, they labeled the geyser picture correctly, but their top labels also included "sandbar", "breakwater" and "leatherback turtle". A better scoring function, perhaps including hierarchies to account for the very vague "restaurant" photos and very specific dog breed photos, might be helpful. Otherwise, it seems like we might be overfitting to the peculiarities of this dataset.