The ImageNet-A dataset (https://github.com/hendrycks/natural-adv-examples https://github.com/hendrycks/natural-adv-examples) contains thousands of hand-curated

The ImageNet-A dataset (https://github.com/hendrycks/natural-adv-examples https://github.com/hendrycks/natural-adv-examples) contains thousands of hand-curated examples that are challenging for off-the-shelf classifiers. It's interesting to look through them and guess why each one's so difficult. For examples like the surgical mask, the ImageNet "mask" category is for masquerade masks and halloween masks, I suspect it doesn't contain anything similar to a surgical mask. Imagenet categories are a bit weird that way - they're a pretty arbitrary set of labels that were selected for use as a benchmark, not for practical use. It's got 100 different breeds of dogs, because it was a way to test fine-grained classification. It's got crane (the bird) and crane (the machine). It's got 'car mirror' but not 'mirror'. It's got 'yurt' but not 'house'. It's great if the categories you care about happen to match the ones on the list (in both name and type, as seen in the 'mask' example). But otherwise you'll quickly run into the need to fine-tune the model on some of your own data to get the categories you want.