Hey, I'm a ML practitioner for over 6 years and I'm glad to help. 1k distinct labels with a long tail distributions which is what you are describing, is defini

Hey, I'm a ML practitioner for over 6 years and I'm glad to help. 1k distinct labels with a long tail distributions which is what you are describing, is definitely a challenging problem. It's called a imbalanced classification problem. I'd focus first test how well your model will be able to predict these classes doing a stratified cross validation (stratification controlled by the label class), and measure the F-score, Weighted Accuracy and ROC AUC. Check also the precision and recall for each class. You'll definitely see that the model predicts better for the labels with more samples. The code that you use here, you'll be able to reuse later on, so keep it organized and easy to follow. Then you have a couple options, focus on gathering more examples for the labels with small sample size, or try to oversample your dataset. This article is a good place to start https://towardsdatascience.com/4-ways-to-improve-class-imbalance-for-image-data-9adec8f390f1 https://towardsdatascience.com/4-ways-to-improve-class-imbal... Considering this problem of image classification is normally solved with deep learning, the more data you have, the better will be your results.