4 ms·

I could use some help with some heuristics for Machine Learning, like how much data do I need to make a workable model, what framework/approach makes more sense

I could use some help with some heuristics for Machine Learning, like how much data do I need to make a workable model, what framework/approach makes more sense given my ultimate goals. Here's an example: there's a lot of ML tutorials on doing image identification. Like you have a series of images: picture one might have an apple and a pear in it, picture 2 might have an apple, orange, and a banana in it. Where I'm struggling is putting this into my domain. I have a 100k images and from that around 1k distinct labels (individual images can have between 1 to 7 of these different labels), with between 13,000 to 100 images as examples in each label. Is that enough data? Should I just start working on gathering more? Is this a bad fit for a ML solution?

ad404b8a372f2b9 5y ago
Computer Vision is the one domain where ML (and neural networks in particular) is the undisputed king. Unless you're in an embedded application where you can't use neural networks in which case you might want to go with handcrafted features to trade-off accuracy for speed/compute-efficiency. With regards to the size of your dataset, there's no hard rules it depends on the complexity of the task. Problems with a high number of classes are among the more difficult ones, 100 samples per class might be enough or it might not . The only way to know for sure it to try and see if you reach a performance that's acceptable for your application. I recommend the Pytorch framework, it's coherent, easy to use and well documented (both the API reference and the examples available on SO and throughout the net). Your problem is similar to imagenet (assuming you want to detect the presence of a class and not its position, in which case it's a different problem.) so you can try to run one of the pytorch tutorials and see how well it does. The only difference is you want to detect multiple classes in one single image so you'll have to adjust your output layer and loss potentially but the network itself could remain the same. You also might want to look into doing transfer learning with a pretrained imagenet network to speed-up the training.
Y_Y 5y ago
The answer is, "it depends". My recommendation would be to just start trying. What you describe sounds like you could train it in a couple of days. Grab an off-the-shelf resnet and just see what happens. This is a well-studied problem, you can just look at papers that train on imagenet, and then tweak their approaches.
thegginthesky 5y ago
Hey, I'm a ML practitioner for over 6 years and I'm glad to help. 1k distinct labels with a long tail distributions which is what you are describing, is definitely a challenging problem. It's called a imbalanced classification problem. I'd focus first test how well your model will be able to predict these classes doing a stratified cross validation (stratification controlled by the label class), and measure the F-score, Weighted Accuracy and ROC AUC. Check also the precision and recall for each class. You'll definitely see that the model predicts better for the labels with more samples. The code that you use here, you'll be able to reuse later on, so keep it organized and easy to follow. Then you have a couple options, focus on gathering more examples for the labels with small sample size, or try to oversample your dataset. This article is a good place to start https://towardsdatascience.com/4-ways-to-improve-class-imbalance-for-image-data-9adec8f390f1 https://towardsdatascience.com/4-ways-to-improve-class-imbal... Considering this problem of image classification is normally solved with deep learning, the more data you have, the better will be your results.
- michaelbuckbee 5y ago
  This was very helpful, thank you so much!
mhuffman 5y ago
If you have not at least skimmed through fast.ai, you should definitely do that, as the course itself addresses some of this and the people on their forums are among the most helpful I have ever seen! Second, this could be more than enough! Especially if you are doing transfer learning. Third, you can "inflate" the amount of images you have now with "Image Data Augmentation"
rolisz 5y ago
Define workable model. Do you care more about recall (how many of the images with label X will be labeled by your model with label X) or precision (how many of the images where the model says have label X are actually with label X)? It is a good fit for ML, but you need to be clear on what the results will be. If you expect 100% accuracy, that won't happen. Even 90%+ accuracy would be require a lot of effort.
TrackerFF 5y ago
Sounds like a problem which YOLO can solve pretty easily , i.e object detection and classification, transfer learning, etc. Try downloading a pre-trained model, and play around with that. (The other replies have outlines what you need to focus on)

ad404b8a372f2b9 5y ago

Computer Vision is the one domain where ML (and neural networks in particular) is the undisputed king. Unless you're in an embedded application where you can't use neural networks in which case you might want to go with handcrafted features to trade-off accuracy for speed/compute-efficiency. With regards to the size of your dataset, there's no hard rules it depends on the complexity of the task. Problems with a high number of classes are among the more difficult ones, 100 samples per class might be enough or it might not . The only way to know for sure it to try and see if you reach a performance that's acceptable for your application. I recommend the Pytorch framework, it's coherent, easy to use and well documented (both the API reference and the examples available on SO and throughout the net). Your problem is similar to imagenet (assuming you want to detect the presence of a class and not its position, in which case it's a different problem.) so you can try to run one of the pytorch tutorials and see how well it does. The only difference is you want to detect multiple classes in one single image so you'll have to adjust your output layer and loss potentially but the network itself could remain the same. You also might want to look into doing transfer learning with a pretrained imagenet network to speed-up the training.

Y_Y 5y ago

The answer is, "it depends". My recommendation would be to just start trying. What you describe sounds like you could train it in a couple of days. Grab an off-the-shelf resnet and just see what happens. This is a well-studied problem, you can just look at papers that train on imagenet, and then tweak their approaches.

thegginthesky 5y ago

Hey, I'm a ML practitioner for over 6 years and I'm glad to help. 1k distinct labels with a long tail distributions which is what you are describing, is definitely a challenging problem. It's called a imbalanced classification problem. I'd focus first test how well your model will be able to predict these classes doing a stratified cross validation (stratification controlled by the label class), and measure the F-score, Weighted Accuracy and ROC AUC. Check also the precision and recall for each class. You'll definitely see that the model predicts better for the labels with more samples. The code that you use here, you'll be able to reuse later on, so keep it organized and easy to follow. Then you have a couple options, focus on gathering more examples for the labels with small sample size, or try to oversample your dataset. This article is a good place to start https://towardsdatascience.com/4-ways-to-improve-class-imbalance-for-image-data-9adec8f390f1 https://towardsdatascience.com/4-ways-to-improve-class-imbal... Considering this problem of image classification is normally solved with deep learning, the more data you have, the better will be your results.

michaelbuckbee 5y ago
This was very helpful, thank you so much!

mhuffman 5y ago

If you have not at least skimmed through fast.ai, you should definitely do that, as the course itself addresses some of this and the people on their forums are among the most helpful I have ever seen! Second, this could be more than enough! Especially if you are doing transfer learning. Third, you can "inflate" the amount of images you have now with "Image Data Augmentation"

rolisz 5y ago

Define workable model. Do you care more about recall (how many of the images with label X will be labeled by your model with label X) or precision (how many of the images where the model says have label X are actually with label X)? It is a good fit for ML, but you need to be clear on what the results will be. If you expect 100% accuracy, that won't happen. Even 90%+ accuracy would be require a lot of effort.

TrackerFF 5y ago

Sounds like a problem which YOLO can solve pretty easily , i.e object detection and classification, transfer learning, etc. Try downloading a pre-trained model, and play around with that. (The other replies have outlines what you need to focus on)