6 ms·

He didn't say you couldn't, he said that's what Google means.

sanxiyn 9y ago
My understanding is that Google very much intends Cloud AutoML Vision usable for small datasets. To me "deep learning ... inherently impossible to train small" sounds entirely confused.
- yorwba 9y ago
  The whole advantage of deep learning comes from the large number of parameters that can fit very complex feature distributions. If you don't have a lot of data and try to train a deep net on it, you're just going to overfit. Of course there's the opportunity for transfer learning where you try to adjust a pretrained model slightly for a different task, but that only works if the model already mostly solves your problem. (e.g. it picks up on the necessary features, but the class you want is not one of the possible outputs.) If your data is unlike any popular task with large datasets available, (say because you're trying to predict something from a small in-house database), transfer learning can't work (because there's no knowledge you could transfer) and you are better off with simpler models.
YeGoblynQueenne 9y ago
Cheers for the assist, but I really did mean that it's impossible to train deep learning with small datasets (also, I'm a "she" :). yorwba below explains why transfer learning is not a way to train a deep net with small data sets. Rather, it's a way to re-train an already trained deep net. If you don't have a deep net trained on your target domain, then what? Software devs are used to having access to libraries that make our lives simple. You don't have to invent a new language and write a compiler for it everytime you want to code, because java etc. That is the hope of services like the one discussed above. However, because trained deep nets are very task-specific and it's virtually impossible for a small developer team with few resources to train one from scratch to reach state-of-the-art performance, developers who choose to use pre-trained deep nets as a library are perpetually dependent on Google, or whatever big company decides to offer its deep nets as a service, to determine the range of uses of their re-trained nets.

My understanding is that Google very much intends Cloud AutoML Vision usable for small datasets. To me "deep learning ... inherently impossible to train small" sounds entirely confused.

yorwba 9y ago
The whole advantage of deep learning comes from the large number of parameters that can fit very complex feature distributions. If you don't have a lot of data and try to train a deep net on it, you're just going to overfit. Of course there's the opportunity for transfer learning where you try to adjust a pretrained model slightly for a different task, but that only works if the model already mostly solves your problem. (e.g. it picks up on the necessary features, but the class you want is not one of the possible outputs.) If your data is unlike any popular task with large datasets available, (say because you're trying to predict something from a small in-house database), transfer learning can't work (because there's no knowledge you could transfer) and you are better off with simpler models.

YeGoblynQueenne 9y ago

Cheers for the assist, but I really did mean that it's impossible to train deep learning with small datasets (also, I'm a "she" :). yorwba below explains why transfer learning is not a way to train a deep net with small data sets. Rather, it's a way to re-train an already trained deep net. If you don't have a deep net trained on your target domain, then what? Software devs are used to having access to libraries that make our lives simple. You don't have to invent a new language and write a compiler for it everytime you want to code, because java etc. That is the hope of services like the one discussed above. However, because trained deep nets are very task-specific and it's virtually impossible for a small developer team with few resources to train one from scratch to reach state-of-the-art performance, developers who choose to use pre-trained deep nets as a library are perpetually dependent on Google, or whatever big company decides to offer its deep nets as a service, to determine the range of uses of their re-trained nets.