5 ms·

Is transfer learning really not widely known by people doing AI? In my field, computer vision, it is used by most of the papers in the past three years in CVPR,

Is transfer learning really not widely known by people doing AI? In my field, computer vision, it is used by most of the papers in the past three years in CVPR, etc. All of the students that take either my deep learning or my computer vision courses have to do assignments on transfer learning with deep neural networks.

sarthakjain 10y ago
Totally agree, everybody in the industry knows about it. However if you look at https://www.google.com/trends/explore?date=2014-01-01%202017-01-01&q=Deep%20Learning,Machine%20Learning,Transfer%20Learning#TIMESERIES https://www.google.com/trends/explore?date=2014-01-01%202017... nobody outside seems to know. I might be wrong but a lot of people outside the ML community seem to be hesitant to using ml because they don't have enough data, trying to remove the misconception if it exists
- averagewall 10y ago
  That graph might be more because it's not as exciting. Google searches are probably dominated by hobbyists and casually interested people. If they're not trying to achieve a specific goal, then they might prefer to work out the basics and make it themselves instead of just taking part of someone else's work and reusing it. If you were going to do that for fun, why not go the whole hog and reuse an entire pretrained network? Personally, I'm a hobbyist and I don't want to know about these shortcuts until I start to need them - which is a stage I might never reach. People who've progressed far enough to need them are probably far fewer than those who are just curious what these words mean. Another possibility is the words "transfer learning" might be more generally meaningful outside the ML field than the other search terms on the graph, so most of the searches for it are really schoolteachers or something else.
  - sarthakjain 10y ago
    Spot on. If you take that argument a step further that means an average developer who is not a data scientist or ml researcher might not know about it. Which implies a super easy to use dead simple technique which is used by most researchers is not available to the common developer even though it is easy enough for them to use.
  - deleted 10y ago
    [deleted]
- qxcv 10y ago
  Google Trends can tell you a lot of things :) https://www.google.com/trends/explore?date=2014-01-01%202017-01-01&q=Transfer%20Learning,backpropagation,Stochastic%20gradient%20descent#TIMESERIES https://www.google.com/trends/explore?date=2014-01-01%202017... More seriously though: others have pointed out that finetuning is pretty popular in some subfields, but it's just one hammer in a of a whole toolbox of techniques which are necessary to make neural nets train (even when you have a tonne of data). Standardisation, choice of initialisation, and choice of learning rate schedule all come to mind as other factors which seem simple, but which can have a huge impact in practice. Of course, each tool has its limitations. The most obvious limitation of finetuning is that you need a network that's already been trained on vaguely similar data. Pretraining on ImageNet is probably not going to help you solve problems where the size of objects matters, for example, because most ImageNet performance tends to benefit from scale invariance. I wish you luck with nanonets.ai, but I think it's irresponsible to market this as the "1 weird trick" to bring data efficiency to neural nets.
kyloon 10y ago
It is probably transfer learning has become more of a technique used for training neural network models (such as Adam being one of the most commonly used optimizer) whereas most of the excitement stems from new and/or complex neural network architectures rather than the technique or tools that made training those neural networks possible.
- prats226 10y ago
  True that most excitement in the field has been in finding new architectures and building general purpose AI. However I would like to point out that people often underestimate potential of transfer learning. We have started solving real problems companies are facing right now because of lack of enough data to be able to train an accurate model themselves.
- sarthakjain 10y ago
  Your right about excitement been generated from new techniques in the research community. Also if you look at Prisma, Google photos or any number of applications that have gained mass popularity. They are just implementations of existing networks and techniques. The paper on style transfer made Prisma possible, commercialization usually has been a few steps behind research.
- almondai 10y ago
  Transfer learning in ML refers to the general idea of taking a model trained for one domain and applying it to another. However this article seems to only focus on feature mapping, ie. breaking down images into features using hidden layers of ImageNet models. In this case, the pretrained model is only acting as a feature extractor because it is not trained to maximize the embedded distance between the classes you are trying to differentiate.
  - sarthakjain 10y ago
    Your right the article doesn't do justice to all of the transfer learning techniques. I cut out large portions of the original draft for brevity. Will write a longer post referencing all possible techniques, hopefully it won't turn into a review paper.

sarthakjain 10y ago

Totally agree, everybody in the industry knows about it. However if you look at https://www.google.com/trends/explore?date=2014-01-01%202017-01-01&q=Deep%20Learning,Machine%20Learning,Transfer%20Learning#TIMESERIES https://www.google.com/trends/explore?date=2014-01-01%202017... nobody outside seems to know. I might be wrong but a lot of people outside the ML community seem to be hesitant to using ml because they don't have enough data, trying to remove the misconception if it exists

averagewall 10y ago
That graph might be more because it's not as exciting. Google searches are probably dominated by hobbyists and casually interested people. If they're not trying to achieve a specific goal, then they might prefer to work out the basics and make it themselves instead of just taking part of someone else's work and reusing it. If you were going to do that for fun, why not go the whole hog and reuse an entire pretrained network? Personally, I'm a hobbyist and I don't want to know about these shortcuts until I start to need them - which is a stage I might never reach. People who've progressed far enough to need them are probably far fewer than those who are just curious what these words mean. Another possibility is the words "transfer learning" might be more generally meaningful outside the ML field than the other search terms on the graph, so most of the searches for it are really schoolteachers or something else.
- sarthakjain 10y ago
  Spot on. If you take that argument a step further that means an average developer who is not a data scientist or ml researcher might not know about it. Which implies a super easy to use dead simple technique which is used by most researchers is not available to the common developer even though it is easy enough for them to use.
- deleted 10y ago
  [deleted]
qxcv 10y ago
Google Trends can tell you a lot of things :) https://www.google.com/trends/explore?date=2014-01-01%202017-01-01&q=Transfer%20Learning,backpropagation,Stochastic%20gradient%20descent#TIMESERIES https://www.google.com/trends/explore?date=2014-01-01%202017... More seriously though: others have pointed out that finetuning is pretty popular in some subfields, but it's just one hammer in a of a whole toolbox of techniques which are necessary to make neural nets train (even when you have a tonne of data). Standardisation, choice of initialisation, and choice of learning rate schedule all come to mind as other factors which seem simple, but which can have a huge impact in practice. Of course, each tool has its limitations. The most obvious limitation of finetuning is that you need a network that's already been trained on vaguely similar data. Pretraining on ImageNet is probably not going to help you solve problems where the size of objects matters, for example, because most ImageNet performance tends to benefit from scale invariance. I wish you luck with nanonets.ai, but I think it's irresponsible to market this as the "1 weird trick" to bring data efficiency to neural nets.

kyloon 10y ago

It is probably transfer learning has become more of a technique used for training neural network models (such as Adam being one of the most commonly used optimizer) whereas most of the excitement stems from new and/or complex neural network architectures rather than the technique or tools that made training those neural networks possible.

prats226 10y ago
True that most excitement in the field has been in finding new architectures and building general purpose AI. However I would like to point out that people often underestimate potential of transfer learning. We have started solving real problems companies are facing right now because of lack of enough data to be able to train an accurate model themselves.
sarthakjain 10y ago
Your right about excitement been generated from new techniques in the research community. Also if you look at Prisma, Google photos or any number of applications that have gained mass popularity. They are just implementations of existing networks and techniques. The paper on style transfer made Prisma possible, commercialization usually has been a few steps behind research.
almondai 10y ago
Transfer learning in ML refers to the general idea of taking a model trained for one domain and applying it to another. However this article seems to only focus on feature mapping, ie. breaking down images into features using hidden layers of ImageNet models. In this case, the pretrained model is only acting as a feature extractor because it is not trained to maximize the embedded distance between the classes you are trying to differentiate.
- sarthakjain 10y ago
  Your right the article doesn't do justice to all of the transfer learning techniques. I cut out large portions of the original draft for brevity. Will write a longer post referencing all possible techniques, hopefully it won't turn into a review paper.