2 ms·
A few caveats here: - It works (that well) only for vision (for language it sort-of-works only at the word level: http://p.migdal.pl/2017/01/06/king-man-woman-
by stared 8y ago
A few caveats here:
- It works (that well) only for vision (for language it sort-of-works only at the word level: http://p.migdal.pl/2017/01/06/king-man-woman-queen-why.html http://p.migdal.pl/2017/01/06/king-man-woman-queen-why.html)
- "Do Better ImageNet Models Transfer Better?" https://arxiv.org/abs/1805.08974 https://arxiv.org/abs/1805.08974
And if you want to play with transfer learning, here is a tutorial with a working notebook: https://deepsense.ai/keras-vs-pytorch-avp-transfer-learning/ https://deepsense.ai/keras-vs-pytorch-avp-transfer-learning/
- mlucy 8y agoThere's actually been a lot of really good work recently around textual transfer learning. Google's BERT paper does sentence-level pretraining and transfer to get state of the art results on a bunch of problems: https://arxiv.org/pdf/1810.04805.pdf https://arxiv.org/pdf/1810.04805.pdf
- stared 8y agoThanks for this reference, I will look it up. Though, from my experience people in NLP still (be default) train from scratch, with some exceptions for tasks on the same dataset: - https://blog.openai.com/unsupervised-sentiment-neuron/ https://blog.openai.com/unsupervised-sentiment-neuron/ - http://ruder.io/nlp-imagenet/ http://ruder.io/nlp-imagenet/
- samcodes 8y agoThis is true, but rapidly changing. In addition to fine tuneable language models, you can do deep feature extraction with something like bert-as-service [0] ... You can even fine tune Bert on your days, then use the fine tuned model as a feature extractor. [0] https://github.com/hanxiao/bert-as-service https://github.com/hanxiao/bert-as-service