4 ms·
There's actually been a lot of really good work recently around textual transfer learning. Google's BERT paper does sentence-level pretraining and transfer to
by mlucy 8y ago
There's actually been a lot of really good work recently around textual transfer learning. Google's BERT paper does sentence-level pretraining and transfer to get state of the art results on a bunch of problems: https://arxiv.org/pdf/1810.04805.pdf https://arxiv.org/pdf/1810.04805.pdf
- stared 8y agoThanks for this reference, I will look it up. Though, from my experience people in NLP still (be default) train from scratch, with some exceptions for tasks on the same dataset: - https://blog.openai.com/unsupervised-sentiment-neuron/ https://blog.openai.com/unsupervised-sentiment-neuron/ - http://ruder.io/nlp-imagenet/ http://ruder.io/nlp-imagenet/
- samcodes 8y agoThis is true, but rapidly changing. In addition to fine tuneable language models, you can do deep feature extraction with something like bert-as-service [0] ... You can even fine tune Bert on your days, then use the fine tuned model as a feature extractor. [0] https://github.com/hanxiao/bert-as-service https://github.com/hanxiao/bert-as-service