4 ms·
Also: * OpenAI GPT adapted idea of fine-tuning of language model for specific NLP task, which has been introduced in ELMo model. * BERT created bigger model (
by riku_iki 8y ago
Also:
* OpenAI GPT adapted idea of fine-tuning of language model for specific NLP task, which has been introduced in ELMo model.
* BERT created bigger model (16 layers in GPT vs 24 layers in BERT), proving that larger Transformer models increase performance
- solomatov 8y ago>OpenAI GPT adapted idea of fine-tuning of language model for specific NLP task, which has been introduced in ELMo model. Idea of transfer learning of deep representations for NLP tasks was before, but nobody was able to achieve it before ELMo. If we are pedantic we can include the whole word2vec stuff. It's a shallow transfer learning.
- riku_iki 8y agoYeah, but in case of ELMo it was fine-tuning (training of pretrained language model and task model together), not just transfer learning.
- phowon 8y agoWith ELMo, the pretrained weights are frozen. Only the scalars for ELMo layers are tuned (as well as the additional top-level model, of course).
- phowon 8y agoThe BERT paper also introduced BERT Base, with is 12 layers with approximately the same number of parameters as GPT, but still outperforms GPT on GLUE.