Initially embeddings of OOV terms are initialized to mean embeddings, and then the language model fine-tuning step allows the model to adjust these values. See

Initially embeddings of OOV terms are initialized to mean embeddings, and then the language model fine-tuning step allows the model to adjust these values. See [18] here: https://github.com/fastai/fastai/blob/master/courses/dl2/imdb.ipynb https://github.com/fastai/fastai/blob/master/courses/dl2/imd....