4 ms·
professionals don't use pretrained word2vec vectors in the really complex (like neural machine transnation) deep learning models anymore, they let the models tr
by make3 9y ago
professionals don't use pretrained word2vec vectors in the really complex (like neural machine transnation) deep learning models anymore, they let the models train their own word embeddings directly, or let the models learn character level embeddings.
- arrmn 9y agoWhat exactly do you mean with "they let the models train their own word embeddings", can you elaborate more on this or are there any current papers about this topic?
- make3 9y agothe embedding layer is the layer that converts the one hot word feature in to a continuous multi dimensional vector that the deep net can learn with. they used to pretrain that layer separately with word2vec. now as it's just a neural net layer, they let the translation model train it with backprop on the main (translation /dialog / qa, etc) task as a regular layer
- dontreact 9y agoThis is false. Sometimes you have limited data for your task and initializing from pretrained embeddings boosts performance.
- oh-kumudo 9y agoI would say it depends on your task, if pre-trained ones performs better, you use pre-trained ones, otherwise you let the model learns the embedding itself.