I’m not convinced by this paper. I’ve trained a lot of from-scratch word, sentence and query embeddings in my career, it’s probably the single main thing I’ve d

I’m not convinced by this paper. I’ve trained a lot of from-scratch word, sentence and query embeddings in my career, it’s probably the single main thing I’ve done. I’ve never observed rescaling the average context vector to have an impact on application performance. It amounts to rescaling gradient terms, but most of those are being backprop’d from layers with batch normalization, strict activation functions, clipping, etc. There are many, many non-linear effects contributing to how that rescaling constant plays a role, and in anything other a completely shallow word2vec model with no further layers and where you just want to extract the embeddings in some application-agnostic way, that normalizing constant is not going to matter.