Thank you Sebastian. Keep up the great work ! You will note that negative sampling improved by leveraging information on word pairs form dictionaries entries

Thank you Sebastian. Keep up the great work ! You will note that negative sampling improved by leveraging information on word pairs form dictionaries entries (we called it "controlled negative sampling") do help, though not much. It actually really depends in the rare words rate (see section 5.4, improvement ranges from 0.7% up to 10%). But I guess it is already an interesting, somehow counter-intuitive, observation. Another very interesting observation is that you can also choose to just clamp a general purpose dataset and expand it with external contextual information (meaning not using is for supervision but rather just collapse it at the end of the training corpus in a raw form [^]). In our case, we call those corpus : - corpus A : plain old wikipedia dump - corpus B : plain old wikipedia dump + dictionaries text collapse at the end of it. It sounds a bit naive : the latter part of the training corpus is really small w.r.t. the full wikipedia dump. Nonetheless, it has an significant impact on word similarity (see improvement in Table 2 to see how those training corpus influences representations learnt by word2vec, fasttext and dict2vec). (Related to the effect of the training corpus : https://arxiv.org/pdf/1507.05523v1.pdf https://arxiv.org/pdf/1507.05523v1.pdf) I mention this effect of training corpus content here since it sounds like an interesting info for the working natural language processing practitioners (get a mid size general training corpus, add as many contextual corpus as possible => may yield useful embeddings...). [^] to be entirely fair, this has been suggested to us by an anonymous reviewer, many thanks for him/her for pointing this out : I found the results surprising.