That's a really good idea. It would test how much semantic information the word vector actually contains about the word. Another idea is to generate word vecto

That's a really good idea. It would test how much semantic information the word vector actually contains about the word. Another idea is to generate word vectors from a dictionary. Instead of trying to infer it's meaning from context, a dictionary gives you it's meaning directly. And even alternate meanings. Or a hybrid approach might be best. I recently read a paper where they used wordnet relationships to improve regular word2vec vectors a lot.