4 ms·

What Ive wondered about word-vectors is, are different meanings split or not? A word having many meanings would otherwise means its vectors are more "smeared"

• extasia 3y ago
  In the skip-gram algorithm (aka word-2-vec), words vectors are global; i.e one word maps to one vector. In practice, as you intuit, contextual embeddings are in fact useful for obtaining more precise word vectors. The example given in[0] is BERT's embeddings, though I would imagine that most NLP applications nowadays all use contextual embeddings since they're a strict improvement over static embeddings. [0]. https://web.stanford.edu/~jurafsky/slp3/6.pdf https://web.stanford.edu/~jurafsky/slp3/6.pdf (search for 'static embeddings')
  • rolisz 3y ago
    Contextual embeddings are not a strict improvement over static embeddings: if you don't have context, they tend to be worse. For example, if you have tags for example, which usually are just single words, word2vec style embeddings are better usually.
    • extasia 3y ago
      Do you mean html tags? I'm not sure I follow!
      • sp332 3y ago
        No, a tag like a hashtag.
      • chaxor 3y ago
        I think they're trying to say there is a good use case for everything. For instance, if you have a set of keyword-like IDs (normalized items like EntityID:47283 for example) for a container of these things, like documents, you can make a quick GLoVe or W2V representation for each entityID, and then the cosine similarity between an entity vector and itself is exactly 1, not 0.98 or something. So in the scenario that these are products which have IDs, and the container is a document that mentions these products together, you can reduce it to a key-value map of documentID-[list of product IDs], and then by selecting the vectors that have cosine similarity of .95-0.9999, you can get the object similar to that product but not that product. Using contextualized vectors, in the best case would give you things similar as well as the query product, but for every single instance, not just the single abstract entity.
      • rolisz 3y ago
        Think Instagram or Twitter tags.
• VHRanger 3y ago
  That mainly depends on the number of dimensions on the model, and if the training picked up split meaning. With a high dimension vector (>50d, more usually 300d or 500d) one or two dimensions together will pick up a concept. This is why word algebra like Paris - France + Russia ~ Moscow works with original Word2Vec vectors. Language models are better at picking up context-dependent meaning, but word vectors still have good enough performance for many tasks.
• low_tech_love 3y ago
  No, that’s why you need attention. Some might say that’s all you need…