4 ms·

In the skip-gram algorithm (aka word-2-vec), words vectors are global; i.e one word maps to one vector. In practice, as you intuit, contextual embeddings are i

In the skip-gram algorithm (aka word-2-vec), words vectors are global; i.e one word maps to one vector. In practice, as you intuit, contextual embeddings are in fact useful for obtaining more precise word vectors. The example given in[0] is BERT's embeddings, though I would imagine that most NLP applications nowadays all use contextual embeddings since they're a strict improvement over static embeddings. [0]. https://web.stanford.edu/~jurafsky/slp3/6.pdf https://web.stanford.edu/~jurafsky/slp3/6.pdf (search for 'static embeddings')

rolisz 3y ago
Contextual embeddings are not a strict improvement over static embeddings: if you don't have context, they tend to be worse. For example, if you have tags for example, which usually are just single words, word2vec style embeddings are better usually.
- extasia 3y ago
  Do you mean html tags? I'm not sure I follow!
  - sp332 3y ago
    No, a tag like a hashtag.
  - chaxor 3y ago
    I think they're trying to say there is a good use case for everything. For instance, if you have a set of keyword-like IDs (normalized items like EntityID:47283 for example) for a container of these things, like documents, you can make a quick GLoVe or W2V representation for each entityID, and then the cosine similarity between an entity vector and itself is exactly 1, not 0.98 or something. So in the scenario that these are products which have IDs, and the container is a document that mentions these products together, you can reduce it to a key-value map of documentID-[list of product IDs], and then by selecting the vectors that have cosine similarity of .95-0.9999, you can get the object similar to that product but not that product. Using contextualized vectors, in the best case would give you things similar as well as the query product, but for every single instance, not just the single abstract entity.
  - rolisz 3y ago
    Think Instagram or Twitter tags.

rolisz 3y ago

Contextual embeddings are not a strict improvement over static embeddings: if you don't have context, they tend to be worse. For example, if you have tags for example, which usually are just single words, word2vec style embeddings are better usually.

extasia 3y ago
Do you mean html tags? I'm not sure I follow!
- sp332 3y ago
  No, a tag like a hashtag.
- chaxor 3y ago
  I think they're trying to say there is a good use case for everything. For instance, if you have a set of keyword-like IDs (normalized items like EntityID:47283 for example) for a container of these things, like documents, you can make a quick GLoVe or W2V representation for each entityID, and then the cosine similarity between an entity vector and itself is exactly 1, not 0.98 or something. So in the scenario that these are products which have IDs, and the container is a document that mentions these products together, you can reduce it to a key-value map of documentID-[list of product IDs], and then by selecting the vectors that have cosine similarity of .95-0.9999, you can get the object similar to that product but not that product. Using contextualized vectors, in the best case would give you things similar as well as the query product, but for every single instance, not just the single abstract entity.
- rolisz 3y ago
  Think Instagram or Twitter tags.