4 ms·

Embeddings result from computing what word can appear in a given context, so words that would appear in the same spot will have higher cosine score between them

Embeddings result from computing what word can appear in a given context, so words that would appear in the same spot will have higher cosine score between themselves. But it doesn't differentiate further, so you can have "beautiful" and "ugly" embed very close to each other even though they are opposites - they tend to appear in similar places. Another limitation of embeddings and cosine-similarity is that that they can't tell you "how similar" - is it equivalence or just relatedness? They make a mess of equivalent, antonymous and related things.

mo_42 3y ago
Only if those two words appear in the same contexts with the same frequency. In natural language this is probably not the case. There are things typically considered beautiful and others as ugly.
- singularity2001 3y ago
  one key insights is that opposites do have a lot in common, they are often opposites in exactly one of n feature dimensions. For example black and white are both (arguably) colors, are related to (lack of) light, have representations in various formats (RGB, …), appear in the same grammatical position of ordered adjectives …
_t89y 3y ago
They make a mess of language. They are not a suitable representation. They are suitable for their efficiency in information retrieval systems and for sometimes crudely capturing semantic attributes in a way that is unreliable and uninterpretable. It ends there. Here's to ten more years of word2vec.
minimaxir 3y ago
For word2vec-esque word embeddings, yes. For modern embedding models which effectively mean-pool the last hidden state of LLMs (and therefore make use of its optimizations such as attention tricks), embeddings can be much more robust to different contexts both local and global.
- LunaSea 3y ago
  Would you have some in links in mind of models that were pulled from LLMs? The last one I have in mind is BERT and it's variants.
  - minimaxir 3y ago
    BERT embeddings were what proved that taking embeddings using the last hidden state works (in that case, using the [CLS] token representation which is IMO silly). Most of the top embedding models on the MTEB leaderboard are mean-pooled LLMs: https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard The not-quite-large embedding model I like to use now is nomic-embed-text-v1.5 (based on a BERT architecture), which supports a 8192 context window and MRL for reducing the dimensionality if needed: https://huggingface.co/nomic-ai/nomic-embed-text-v1.5 https://huggingface.co/nomic-ai/nomic-embed-text-v1.5
    - _t89y 3y ago
      Works for what? The leaderboards? BPE tiktokens, BPE GPT-2 tokens, SentencePiece, GloVe, word2vec, ..., take your pick, they all end up in a latent space of arbitrary dimensionality and arbitrary vocab size where they can be mapped onto images. This is never going to work for language. The only thing the leaderboards are good for is enabling you to charge more for your model than everyone else for a month or two. The only meaning hyperparameters like dimensionality and vocab size have is in their message that more is always better and scaling up is what matters.
      - yorwba 3y ago
        Works for: Bitext mining: Given a sentence in one language, find its translation in a collection of sentences in another language using the cosine similarity of embeddings. Classification: identify the kind of text you're dealing with using logistic regression on the embeddings. Clustering: group similar texts together using k-means clustering on the embeddings. Pair Classification: determine whether two texts are paraphrases of each other by using a binary threshold on the cosine similarity of the embeddings. Reranking: given a query and a list of potential results, sort relevant results ahead of irrelevant ones by sorting according to the cosine similarity of embeddings. Etc etc. These are MTEB benchmark tasks https://arxiv.org/pdf/2210.07316.pdf https://arxiv.org/pdf/2210.07316.pdf . If you have no need for something like that, good for you, you don't need to care how well embeddings work for these tasks.
bongodongobob 3y ago
Isn't that kind of the point though? That beautiful and ugly are encoded closely as an "idea" when viewed from the appropriate angle?
- Sai_ 3y ago
  Apart from an ESL class explaining antonyms, I can’t think of any use case where an API which could equally return “ugly” or “beautiful” can be used. We need embeddings to give relatedness across axes like synonymity etc.
bruturis 3y ago
I think that yours comment is very interesting, I have reflected many times about how to differentiate things that appear in the same context of things that are similar. Any big idea here could be the spark to initiate a great startup.
carschno 3y ago
That is why computational linguistics prefer the term related over similar here. Similarity is notoriously hard to define, for starters in terms of grammatical vs semantically similarity.

mo_42 3y ago

Only if those two words appear in the same contexts with the same frequency. In natural language this is probably not the case. There are things typically considered beautiful and others as ugly.

singularity2001 3y ago
one key insights is that opposites do have a lot in common, they are often opposites in exactly one of n feature dimensions. For example black and white are both (arguably) colors, are related to (lack of) light, have representations in various formats (RGB, …), appear in the same grammatical position of ordered adjectives …

_t89y 3y ago

They make a mess of language. They are not a suitable representation. They are suitable for their efficiency in information retrieval systems and for sometimes crudely capturing semantic attributes in a way that is unreliable and uninterpretable. It ends there. Here's to ten more years of word2vec.

minimaxir 3y ago

For word2vec-esque word embeddings, yes. For modern embedding models which effectively mean-pool the last hidden state of LLMs (and therefore make use of its optimizations such as attention tricks), embeddings can be much more robust to different contexts both local and global.

LunaSea 3y ago
Would you have some in links in mind of models that were pulled from LLMs? The last one I have in mind is BERT and it's variants.
- minimaxir 3y ago
  BERT embeddings were what proved that taking embeddings using the last hidden state works (in that case, using the [CLS] token representation which is IMO silly). Most of the top embedding models on the MTEB leaderboard are mean-pooled LLMs: https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard The not-quite-large embedding model I like to use now is nomic-embed-text-v1.5 (based on a BERT architecture), which supports a 8192 context window and MRL for reducing the dimensionality if needed: https://huggingface.co/nomic-ai/nomic-embed-text-v1.5 https://huggingface.co/nomic-ai/nomic-embed-text-v1.5
  - _t89y 3y ago
    Works for what? The leaderboards? BPE tiktokens, BPE GPT-2 tokens, SentencePiece, GloVe, word2vec, ..., take your pick, they all end up in a latent space of arbitrary dimensionality and arbitrary vocab size where they can be mapped onto images. This is never going to work for language. The only thing the leaderboards are good for is enabling you to charge more for your model than everyone else for a month or two. The only meaning hyperparameters like dimensionality and vocab size have is in their message that more is always better and scaling up is what matters.
    - yorwba 3y ago
      Works for: Bitext mining: Given a sentence in one language, find its translation in a collection of sentences in another language using the cosine similarity of embeddings. Classification: identify the kind of text you're dealing with using logistic regression on the embeddings. Clustering: group similar texts together using k-means clustering on the embeddings. Pair Classification: determine whether two texts are paraphrases of each other by using a binary threshold on the cosine similarity of the embeddings. Reranking: given a query and a list of potential results, sort relevant results ahead of irrelevant ones by sorting according to the cosine similarity of embeddings. Etc etc. These are MTEB benchmark tasks https://arxiv.org/pdf/2210.07316.pdf https://arxiv.org/pdf/2210.07316.pdf . If you have no need for something like that, good for you, you don't need to care how well embeddings work for these tasks.

bongodongobob 3y ago

Isn't that kind of the point though? That beautiful and ugly are encoded closely as an "idea" when viewed from the appropriate angle?

Sai_ 3y ago
Apart from an ESL class explaining antonyms, I can’t think of any use case where an API which could equally return “ugly” or “beautiful” can be used. We need embeddings to give relatedness across axes like synonymity etc.

bruturis 3y ago

I think that yours comment is very interesting, I have reflected many times about how to differentiate things that appear in the same context of things that are similar. Any big idea here could be the spark to initiate a great startup.

carschno 3y ago

That is why computational linguistics prefer the term related over similar here. Similarity is notoriously hard to define, for starters in terms of grammatical vs semantically similarity.