3 ms·
To compare with the MTEB leaderboard (https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard), the new embedding models a
by minimaxir 3y ago
To compare with the MTEB leaderboard (https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard), the new embedding models are on par with open-source embedding models like BAAI/bge-large-en-v1.5, not a drastic improvement if already using them. Obviously, a cost/performance improvement is still good.
I've found evidence that the OpenAI 1536D embeddings are unnecessairly big for 99% of use cases (and now there's a 3072D model?!) so the ability to reduce dimensionality directly from the API is appreciated for the reasons given in this post. Just chopping off dimensions to an arbitrary dimensionality is not a typical dimensionality reduction technique so that likely requires a special training/alignment technique that's novel.
EDIT: Tested the API: it does support reducing to an arbitrary number of dimensions other than the ones noted into the post. (even 2D for data viz, but may not be as useful since the embeddings are normalized)
The embeddings aren't "chopped off", the first components of the embedding will change as dimensionality reduces, but not much.
- lpasselin 3y agoMost of the leaderboard has much lower sequence length.
- deleted 3y ago[deleted]
- ldjkfkdsjnv 3y agoThese benchmarks never tell the full story. Anyone thats using models for real production use cases with complex ai requirements knows OpenAI is still king. GPT4 in practice is leagues ahead, regardless of what leaderboards show.
- teaearlgraycold 3y agoSubjectively, OpenAI hasn’t been a leader in the embedding model space the way they have been with text completions.
- ldjkfkdsjnv 3y agoYour use case isnt complex/nuanced enough then
- teaearlgraycold 3y agoCould easily be true. It's harder to judge embeddings quality.
- moralestapia 3y agoHmm ... nah. GPT4 is still king but text-embedding-ada-002 is quite bad.
- phantastic 3y agowhat's been your experience with different open source embedding models vs ada?
- moralestapia 3y ago>I've found evidence that the OpenAI 1536D embeddings are unnecessairly big for 99% Agree, I tried a few 384-dim models and they perform 95%+ as good, def. not worth the extra space.
- summarity 3y ago> dimensions to an arbitrary dimensionality is not a typical dimensionality reduction technique so that likely requires a special training/alignment technique that's novel Very basic techniques (e.g. SuperBit random projection) have been extremely effective with OpenAI embeddings in the past. E.g. all embeddings on findsight.ai are OpenAI Ada embeddings stored as SuperBit signatures with a code length of 10,000 (i.e. 157 integers each), and there's almost no recall loss compared to the full vectors.
- brigadier132 3y agoHow low in dimensions have you been able to go without significant recall loss?
- osmarks 3y agohttps://github.com/facebookresearch/faiss/wiki/Vector-codecs https://github.com/facebookresearch/faiss/wiki/Vector-codecs has some good things available too.
- huac 3y agodo you construct an index over the superbit signatures to perform approximate search or do you perform exact search?
- summarity 3y agoParallelised exact search, which is good enough because calculating the hamming similarity is just an XOR+POPCNT per vector component and an addition. But of course you could put this into an HNSW graph for approximate search for >10M vectors. Or do LSH first for even larger data sets.
- sjkoelle 3y agohow does this compare with PCA?
- huac 3y agocurious what the latency impact is to apply the dimensionality reduction... may hint at the specific technique they use