4 ms·
Pgvector: Fewer Dimensions Are Better
- kiwicopple 3y agowe were lucky to discover the new gte-small[0] model while writing this, which is so new that it doesn't have a lot of documentation around it. The neat thing about this model is that it out-performs OpenAI's `text-embedding-ada-002` on Hugging Face's MTEB Leaderboard[1] and it's only 384 dimensions. This is great for Postgres & pgvector - it means less space and faster queries. There are some trade-offs (it was trained on English-only text), but this is a good indicator of the direction where embeddings might go (code-specific, language-specific, etc) [0] https://huggingface.co/Supabase/gte-small https://huggingface.co/Supabase/gte-small [1] MTEB leaderboard: https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard
- jstx1 3y agoOpenAI's embeddings don't add that much. For similarity search you can use a number of open source models from huggingface with small embeddings sizes and do as well or better than OpenAI’s embeddings.
- kiwicopple 3y agoagreed, and this is the other nice thing. You can use any embedding system for search and retrieval, and then use OpenAI/GPT-3.5/GPT4 for generative use cases (which is still miles ahead for many functions) fwiw, 98% of Supabase project use OpenAI for embeddings, so a lot of this content is just to help devs identify the best options they have available
- ZephyrBlu 3y agoI'm guessing most people use OpenAI embeddings because it's very convenient to use an API instead of running the model yourself, and they perform well enough that people aren't actively looking for other models.
- jclem 3y agoIt seems worth noting that the input of gte-small is limited to 512 tokens. The tokenizers for sure aren’t the same, but I imagine this is significantly less than ada-002, whose input is limited to 8191 tokens. That said, I don’t imagine that embedding huge full documents is necessarily the right approach. I would love to see a comparison for some typical use cases using various methods of chunking input documents.
- egorr 3y agoshould be yes, but even in examples from openai, they usually do splitting into chunks For example in chatgpt-retrieval-plugin[0] repo default chunk size is just 200 tokens this is anyway a limitation, no doubt, but chunking is pretty often used [0] https://github.com/openai/chatgpt-retrieval-plugin/blob/main/services/chunks.py https://github.com/openai/chatgpt-retrieval-plugin/blob/main...
- jonathan-adly 3y agoOh - that's a nice model. Can probably run with 0 issues on your low-end linux CPU machine.
- creata 3y ago> and it's only 384 dimensions And the model is only 70MB!
- egorr 3y agoheya, supabase engineer here, i coauthored the part about benchmarking different models with pgvector to compare QPS when using fewer dims models. btw, you can find the dataset with embeddings generated by all 3 mentioned: text-embedding-ada-002, all-MiniLM-L6-v2, and GTE-small on huggingface[0] and big thanks to Stephan Sturges for his dataset[1]. we just extended his OpenAI ones and texts themselves with oss ones [0] https://huggingface.co/datasets/Supabase/wikipedia-en-embeddings https://huggingface.co/datasets/Supabase/wikipedia-en-embedd... [1] https://www.kaggle.com/datasets/stephanst/wikipedia-simple-openai-embeddings https://www.kaggle.com/datasets/stephanst/wikipedia-simple-o...
- roseway4 3y agoKismet this is published today. We've just announced Zep's[0] new Document Vector Store[1] built on pgvector, and the embedding model we ship with is all-MiniLM-L6-v2. Very fast, low-memory, and surprisingly performant. Our focus with Zep is on LLM app use cases, and in particular, searching over chat histories and documents for RAG apps. You can, however, use Zep to turn any Postgres instance into a vector store with great developer experience. pgvector index configuration and query tuning can be challenging. We've tried to do much of this work for developers. Also, I'm super excited about the prospect of HNSW support in pgvector, slated for pgvector 0.5.[2] [0] https://github.com/getzep/zep https://github.com/getzep/zep [1] https://blog.getzep.com/introducing-the-zep-document-vector-store/ https://blog.getzep.com/introducing-the-zep-document-vector-... [2] https://github.com/pgvector/pgvector/issues/181#issuecomment-1663355614 https://github.com/pgvector/pgvector/issues/181#issuecomment...
- jasfi 3y agoSo, better performance at a lower cost. But what about quality?
- kiwicopple 3y agothe gte-small model actually outperforms OpenAI's `text-embedding-ada-002` on the MTEB leaderboard: https://huggingface.co/spaces/mteb/leaderboard https://huggingface.co/spaces/mteb/leaderboard The highest ranking is `gte-large`, which has 1024 dimensions - larger than `gte-small`(384), but still smaller than `text-embedding-ada-002` (1536)