9 ms·
20x faster than pgvector: HNSW index in Postgres with pg_embedding
- nikita 3y agoCEO of Neon here. This was a relatively quick project for us and the index is currently in-memory. However it is fast! We would love your feedback and excited to invest further.
- zorgmonkey 3y agoLooks like a cool project, but it would be great if you could add a license to the project since under US copyright lack of a license implies all rights are reserved and not open source. If you don't know much about software licenses https://choosealicense.com/ https://choosealicense.com/ is a helpful site.
- tristan957 3y agoI have alerted some of my colleagues to add a license. Thanks for pointing this out.
- raoufchebri 3y agoBlog post author here. Let me know if you have any questions regarding the benchmark and pg_embedding
- lysecret 3y agoIs anyone having a really good experience using embedding based semantic retrieval in combination with a downstream LLM ? I am working quite a bit with normal and chained LLMs but so far haven't explored the retrieval route.
- itake 3y agocan you provide an example?
- raoufchebri 3y agoHey @lysecret, what's your use-case?
- azmodeus 3y agoI have enjoyed using haystack for multi doc qa
- jonathan-adly 3y agoYeb - we use pgvector in production[0]. Great results. There are some secret sauce stuff, but basically - depending on your source material and your user use-case you have to do more than just call cosine similarity and call it a day. (Most people don't even know what's happening, just using langchain which has different implementations, but all essentially call cosine similarity on raw text). In my opinion, until LLMs learn to say I don't know, this is the way to do it in accuracy-first tasks. 0. https://galenai.co https://galenai.co
- raoufchebri 3y ago@jonathan-adly do you use ivfflat with pgvector?
- chrisrickard 3y agoOut of curiosity, how big are you embedding chunks?
- vikp 3y agoYes, I use retrieval for Endless Academy [1] , and it works well. Some tips: - Most vector search is basically kNN under the hood, with some kind of compression. If you have too many embeddings in your DB, this starts to pull up irrelevant text very quickly. The key is to segment the DB using other data before doing the embedding search. Postgres extensions are good for this. - The quality of the data you put into your embedding DB matters a lot. - How you chunk text matters. Chunking by paragraph is much better than naive chunking, for example. - This is a good benchmark for embedding models [2] [1] https://www.endless.academy https://www.endless.academy [2] https://huggingface.co/blog/mteb https://huggingface.co/blog/mteb
- raoufchebri 3y agoAdding https://ann-benchmarks.com/ https://ann-benchmarks.com/
- chrisrickard 3y ago> The key is to segment the DB using other data before doing the embedding search By segmenting data, is this as simple as adding a further condition to the SQL? E.g. if your embeddings belonged to a "client" you would start with WHERE client_id - x? > How you chunk text matters. Chunking by paragraph is much better than naive chunking Would love some more info on this. SO if you had a 400 word slab of text, it would be better to create m3 or 4 embeddings from that?
- alach11 3y agoYes. I work in the oil and gas industry. When our employees do their pre-job safety meeting, we have an app that makes tailored recommendations about job hazards. The steps are: 1. User types in the planned work (e.g., "replace discharge line on water pump") 2. The app embeds that and look for similar historical work in a vector database 3. We retrieve the most similar historical work and the safety observations that were made on those jobs 4. We summarize the major hazards or mistakes from those previous jobs using gpt-3.5-turbo and deliver that to the user
- victorbjorklund 3y agoI used pg_vector + Open AI api. Worked great!
- deleted 3y ago[deleted]
- jeffchuber 3y agoIs it open source?
- mattashii 3y agoI believe it's available here: https://github.com/neondatabase/pg_embedding/ https://github.com/neondatabase/pg_embedding/ It doesn't yet have a LICENSE file, so it's not strictly OS quite yet, but that might have been an oversight.
- tristan957 3y agoI have alerted people to add one. Thanks for pointing it out!
- nikita 3y agohttps://github.com/neondatabase/pg_embedding/blob/main/LICENSE https://github.com/neondatabase/pg_embedding/blob/main/LICEN...
- jeffchuber 3y agowhat license is that? seems atypical?
- lawrencechen 3y agohttps://www.postgresql.org/about/licence/ https://www.postgresql.org/about/licence/
- vore 3y agoIt's the Postgres license: https://www.postgresql.org/docs/current/legalnotice.html https://www.postgresql.org/docs/current/legalnotice.html
- jeffchuber 3y agothanks!
- jamesmcintyre 3y agoThis is cool! A few questions: - Given how neon architecture decouples compute with storage using the safekeepers & pageservers will this still just work with neon? (Wondering because you mention the index is in-memory so unless your stateless compute nodes can somehow hot-swap the indexes I wasn't sure how that'd work?) - Do you plan to offer vector search as a plug-and-play offering? If so does neon as a product offering plan to introduce more "out-of-the-box" functionalities like vector search? (Similar to xata's offerings like search & vector search.) - unrelated question- I believe neon has very small cold startups for free/scale-to-zero configurations but are there also inherent small latencies for infrequently accessed data? In other words is it safe to say if there were a large table with records that are "old"/"archival" but also always a sort of ~last 30-days of records that are "fresh"/"accessed with more frequency" would there more likely be slight latency introduced when accessing the older records? Neon looks awesome and thanks for neons open source contributions!
- nikita 3y agoGreat set of questions: - This does work with Neon today, please go ahead and try. Today the index is in-memory and we are working on the on-disk implementation. Neon architecture does make it trickier - no local disk, so we are actively experimenting as we speak. Anything we ship will of course work on Neon. - We will start with vector ANN and see the reception and user feedback. Now that we have 100K databases we are super attuned to what our users tell us. - There is a cache/storage hierarchy: local cache on compute, cache on pageserver, and S3. We are working on cold start (stay tuned for more announcements this week). The goal is to make this unnoticeable for the app.
- zinclozenge 3y agoWould be great to also add the ability to calculate the tanimoto coefficient for cheminformatics purposes.
- nikita 3y agoWould love to work with you on that! Please dm to https://twitter.com/neondatabase https://twitter.com/neondatabase
- nikita 3y agoCan you say more about the use case?
- zinclozenge 3y agoYea, I mean basically there are a variety of techniques for encoding molecules as vectors. Morgan/circular fingerprints (cf https://darkomedin-datascience.medium.com/data-science-for-drug-discovery-research-morgan-fingerprints-using-alanine-and-testosterone-92a2c69dd765 https://darkomedin-datascience.medium.com/data-science-for-d... as a quick google search) are one way of encoding to bit vectors, and the Tanimoto similarity is a well validated measure of similarity (because of the bit vectors, cf https://en.wikipedia.org/wiki/Chemical_similarity https://en.wikipedia.org/wiki/Chemical_similarity). Virtual screens of molecules frequently use steps to filter out molecules, one of which is via chemical similarity.
- nikita 3y agoCan our team reach out? please email nikita@neon.tech and we will help you out.
- kiwicopple 3y agoAny reason you didn’t contribute to pgvector? It would have been nice to get the support of neon in progressing pgvector - since it’s already so widely adopted by the community (disclosure: supabase ceo)
- nikita 3y agoI think IVFFlat and HNSW are very different implementations. So we would be forcing a choice for a pgvector user of what index to use. Our on disk path will require us to run a number of experiments with the storage format. We will move faster with a separate project. Medium to long term we are very open to merge them.
- kiwicopple 3y agoThey are different, but that doesn't preclude pgvector supporting both indexes. I get that it's slower to take a collaborative approach - but that's part of open source. > will require us to run a number of experiments with the storage format I'd personally consider this a feature, if you want to maintain Postgres compatibility and allow your users to migrate in/out easily
- nikita 3y agohttps://github.com/pgvector/pgvector/pull/86 https://github.com/pgvector/pgvector/pull/86 an example of why it's better to have our own project to move fast.
- KRAKRISMOTT 3y agoA more important question is why supabase went with one of the slowest [0] implementations instead of building a better plugin especially since you are a VC funded company that's hardly lacking in cash. Simply gluing together a few random open source extensions is not great for developer experience. [0] https://github.com/erikbern/ann-benchmarks https://github.com/erikbern/ann-benchmarks
- kiwicopple 3y agothat's a fair question. it's our policy to support existing open source tools/companies as much as we can[0] Andrew is working hard on pgvector, and his goals are aligned to ours (and the industry's). We'll continue to support him as long as that's true. > one of the slowest Benchmarks are simply a snapshot in time - there's nothing fundamentally flawed with pgvector, it just needs more support and some better indexes (which are already being worked on) [0] https://supabase.com/docs/guides/getting-started/architecture#support-existing-tools https://supabase.com/docs/guides/getting-started/architectur...
- yunyu 3y agoWhat's the plan/timeline for offering cosine similarity support, given that most OSS embedding models are fine tuned on a contrastive cosine distance objective?
- nikita 3y agoHappy to work with you on that!
- jasfi 3y agoI was wondering how this compared to Qdrant. I found this: "Qdrant currently only uses HNSW as a vector index." - https://qdrant.tech/documentation/concepts/indexing https://qdrant.tech/documentation/concepts/indexing So it would be interesting to see benchmarks between pg_embedding and Qdrant. I would expect them to perform similarly, but perhaps there are other factors?
- raoufchebri 3y agoI think you can find Qdrant here https://ann-benchmarks.com/glove-100-angular_10_angular.html https://ann-benchmarks.com/glove-100-angular_10_angular.html This is different from the dataset we used. We used GIST-960. It could interesting to test different vector store indeed.
- carlsverre 3y agoVery cool! It would be nice to see a working end to end integration with an LLM. Using this to generate relevant context for example. I see multiple folks mention cos similarity which HNSW doesn’t support - how does the lack of that limit what you can do with this library? Also, since this is in memory, I assume this significantly affects startup time in order to rebuild the index? Would be nice to see how bad that is for larger vector datasets.
- akorotkov 3y agoThis blog post makes me uneasy. The pg_embedding code on github gives impression of PoC, while blog post creates impression that pg_embedding is ready for use. If we consider pg_embedding ready for use, why don't you compare with pgvector: 1) The need to rebuild index every instance restart, 2) Replication support?
- kelvich 3y agoThank you, point taken. The blog post does mention memory limitations, but we could elaborate more on the fact that the index is entirely in-memory and clarify the user-visible consequences of this fact. We'll edit the post for better clarity.
- akorotkov 3y agoOk. I have to also mentions that none of the issues I've mentioned are consequences of in-memory nature of index. Both are issues of implementation.
- chrisrickard 3y agoWith people talking about pgvectors current scaling issues, one thing I'm not sure about is whether it's if the Postgres DB table simply contains a lot of vectors - e.g. 500k vectors - or if you are searching over 500k vectors? E.g. If the DB table had 500k vectors, but you were pre-filtering by WHERE client_id = X (returning only 200 rows) then with an AND <embedding search> (returning only 6 rows) - would this still have the same performance issue? Or is it literally if the embedding search is over 500k rows?