3 ms·

I was using pinecone before installing pgvector in Postgres. Pinecone works and all but having the vectors in Postgres resulted in an explosion of use for us.

I was using pinecone before installing pgvector in Postgres. Pinecone works and all but having the vectors in Postgres resulted in an explosion of use for us. Full relational queries with where clauses and order by etc AND vector embeddings is wicked.

javier2 3y ago
How is disk usage in pgvector?
- arecurrence 3y ago
  There's a section in the readme describing vector space https://github.com/pgvector https://github.com/pgvector. . Also, the maintainer is actively working on product quantization https://github.com/pgvector/pgvector/issues/93 https://github.com/pgvector/pgvector/issues/93 Supabase was also asking for sparse vectors https://github.com/pgvector/pgvector/issues/81 https://github.com/pgvector/pgvector/issues/81 Speaking of the repo, they have a number of features they want to add if anyone is interested in contributing, there's lots of room for advancement. Many of these features already have active branches https://github.com/pgvector/pgvector/issues/27 https://github.com/pgvector/pgvector/issues/27
itake 3y ago
Why do you use pgvector instead of pgANN? My understanding is pgANN is built with FAISS. When I compared pgvector with FAISS, pgvector was 3-5x slower. https://github.com/netrasys/pgANN https://github.com/netrasys/pgANN
- extesy 3y ago
  If I understand correctly, pgAnn is using a standard "cube" extension: https://github.com/netrasys/pgANN#setup https://github.com/netrasys/pgANN#setup
- arecurrence 3y ago
  There is certainly a wide variety of problems today for which pgvector is unsuitable due to performance limitations... but fear not! This is an area that is getting significant focus right now. A marqo.ai dev is currently working on adding HNSW-IVF and HNSW support https://news.ycombinator.com/item?id=35551684 https://news.ycombinator.com/item?id=35551684 and the maintainer has recently noted that they are actively working on an IVFPQ/ScaNN implementation https://github.com/pgvector/pgvector/issues/93 https://github.com/pgvector/pgvector/issues/93 The pgAnn creator actually asked about performance a month ago here https://github.com/pgvector/pgvector/issues/58 https://github.com/pgvector/pgvector/issues/58 Expect to see performance improve dramatically later this year.
sv123 3y ago
This hits home, it is a big ask to keep data in sync for yet another store. We already balance MS SQL and Algolia and all the plumbing required to catch updates, deletes, etc. adding another feels like a bridge too far. Hopefully MS will get on this train at some point and catch up to postgres.
Freebytes 3y ago
Do you know of anything similar available for SQL Server 2019?
neximo64 3y ago
What does it do is it just storing vectors and cosine similarity? Why is there a whole new category of db for that

javier2 3y ago

How is disk usage in pgvector?

arecurrence 3y ago
There's a section in the readme describing vector space https://github.com/pgvector https://github.com/pgvector. . Also, the maintainer is actively working on product quantization https://github.com/pgvector/pgvector/issues/93 https://github.com/pgvector/pgvector/issues/93 Supabase was also asking for sparse vectors https://github.com/pgvector/pgvector/issues/81 https://github.com/pgvector/pgvector/issues/81 Speaking of the repo, they have a number of features they want to add if anyone is interested in contributing, there's lots of room for advancement. Many of these features already have active branches https://github.com/pgvector/pgvector/issues/27 https://github.com/pgvector/pgvector/issues/27

itake 3y ago

Why do you use pgvector instead of pgANN? My understanding is pgANN is built with FAISS. When I compared pgvector with FAISS, pgvector was 3-5x slower. https://github.com/netrasys/pgANN https://github.com/netrasys/pgANN

extesy 3y ago
If I understand correctly, pgAnn is using a standard "cube" extension: https://github.com/netrasys/pgANN#setup https://github.com/netrasys/pgANN#setup
arecurrence 3y ago
There is certainly a wide variety of problems today for which pgvector is unsuitable due to performance limitations... but fear not! This is an area that is getting significant focus right now. A marqo.ai dev is currently working on adding HNSW-IVF and HNSW support https://news.ycombinator.com/item?id=35551684 https://news.ycombinator.com/item?id=35551684 and the maintainer has recently noted that they are actively working on an IVFPQ/ScaNN implementation https://github.com/pgvector/pgvector/issues/93 https://github.com/pgvector/pgvector/issues/93 The pgAnn creator actually asked about performance a month ago here https://github.com/pgvector/pgvector/issues/58 https://github.com/pgvector/pgvector/issues/58 Expect to see performance improve dramatically later this year.

sv123 3y ago

This hits home, it is a big ask to keep data in sync for yet another store. We already balance MS SQL and Algolia and all the plumbing required to catch updates, deletes, etc. adding another feels like a bridge too far. Hopefully MS will get on this train at some point and catch up to postgres.

Freebytes 3y ago

Do you know of anything similar available for SQL Server 2019?

neximo64 3y ago

What does it do is it just storing vectors and cosine similarity? Why is there a whole new category of db for that