3 ms·

PostgreSQL is one of those tools I know I can always rely on for a new use-case. There are very few cases where it can't do exactly what I need (large scale vec

by TOMDM 5y ago

PostgreSQL is one of those tools I know I can always rely on for a new use-case. There are very few cases where it can't do exactly what I need (large scale vector search/retrieval). Congrats on the 14.0 release. The pace of open source has me wondering what we'll be seeing 50 years from now.

gk1 5y ago
I recall seeing some library that adds vector search to Postgres. Maybe https://github.com/ankane/pgvector https://github.com/ankane/pgvector? Also there's Pinecone (https://www.pinecone.io https://www.pinecone.io) which can sit alongside Postgres or any other data warehouse and ingest vector embeddings + metadata for vector search/retrieval.
- gavinray 5y ago
  ElasticSearch has "dense_vector" datatype and vector-specific functions. https://www.elastic.co/guide/en/elasticsearch/reference/current/dense-vector.html https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-script-score-query.html#vector-functions ZomboDB integrates ElasticSearch as a PG extension, written in Rust: https://github.com/zombodb/zombodb I dunno what exactly a "dense_vector" is, but if you can't use the native "tsvector" maybe you could use this?
  - anentropic 5y ago
    I think a dense vector is the opposite of a sparse vector i.e. in a dense vector every value in the vector is stored whereas sparse vectors exist to save space when you have large vectors where most of the values are usually zero - they reconstruct the full vector by storing only the non-zero values, plus their indices
    - gavinray 5y ago
      "a dense vector is the opposite of a sparse vector" I think there's another thing besides vectors that are a bit dense in the room here, eh? Yeah that makes sense hahaha -- thank you.
  - TOMDM 5y ago
    ZomboDB has always seemed really interesting to me. Elastics search, with PostgreSQL's everything else sounds like a perfect match, but I've never seen it used anywhere which is kind of spooky to me. Do you know of any non-toy use cases I can look at?
    - gavinray 5y ago
      You can ask Eric Ridge on the ZomboDB Discord, he's really nice: https://discord.gg/hPb93Y9 https://discord.gg/hPb93Y9 Also probably one of the most knowledgeable people about both Postgres and Rust I've ever met. He makes his living primarily from supporting enterprises using ZomboDB (I think), not sure how much he can say, but it's worth asking.
      - TOMDM 5y ago
        You're not kidding, they were incredibly friendly and helpful. Makes me wish I had a big deployment so I could justify sponsoring them. I paid in cat pictures for now.
- TOMDM 5y ago
  I need a self managed solution, so I'm not sure Pinecone is feasible and I don't think pgvector scales well enough for my use-case (hundreds of millions of vectors). So far I think I'm going to go with Milvus[1], ideally I'd just have a foreign data wrapper for Milvus or FAISS. [1] https://github.com/milvus-io/milvus https://github.com/milvus-io/milvus

by TOMDM 5y ago

gk1 5y ago

I recall seeing some library that adds vector search to Postgres. Maybe https://github.com/ankane/pgvector https://github.com/ankane/pgvector? Also there's Pinecone (https://www.pinecone.io https://www.pinecone.io) which can sit alongside Postgres or any other data warehouse and ingest vector embeddings + metadata for vector search/retrieval.

gavinray 5y ago
ElasticSearch has "dense_vector" datatype and vector-specific functions. https://www.elastic.co/guide/en/elasticsearch/reference/current/dense-vector.html https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-script-score-query.html#vector-functions ZomboDB integrates ElasticSearch as a PG extension, written in Rust: https://github.com/zombodb/zombodb I dunno what exactly a "dense_vector" is, but if you can't use the native "tsvector" maybe you could use this?
- anentropic 5y ago
  I think a dense vector is the opposite of a sparse vector i.e. in a dense vector every value in the vector is stored whereas sparse vectors exist to save space when you have large vectors where most of the values are usually zero - they reconstruct the full vector by storing only the non-zero values, plus their indices
  - gavinray 5y ago
    "a dense vector is the opposite of a sparse vector" I think there's another thing besides vectors that are a bit dense in the room here, eh? Yeah that makes sense hahaha -- thank you.
- TOMDM 5y ago
  ZomboDB has always seemed really interesting to me. Elastics search, with PostgreSQL's everything else sounds like a perfect match, but I've never seen it used anywhere which is kind of spooky to me. Do you know of any non-toy use cases I can look at?
  - gavinray 5y ago
    You can ask Eric Ridge on the ZomboDB Discord, he's really nice: https://discord.gg/hPb93Y9 https://discord.gg/hPb93Y9 Also probably one of the most knowledgeable people about both Postgres and Rust I've ever met. He makes his living primarily from supporting enterprises using ZomboDB (I think), not sure how much he can say, but it's worth asking.
    - TOMDM 5y ago
      You're not kidding, they were incredibly friendly and helpful. Makes me wish I had a big deployment so I could justify sponsoring them. I paid in cat pictures for now.
TOMDM 5y ago
I need a self managed solution, so I'm not sure Pinecone is feasible and I don't think pgvector scales well enough for my use-case (hundreds of millions of vectors). So far I think I'm going to go with Milvus[1], ideally I'd just have a foreign data wrapper for Milvus or FAISS. [1] https://github.com/milvus-io/milvus https://github.com/milvus-io/milvus