4 ms·

We embed Postgres and pgvector in Zep[0], a memory store for LLM apps. It’s rock solid, even with very wide vectors, and with the right indexes ANN is performan

We embed Postgres and pgvector in Zep[0], a memory store for LLM apps. It’s rock solid, even with very wide vectors, and with the right indexes ANN is performant up to tens of millions of rows. Haven’t tested beyond this, though. It’s super useful for hybrid search: vector similarity search over unstructured data joined with filtering by structured data. One caveat: It’s important indexes are added after a fair number of vectors have been persisted, which can make some production use cases challenging to implement. [0] https://github.com/getzep/zep https://github.com/getzep/zep

brigadier132 3y ago
When you say you can do vss along with filtering. Does the filtering happen before or after the vss?
- roseway4 3y ago
  Logically at the same time in the sql query. This may not be how the query is executed by the database, though, and is up to the optimizer.
kiwicopple 3y ago
> It’s important indexes are added after a fair number of vectors have been persisted, which can make some production use cases challenging to implement. managing indexes are the most challenging part of using pgvector right now. There is a bit of a "goldilocks zone" - you can't create the index with zero data, but if you create it too late then it can take a lot of memory. Andrew (and a few others, like AWS) are working hard on improving this, as well as researching other index types.
- deepsquirrelnet 3y ago
  That’s good to know. I tested it in a docker container and indexed after loading ~1M vectors. Checking EXPLAIN showed it wasn’t using the index at all. I wonder if it’s because it couldn’t load into memory. Without an index, I found it was too slow for data that size. I didn’t expect issues for data at this “small”, but it’s pretty new to me.
sodiumSalt 3y ago
Zep seems really cool! I think hybrid search is a great paradigm and believe it will continue to grow. How do you guys plan on dealing with data deletion requests and other GDPR rules around storage?
- roseway4 3y ago
  Great question. We have a deletion endpoint that allows sessions (think users) to be deleted. All memory artifacts are timestamped, and we run soft-deletes with deleted_at timestamps. This allows enterprises to execute retention strategies (keep all records for X months) and hard deletes / expungement at scale set by policy (all soft deletes must be expunged within 48 hours of a request). One of the benefits of using Postgres + pgvector, is it is much easier to delete records from a vector index versus using a vector database or Redis. While Redis has ttls for objects, there isn't an easy way to remove data from a vector index.

brigadier132 3y ago

When you say you can do vss along with filtering. Does the filtering happen before or after the vss?

roseway4 3y ago
Logically at the same time in the sql query. This may not be how the query is executed by the database, though, and is up to the optimizer.

kiwicopple 3y ago

> It’s important indexes are added after a fair number of vectors have been persisted, which can make some production use cases challenging to implement. managing indexes are the most challenging part of using pgvector right now. There is a bit of a "goldilocks zone" - you can't create the index with zero data, but if you create it too late then it can take a lot of memory. Andrew (and a few others, like AWS) are working hard on improving this, as well as researching other index types.

deepsquirrelnet 3y ago
That’s good to know. I tested it in a docker container and indexed after loading ~1M vectors. Checking EXPLAIN showed it wasn’t using the index at all. I wonder if it’s because it couldn’t load into memory. Without an index, I found it was too slow for data that size. I didn’t expect issues for data at this “small”, but it’s pretty new to me.

sodiumSalt 3y ago

Zep seems really cool! I think hybrid search is a great paradigm and believe it will continue to grow. How do you guys plan on dealing with data deletion requests and other GDPR rules around storage?

roseway4 3y ago
Great question. We have a deletion endpoint that allows sessions (think users) to be deleted. All memory artifacts are timestamped, and we run soft-deletes with deleted_at timestamps. This allows enterprises to execute retention strategies (keep all records for X months) and hard deletes / expungement at scale set by policy (all soft deletes must be expunged within 48 hours of a request). One of the benefits of using Postgres + pgvector, is it is much easier to delete records from a vector index versus using a vector database or Redis. While Redis has ttls for objects, there isn't an easy way to remove data from a vector index.