4 ms·

Pgvector 0.4.0 Performance

11 points by kiwicopple 3y ago

kiwicopple 3y ago
hey hn, supabase ceo here this is a response to a recent pgvector vs qdrant benchmark. As we mention in the post, we’re not trying to prove that pgvector is faster than Qdrant. Our goal is to show that pgvector/postgres scales predictably for the workloads that most companies need. We scale it to ~1800 requests per second with 91% precision in this set of benchmarks. Still, it could be better. Luckily there are quite a few features and improvements planned for v0.5.0 [0]: - Adding HNSW: an index with better speed & precision than IVFFlat (at a higher memory cost) - Product quantization: better storage for IVFFLAT, improving speed and recall - Parallel index builds: building your IVFFLAT indexes will be much faster [0]: pgvector v0.5: https://github.com/pgvector/pgvector/issues/27 https://github.com/pgvector/pgvector/issues/27
- egorr 3y ago
  supabase engineer here, would be happy to share details regarding our research and outcomes if you have any questions
  - henry_pulver 3y ago
    Had a question about the "pre-warming" technique you used - why does this help? You say it helps with RAM utilization, but why? Does it just spin up the right number of probes up so the experiments don't include this extra cost of spinning up these probes? (apologies if this is a dumb question - I don't know enough about vector databases to know if this is a dumb question or not!)
deleted 3y ago
[deleted]
andre-z 3y ago
Hey, qdrant ceo here. Paul, you and your team are building amazing open-source tech! And, of course, you can scale (almost) anything with just more hardware. As Nirant mentioned in his Twitter response. https://twitter.com/NirantK/status/1679525251213066245 https://twitter.com/NirantK/status/1679525251213066245 So, it is not only a question of performance but also efficiency. PS: What about offering Qdrant as an alternative to PG on the supabase platform? Just an idea. :)
- egorr 3y ago
  i am not Paul of course, but i just want to highlight that the idea behind blogpost was to show that if you set some settings for postgres correctly, you can achieve much better performance ie you can notice that most of results are on the same hardware as in nirantk, and we achieved 7 times better RPS with 0.98 precision on it and 19 times better RPS with 0.91 precision on the same amount of cpu and memory goal was to compare pgvector with pgvector rather than any other technology
- kiwicopple 3y ago
  > What about offering Qdrant as an alternative to PG on the supabase platform We are "all in" on Postgres, but we could definitely find ways to support qdrant (probably a Wrapper: https://supabase.com/blog/postgres-foreign-data-wrappers-rust https://supabase.com/blog/postgres-foreign-data-wrappers-rus...)
pjs_sudo 3y ago
What does "precision@10" mean in the graphs shown in the blog?
- egorr 3y ago
  it means the precision at K(10) or precision for first 10 documents retrieved. In other words - number of relevant documents amongst retrieved 10(k) of them So 0.98 precision would mean that 9.8 out of 10 documents are relevant on average

11 points by kiwicopple 3y ago

hey hn, supabase ceo here this is a response to a recent pgvector vs qdrant benchmark. As we mention in the post, we’re not trying to prove that pgvector is faster than Qdrant. Our goal is to show that pgvector/postgres scales predictably for the workloads that most companies need. We scale it to ~1800 requests per second with 91% precision in this set of benchmarks. Still, it could be better. Luckily there are quite a few features and improvements planned for v0.5.0 [0]: - Adding HNSW: an index with better speed & precision than IVFFlat (at a higher memory cost) - Product quantization: better storage for IVFFLAT, improving speed and recall - Parallel index builds: building your IVFFLAT indexes will be much faster [0]: pgvector v0.5: https://github.com/pgvector/pgvector/issues/27 https://github.com/pgvector/pgvector/issues/27

egorr 3y ago
supabase engineer here, would be happy to share details regarding our research and outcomes if you have any questions
- henry_pulver 3y ago
  Had a question about the "pre-warming" technique you used - why does this help? You say it helps with RAM utilization, but why? Does it just spin up the right number of probes up so the experiments don't include this extra cost of spinning up these probes? (apologies if this is a dumb question - I don't know enough about vector databases to know if this is a dumb question or not!)

deleted 3y ago

[deleted]

andre-z 3y ago

Hey, qdrant ceo here. Paul, you and your team are building amazing open-source tech! And, of course, you can scale (almost) anything with just more hardware. As Nirant mentioned in his Twitter response. https://twitter.com/NirantK/status/1679525251213066245 https://twitter.com/NirantK/status/1679525251213066245 So, it is not only a question of performance but also efficiency. PS: What about offering Qdrant as an alternative to PG on the supabase platform? Just an idea. :)

egorr 3y ago
i am not Paul of course, but i just want to highlight that the idea behind blogpost was to show that if you set some settings for postgres correctly, you can achieve much better performance ie you can notice that most of results are on the same hardware as in nirantk, and we achieved 7 times better RPS with 0.98 precision on it and 19 times better RPS with 0.91 precision on the same amount of cpu and memory goal was to compare pgvector with pgvector rather than any other technology
kiwicopple 3y ago
> What about offering Qdrant as an alternative to PG on the supabase platform We are "all in" on Postgres, but we could definitely find ways to support qdrant (probably a Wrapper: https://supabase.com/blog/postgres-foreign-data-wrappers-rust https://supabase.com/blog/postgres-foreign-data-wrappers-rus...)

pjs_sudo 3y ago

What does "precision@10" mean in the graphs shown in the blog?

egorr 3y ago
it means the precision at K(10) or precision for first 10 documents retrieved. In other words - number of relevant documents amongst retrieved 10(k) of them So 0.98 precision would mean that 9.8 out of 10 documents are relevant on average