4 ms·
Be careful with PQ, streaming updates will degrade recall performance a lot (and in ways you may miss)
by jeffchuber 3y ago
Be careful with PQ, streaming updates will degrade recall performance a lot (and in ways you may miss)
- fzliu 3y agoA naive PQ index that trains the quantizer once and reuses the same codebook for recent data definitely has this problem. We (Milvus/Zilliz) break up all the vectors into multiple indexes at certain checkpoints which we call segments. This doesn't fully solve the problem of degraded performance with PQ or hybrid PQ indexes, but certainly helps mitigate it. More info here: https://milvus.io/docs/glossary.md#Segment https://milvus.io/docs/glossary.md#Segment