4 ms·
I experimented with that a few months ago. Building a fresh FAISS index for a few thousand matches is really quick, so I think it's often better to filter first
by simonw 3y ago
I experimented with that a few months ago. Building a fresh FAISS index for a few thousand matches is really quick, so I think it's often better to filter first, build a scratch index and then use that for similarity: https://github.com/simonw/datasette-faiss/issues/3 https://github.com/simonw/datasette-faiss/issues/3
... although on thinking about this more I realize that a better approach may well be to just filter down to ~1,000 and then run a brute-force score across all of them, rather than messing around with an index.
- rolisz 3y agoFor 1000 points, brute force is super quick. Actually, up to 100k (on my machine), brute force takes less than 1 second.