3 ms·
sqlite has r-trees for instance [0]. Could it be good enough for most use cases? If it's to query a knowledge base for instance, a couple dimensions should be s
by sgu999 3y ago
sqlite has r-trees for instance [0]. Could it be good enough for most use cases? If it's to query a knowledge base for instance, a couple dimensions should be sufficient. With the added benefit of being able to query your data in other ways.
[0] https://www.sqlite.org/rtree.html https://www.sqlite.org/rtree.html
- mattashii 3y agor*-trees work doesn't work well when the number of dimensions stored in the index is much higher than the logarithm of the number of indexed entries, and this is a prevailing property of divide-and-conquer spatial index types when the keyspace is divided based on a single dimension at a time. As vectors regularly have 100+ dimensions, normal spatial indexing methods applied to vectors wouldn't be very efficient for anything with much less than 2^100 index entries; which is quite suboptimal for most datasets that you would want to have indexed.
- KirillPanov 3y agoAlso the distance metric for r*-trees is just plain wacky for anything other than low-dimensional Euclidean space. Even if you could make it perform well, it would not do what you want.
- modulovalue 3y agoAre you saying this because r-trees expect a proper metric space, and people have the need to index datasets over non-metric spaces?
- pornel 3y agoThe curse of dimensionality creates a seemingly paradoxical situation where you have a vast vast search space, but everything is incredibly close to each other. Space subdivision algorithms become ineffective.
- jpcapdevila 3y agoHere is a SQLite extension that uses Faiss under the hood. https://github.com/asg017/sqlite-vss https://github.com/asg017/sqlite-vss Not associated with the project, just love SQLite and find it very useful.