4 ms·
This looks really interesting. Full search engines rank documents based on the similarity of a query to documents in the index, but it seems like there are a lo
by garysieling 10y ago
This looks really interesting. Full search engines rank documents based on the similarity of a query to documents in the index, but it seems like there are a lot of opportunities to do more with similarities between content.
One thing I'm interested in is the ability to find the most dis-similar document to a query / other documents (perhaps within some constraints). I think this would potentially be really helpful for discovery. For context, on https://www.findlectures.com https://www.findlectures.com, I'd like to show talks on subjects I know the least about i.e., most dis-similar from all talks I've previously watched.
- jhj 10y agoFaiss query, at least for IVFPQ which is the main emphasis of the library, is based on real vectors as input but against quantized database vectors; the so called “ADC” (asymmetric distance computation), so you don't have to query based on a database vector. If the data is normalized to lie on a hypersphere, you could find nearest neighbors to a query vector on the opposite point of the hypersphere, for instance. But, as we sometimes find with, say, negative mining, sometimes what you are interested in is not the “hardest” negative or most dissimilar object as those don’t provide much utility in comparison to your current object; you might something that is reasonably close in some ways but quite different otherwise, which is a more useful negative example. This is more difficult.
- garysieling 10y agoThanks for the comment, that is a good way to articulate the problem I'm interested in. For instance, seeing novel topics, but retaining some form of high quality results - "low quality" would also be an opposite result, but clearly isn't the right answer.