4 ms·
You might want to look at Morelikethis queries to boost performance. I worked on a proprietary version of this and at the time Lucene performance dropped off ne
by richmarr 11y ago
You might want to look at Morelikethis queries to boost performance. I worked on a proprietary version of this and at the time Lucene performance dropped off nearly linearly with the number of query terms.
We used MoreLikeThis to reduce our queries count to the 30-40 most statistically interesting terms. The one hiccup being an issue in Lucene [1] where the term cache wasn't operating properly. We just added our own image query term cache and a custom MLT query to leverage it, which gave us a 10x speed bump over any other methods we tried.
The interestingness of the terms is assessed on a per-term basis though, so you might see a relevence drop for some types of image if you set MoreLikeThis to use too few terms.
[1] https://issues.apache.org/jira/browse/LUCENE-1690 https://issues.apache.org/jira/browse/LUCENE-1690
- rhsimplex 11y agoThank you for the suggestion. I actually did try restricting the terms by measuring correlation between columns -- the idea being that more discriminating terms should be searched first. This did result in modest speedups. Fortunately or unfortunately, we were already achieving pretty good speed with Elasticsearch so we didn't implement it. However, it didn't occur to me to try a MoreLikeThis query, which should be even simpler -- I will look into it!
- richmarr 11y agoCool. Impressive project by the way; I forgot to say that before. I tried something similar; but with a different approach. I tried creating compound words, a bit like n-grams. I didn't get it working as that was a side-project and I couldn't commit enough time.