4 ms·
Looks like solr is being used to rank messages using a few features and then re-ranking the top n at application layer using another set of features. This would
by ankit-singh 10y ago
Looks like solr is being used to rank messages using a few features and then re-ranking the top n at application layer using another set of features. This would constrain the search quality as 1) you have no control as to how these two set of features interact and 2) messages ranked low based on first set of features could be highly relevant according to the second set.
Another possible approach could have been using a custom scorer to influence scoring at lucene level, thereby combining all the features at a single point. Was this approach evaluated? If so, any insights as to what could be a limitation?
- isabellat 10y agoGreat question. We rank in two stages for a number of different reasons. First, it would be too expensive from a performance perspective to rank all of the messages in your corpus. Second, some of the features that we use to rank are much more easily accessed at the application layer. It would require more of an engineering effort to make these signals accessible in SOLR. The first pass which is done in SOLR is a high recall, low precision pass. The second pass through our custom ranker is a high precision pass. It is possible that we would lose some messages that might end up being important in the first pass but it's a tradeoff between performance and accuracy. Hope this helps answer your question.
- Eridrus 10y agoBesides the technical answer already given, this is a pretty standard architecture for search ranking and other problems where your fine grained decisions are too expensive to run on everything.