3 ms·
The industry standard tool is probably Apache Lucene [1]. It uses a well documented mechanism and is open source so it would make a good place to start research
by brudgers 9y ago
The industry standard tool is probably Apache Lucene [1]. It uses a well documented mechanism and is open source so it would make a good place to start research.
Good luck.
[1]: https://en.wikipedia.org/wiki/Apache_Lucene https://en.wikipedia.org/wiki/Apache_Lucene
- Skinish 9y agoThanks, but Apache Lucene is a search engine. I want to build a relatedness classifier
- PaulHoule 9y agoThe mainstream of IR research has deliberately avoided the development of a "relatedness classifier" in the sense of a system that produces a probability score that a document is relevant. The TREC organizers have been mostly interested in "get good recall" and less interested in P@1 or rewarding the kid who sticks his hand up really high when it's an easy question and knows the answer. You can tune up a conventional IR function with these kind of methods http://scikit-learn.org/stable/modules/calibration.html http://scikit-learn.org/stable/modules/calibration.html with the data that TREC publishes. The results are a little disappointing because the system will probably never predict more than a 70% probability of relevance. I am interested in the promise of "Siamese networks" for learning relatedness, see https://www.cs.cmu.edu/~rsalakhu/papers/oneshot1.pdf https://www.cs.cmu.edu/~rsalakhu/papers/oneshot1.pdf for example. For document-document matching I think this approach would be great (and supportive of relevance feedback, something that mainstream IR algorithms have never done that well at) Query-document matching may require something similar but different, since there is an asymmetry between queries (short) and documents (long.) Click on my HN profile link and we can talk more. Thanks!
- Skinish 9y agoThank you for such a detailed answer! I'll get in touch you tomorrow then, for sure :)