5 ms·
As others have pointed out there are some features considered essential in this space that would be good additions to the code: Stemming: (removing prefix/suff
by michelpp 7y ago
As others have pointed out there are some features considered essential in this space that would be good additions to the code:
Stemming: (removing prefix/suffixes walked, walking, walker -> walk) The snowball library contains many language stemmers https://snowballstem.org https://snowballstem.org
Stopping: removing many common words like the/and that almost every document will surely match anyway. Again, per language.
Relevance ranking: Generally some variant of TF/IDF like BM25. The book "Managing Gigabytes" is an excellent intro to the subject and information retrieval in general: https://people.eng.unimelb.edu.au/ammoffat/mg/ https://people.eng.unimelb.edu.au/ammoffat/mg/
A Document/Term/Value model, this is how Lucence, Xapian, and other IR systems model the store. It's worth sticking with that same pattern.
Good luck!
- max0563 7y agoThis is very helpful. I am working on all of this, but summarizing it all here helped me a lot. Thank you.