3 ms·
Show HN: SearchArray – Text Search in Pandas
I've long worked with lexical Lucene based search engines like Solr and Elasticsearch. Anytime I need to experiment with relevance ranking in these systems, I'm exhausted by needing to set them up. It's not conducive to productivity work with something so disjoint from normal data tooling.
Further - the underlying ranking is buried in needless mystique (you know a boolean should query, sums the scores, right?). You shouldn't need to read a book (like Relevant Search ;) ) to unpack mystique that's really basic math.
Why not just let people build ranking systems with vectorized math in a numpy/pandas stack?
SearchArray lets anyone build a relevance prototype in Pandas. Typically building / experimenting with a smaller labeled dataset. If it works out, you can transfer it relatively easily to Elasticsearch or Solr for implementation and testing.
SearchArray is a pandas extension array that creates an underlying search index for BM25 term/phrase based searching.
It's not quite done (will it ever be?) but its getting far enough along to be useful. So feedback is very welcome.
https://github.com/softwaredoug/searcharray
- benrules2 3y agoWhat sort of BM25 indices does this support? Does it allow experimenting with an elasticsearch query and mapping for example?
- softwaredoug 3y agoRight now it’s the typical inverted index term -> list of doc ids And the main purpose of the data structure is to recover term frequencies and document frequencies. We also store positional information to allow phrase matching. BM25 of course is just one such way of using these stats. But you can also get raw termfreqs and docfreqs of matching terms and do whatever you want with them mathematically :). The BM25 here tries to align to Lucenes internal BM25 calculation.