4 ms·

This is really cool. I have a pretty fast BM25 search engine in Pandas I've been working on for local testing. https://github.com/softwaredoug/searcharray http

This is really cool. I have a pretty fast BM25 search engine in Pandas I've been working on for local testing. https://github.com/softwaredoug/searcharray https://github.com/softwaredoug/searcharray Why Pandas? Because BM25 is one thing, but you also want to combine with other factors (recency, popularity, etc) easily computed in pandas / numpy... BTW phrases are the hard thing. There are a lot of edge case in phrase matching. Not to mention slop, etc. And you want to compact positions into as little memory as possible https://github.com/softwaredoug/searcharray/blob/main/searcharray/utils/roaringish.py https://github.com/softwaredoug/searcharray/blob/main/search...

alexmolas 3y ago
Thanks for your comment Doug! I just bought your book (Relevance Search) and I'm planning to read it asap. I'll give a look to your project and try it for experimenting, thanks for sharing.
- Xenoamorphous 3y ago
  I also bought the book (it’s Relevant not Relevance BTW), that one and Deep Learning for Search were invaluable when building a news search engine a few years ago (right before and in the middle of the pandemic).
  - boyter 3y ago
    Can confirm this book is excellent. It's one I point people at all the time when they ask me why something isn't working as expected in any standard search tool, and something I reference from time to time to refresh my own knowledge. Well worth the money.
- softwaredoug 3y ago
  Thank you! It's a bit old now, but I think the principles in it are still valid :)
thierrydamiba 3y ago
Have you found that including sentiment analysis has helped(or hurt) with phrases? I also have found that phrases are difficult to work with and I’m wondering what I can do to improve performance.
karolist 3y ago
How come you found this post and commented on it so quickly, do you have some sort of search scanning frontpage for terms of interest or was it by chance? Curious
- serialNumber 3y ago
  Huh? I check Hacker News multiple times a day - it's not odd to click on an article within an hour of it being posted.
  - karolist 3y ago
    It's not the first time I saw an article posted and then an expert in the field comment on it rather quickly, I thought I may be missing something how other people use this site, had no negative intentions asking this and thanks for the answer ;)
    - abetusk 3y ago
      HN has an RSS feed [0] so there's no need to keep refreshing or make the rounds of this and other sites that have interesting information. I have my own feed setup with sites I like to frequent [1]. [0] https://hackaday.com/blog/feed/ https://hackaday.com/blog/feed/ [1] https://mechaelephant.com/feed https://mechaelephant.com/feed
      - tsukurimashou 3y ago
        what software did you use for your own feed?
        abetusk 3y ago
        Various scripts in python, shell and javascript [0]. [0] https://github.com/abetusk/www.mechaelephant.com/tree/release/home/meow/rss https://github.com/abetusk/www.mechaelephant.com/tree/releas...
alpinelogic 3y ago
Hey, I tackled phrase matching in my toy project here: https://github.com/vasilionjea/lofi-dx/blob/main/test/search/inverted-search.test.ts#L140 https://github.com/vasilionjea/lofi-dx/blob/main/test/search... I think I tested it thoroughly but any feedback would be appreciated! Edit: I delta-encoded and base36-encoded the positions
- softwaredoug 3y ago
  Oh nice I’ll take a look!

alexmolas 3y ago

Thanks for your comment Doug! I just bought your book (Relevance Search) and I'm planning to read it asap. I'll give a look to your project and try it for experimenting, thanks for sharing.

Xenoamorphous 3y ago
I also bought the book (it’s Relevant not Relevance BTW), that one and Deep Learning for Search were invaluable when building a news search engine a few years ago (right before and in the middle of the pandemic).
- boyter 3y ago
  Can confirm this book is excellent. It's one I point people at all the time when they ask me why something isn't working as expected in any standard search tool, and something I reference from time to time to refresh my own knowledge. Well worth the money.
softwaredoug 3y ago
Thank you! It's a bit old now, but I think the principles in it are still valid :)

thierrydamiba 3y ago

Have you found that including sentiment analysis has helped(or hurt) with phrases? I also have found that phrases are difficult to work with and I’m wondering what I can do to improve performance.

karolist 3y ago

How come you found this post and commented on it so quickly, do you have some sort of search scanning frontpage for terms of interest or was it by chance? Curious

serialNumber 3y ago
Huh? I check Hacker News multiple times a day - it's not odd to click on an article within an hour of it being posted.
- karolist 3y ago
  It's not the first time I saw an article posted and then an expert in the field comment on it rather quickly, I thought I may be missing something how other people use this site, had no negative intentions asking this and thanks for the answer ;)
  - abetusk 3y ago
    HN has an RSS feed [0] so there's no need to keep refreshing or make the rounds of this and other sites that have interesting information. I have my own feed setup with sites I like to frequent [1]. [0] https://hackaday.com/blog/feed/ https://hackaday.com/blog/feed/ [1] https://mechaelephant.com/feed https://mechaelephant.com/feed
    - tsukurimashou 3y ago
      what software did you use for your own feed?
      - abetusk 3y ago
        Various scripts in python, shell and javascript [0]. [0] https://github.com/abetusk/www.mechaelephant.com/tree/release/home/meow/rss https://github.com/abetusk/www.mechaelephant.com/tree/releas...

alpinelogic 3y ago

Hey, I tackled phrase matching in my toy project here: https://github.com/vasilionjea/lofi-dx/blob/main/test/search/inverted-search.test.ts#L140 https://github.com/vasilionjea/lofi-dx/blob/main/test/search... I think I tested it thoroughly but any feedback would be appreciated! Edit: I delta-encoded and base36-encoded the positions

softwaredoug 3y ago
Oh nice I’ll take a look!