5 ms·

Instacart launches Searchkick - Bootstrap for Search

94 points by apoorvamehta 13y ago

asnyder 13y ago
This looks pretty good. Though many of the features are already baked into PostgreSQL's full-text search, out of the box this seems to add enough value to get me interested. I'll definitely be checking this out further.
- akane 13y ago
  We looked into PostgreSQL search for a bit, but didn't see a way to train search results from past conversions. Does anyone know if this is possible?
- sheraz 13y ago
  Upvoted, and I'll pile on. Wordnet [1] already has addons for Postgres. [1] - https://github.com/taotetek/wordnet-postgres https://github.com/taotetek/wordnet-postgres
  - gojomo 13y ago
    Can you clarify... does this addon simply store the Wordnet data inside Postgres, or actually let the postgres full-text search do query-expansion-to-synonyms?
    - sheraz 13y ago
      Sure. There are functions in this package. This link is better, which explains how the functions will work. http://wnsqlbuilder.sourceforge.net/sql.html http://wnsqlbuilder.sourceforge.net/sql.html
- MitziMoto 13y ago
  I recently had to implement search on my product and found Postgres' full-text search to be fairly impressive. More than enough for what I needed. Implemented through the wonderful textacular gem: https://github.com/textacular/textacular https://github.com/textacular/textacular
- apoorvamehta 13y ago
  Interesting. I'll have to look into PostgreSQL's full text search. Does it support token filters similar to Elasticsearch?
  - asnyder 13y ago
    To a degree, and is also very customizeable. It breaks things out to the lexeme level, read http://www.postgresql.org/docs/9.2/interactive/textsearch-intro.html http://www.postgresql.org/docs/9.2/interactive/textsearch-in....
jmduke 13y ago
This may be a dumb question, but does Instacart use Searchkick?
- shiftb 13y ago
  Yes, Instacart uses Searchkick.
druiid 13y ago
Looks very cool! Going to play around with it some. Is this a first open release of the project or is it in quiet use other places than Instacart?
- akane 13y ago
  It's been on Github for a bit, but this is the first big release.
fowkswe 13y ago
This is really nice. Thanks.
kapilvt 13y ago
Elasticsearch should be the bootstrap of search imo. Afaics the only interesting bit in this project is the reindex technique. The rest is just trying to make a nice rails idiomatic client library (perhaps for ruby too) around search querying. For a python idiomatic client lib comparison, see mozilla services http://elasticutils.readthedocs.org/en/latest/ http://elasticutils.readthedocs.org/en/latest/
- akane 13y ago
  Training searches from conversions (self-improving searches) is kinda interesting. Elasticsearch is an amazing product but it can be hard to configure, especially for beginners. Edit: spelling
jordanthoms 13y ago
This is nice - We use elasticsearch, but getting good quality has been hard - setting up the tokenizers, filters etc in a good way is tricky, and sometimes our search gives really bad results.
- akane 13y ago
  That's exactly why we open-sourced Searchkick :)
boomzilla 13y ago
Hardcoding your scoring function here is a bad, bad idea: https://github.com/ankane/searchkick/blob/master/lib/searchkick/search.rb#L11 https://github.com/ankane/searchkick/blob/master/lib/searchk... Just curious, how many docs are you indexing?
- akane 13y ago
  We're indexing around 100k documents. What line are you referring to, and what's bad about it?
- smartt 13y ago
  I'm curious about this "bad, bad idea" theory too. For our site, I have a handful of hard-coded scoring functions that the site can select from based on search context, and it's far more supportable and debuggable than some of the dynamic query assembly approaches I've tried in the past. ElasticSearch is a wonderful product that is a real pain to tune, so I'm curious if there's a better approach I'm missing.
brunomiranda 13y ago
I appreciate that you decided to open-source this project. I do have a few (hopefully) constructive criticisms to make: 1. Using aliases the reindex with near zero downtime is very clever, mad props to Jaroslav Kalistsuk 2. Tire is excellent, I am glad you didn't try to reinvent the wheel here, please keep it that way. 3. Conversion based score boosting, although clever is flawed. Hardcoding values https://github.com/ankane/searchkick/blob/master/lib/searchkick/search.rb#L17 https://github.com/ankane/searchkick/blob/master/lib/searchk... is convoluted. You should consider providing a more flexible DSL to customize how conversion based score is calculated. 4. Analyzers https://github.com/ankane/searchkick/blob/master/lib/searchkick/reindex.rb#L42 https://github.com/ankane/searchkick/blob/master/lib/searchk... are poorly named and do not represent what they do. Again, this is very nice for version 0.1.2
apapli 13y ago
Thanks! I'll give this a go for sure!

94 points by apoorvamehta 13y ago

This looks pretty good. Though many of the features are already baked into PostgreSQL's full-text search, out of the box this seems to add enough value to get me interested. I'll definitely be checking this out further.

akane 13y ago
We looked into PostgreSQL search for a bit, but didn't see a way to train search results from past conversions. Does anyone know if this is possible?
sheraz 13y ago
Upvoted, and I'll pile on. Wordnet [1] already has addons for Postgres. [1] - https://github.com/taotetek/wordnet-postgres https://github.com/taotetek/wordnet-postgres
- gojomo 13y ago
  Can you clarify... does this addon simply store the Wordnet data inside Postgres, or actually let the postgres full-text search do query-expansion-to-synonyms?
  - sheraz 13y ago
    Sure. There are functions in this package. This link is better, which explains how the functions will work. http://wnsqlbuilder.sourceforge.net/sql.html http://wnsqlbuilder.sourceforge.net/sql.html
MitziMoto 13y ago
I recently had to implement search on my product and found Postgres' full-text search to be fairly impressive. More than enough for what I needed. Implemented through the wonderful textacular gem: https://github.com/textacular/textacular https://github.com/textacular/textacular
apoorvamehta 13y ago
Interesting. I'll have to look into PostgreSQL's full text search. Does it support token filters similar to Elasticsearch?
- asnyder 13y ago
  To a degree, and is also very customizeable. It breaks things out to the lexeme level, read http://www.postgresql.org/docs/9.2/interactive/textsearch-intro.html http://www.postgresql.org/docs/9.2/interactive/textsearch-in....

jmduke 13y ago

This may be a dumb question, but does Instacart use Searchkick?

shiftb 13y ago
Yes, Instacart uses Searchkick.

druiid 13y ago

Looks very cool! Going to play around with it some. Is this a first open release of the project or is it in quiet use other places than Instacart?

akane 13y ago
It's been on Github for a bit, but this is the first big release.

fowkswe 13y ago

This is really nice. Thanks.

kapilvt 13y ago

Elasticsearch should be the bootstrap of search imo. Afaics the only interesting bit in this project is the reindex technique. The rest is just trying to make a nice rails idiomatic client library (perhaps for ruby too) around search querying. For a python idiomatic client lib comparison, see mozilla services http://elasticutils.readthedocs.org/en/latest/ http://elasticutils.readthedocs.org/en/latest/

akane 13y ago
Training searches from conversions (self-improving searches) is kinda interesting. Elasticsearch is an amazing product but it can be hard to configure, especially for beginners. Edit: spelling

jordanthoms 13y ago

This is nice - We use elasticsearch, but getting good quality has been hard - setting up the tokenizers, filters etc in a good way is tricky, and sometimes our search gives really bad results.

akane 13y ago
That's exactly why we open-sourced Searchkick :)

boomzilla 13y ago

Hardcoding your scoring function here is a bad, bad idea: https://github.com/ankane/searchkick/blob/master/lib/searchkick/search.rb#L11 https://github.com/ankane/searchkick/blob/master/lib/searchk... Just curious, how many docs are you indexing?

akane 13y ago
We're indexing around 100k documents. What line are you referring to, and what's bad about it?
smartt 13y ago
I'm curious about this "bad, bad idea" theory too. For our site, I have a handful of hard-coded scoring functions that the site can select from based on search context, and it's far more supportable and debuggable than some of the dynamic query assembly approaches I've tried in the past. ElasticSearch is a wonderful product that is a real pain to tune, so I'm curious if there's a better approach I'm missing.

brunomiranda 13y ago

I appreciate that you decided to open-source this project. I do have a few (hopefully) constructive criticisms to make: 1. Using aliases the reindex with near zero downtime is very clever, mad props to Jaroslav Kalistsuk 2. Tire is excellent, I am glad you didn't try to reinvent the wheel here, please keep it that way. 3. Conversion based score boosting, although clever is flawed. Hardcoding values https://github.com/ankane/searchkick/blob/master/lib/searchkick/search.rb#L17 https://github.com/ankane/searchkick/blob/master/lib/searchk... is convoluted. You should consider providing a more flexible DSL to customize how conversion based score is calculated. 4. Analyzers https://github.com/ankane/searchkick/blob/master/lib/searchkick/reindex.rb#L42 https://github.com/ankane/searchkick/blob/master/lib/searchk... are poorly named and do not represent what they do. Again, this is very nice for version 0.1.2

apapli 13y ago

Thanks! I'll give this a go for sure!