5 ms·

First Monthly Challenge: Elasticsearch

37 points by fheisler 12y ago

linux_devil 12y ago
I use elastic search + Python scripts(running as bolt on Storm)+ Kibana (with better map) for analytics and its uber cool . Only things where Kibana miss out is 'unique counts' , multiple dashboards across different Document types etc. I hope these get addressed in near future .
- ehurrell 12y ago
  That sounds like an interesting stack, I use python quite a bit and have been dabbling with elasticsearch, glad to see the options for combining the two!
- divideby0 12y ago
  Kibana 4 should have some of that: http://www.elasticsearch.org/blog/kibana-4-beta-1-released/ http://www.elasticsearch.org/blog/kibana-4-beta-1-released/ It exposes quite a bit of the new aggregation functionality added to ES 1.x.
  - Karunamon 12y ago
    Unfortunately Kibana 4 is missing a lot of functionality from 3.x. One feature that we really want back is the global search scope per dashboard. Example: Say you've got a dashboard showing hits on a web service. You've got a pie chart showing HTTP return codes, a bar chart showing response times, and another few graphs and charts detailing various data out of the requests themselves. You could click on, say, the "500" in your return code pie chart, and then every visualization on the page would redraw and show you stats for just requests that that were 500s. (What's unique about the requests that return server errors?) Or turn it around - click on the section of a chart that denotes requests that took longer than 100ms to process, and now you see info about those requests only. (What makes these long-time requests so special?) This was a jaw-droppingly awesome troubleshooting tool, and now it's gone. I hope they return it before Kibana4 gets out of beta!
arafalov 12y ago
> Provide real-time text search over a large corpus (ie, some subset of Project Gutenburg, a bunch of product reviews, etc.) That would be nice. But have you looked at the PG export yet? It's a million of individual files in the semantic web notation with duplication for the author information, etc. I would LOVE for somebody to convert that into the full-fidelity form suitable for import into the search engine. So, please do it for real, not just as a 'possible'. Another dataset with similar problem is Google Mail takeout. Theoretically in an mbox form, there is apparently enough quirks in there to not be parsable by the standalone 3rd party libraries. Somebody said python might be able to do it, but I haven't seen the confirmation specifically for Google Mail. If you manage to do either and document the steps/share the code, I'll personally sing your praises at the search engine workshops/meetup/lectures I do (usually Solr rather ES).
- fheisler 12y ago
  I love the idea of analyzing your own gmail - will take a look into parsing MBOX data!
  - arafalov 12y ago
    Ok, if you are going to do it, here is the bunch of links that got there part of the way and need someone to take further: http://googleappsdeveloper.blogspot.com/2011/11/parsing-exported-mailboxes-using-python.html http://googleappsdeveloper.blogspot.com/2011/11/parsing-expo... https://docs.python.org/3/library/mailbox.html?highlight=mailbox#module-mailbox https://docs.python.org/3/library/mailbox.html?highlight=mai... https://gist.github.com/ptwobrussell/8791064 https://gist.github.com/ptwobrussell/8791064 http://www.kryogenix.org/days/2013/12/19/searching-my-email/ http://www.kryogenix.org/days/2013/12/19/searching-my-email/ http://batleth.sapienti-sat.org/projects/mb2md/ http://batleth.sapienti-sat.org/projects/mb2md/ (if going from mbox to mailbox is easier)
- makmanalp 12y ago
  I think you can also dump it in json, which would be ideal for this.
mercnet 12y ago
Great article as I now feel the need to research Elasticsearch vs Postgres Full Text Search to see which one would benefit my side project.
- chishaku 12y ago
  I had the same question and went with Elasticsearch. Also check this re: postgres... http://blog.lostpropertyhq.com/postgres-full-text-search-is-good-enough/#1 http://blog.lostpropertyhq.com/postgres-full-text-search-is-...
troygoode 12y ago
Great idea Fletcher. We <3 ElasticSearch here at Lanetix as well and I think some of our developers that haven't worked with it as much will give your challenge a shot.
- fheisler 12y ago
  Awesome, send along links to anything they come up with! We're thinking of live streaming the meetup, so that might be a good way to share around as well.
  - troygoode 12y ago
    Will do. And we'd love to watch if you live stream it.

37 points by fheisler 12y ago

I use elastic search + Python scripts(running as bolt on Storm)+ Kibana (with better map) for analytics and its uber cool . Only things where Kibana miss out is 'unique counts' , multiple dashboards across different Document types etc. I hope these get addressed in near future .

ehurrell 12y ago
That sounds like an interesting stack, I use python quite a bit and have been dabbling with elasticsearch, glad to see the options for combining the two!
divideby0 12y ago
Kibana 4 should have some of that: http://www.elasticsearch.org/blog/kibana-4-beta-1-released/ http://www.elasticsearch.org/blog/kibana-4-beta-1-released/ It exposes quite a bit of the new aggregation functionality added to ES 1.x.
- Karunamon 12y ago
  Unfortunately Kibana 4 is missing a lot of functionality from 3.x. One feature that we really want back is the global search scope per dashboard. Example: Say you've got a dashboard showing hits on a web service. You've got a pie chart showing HTTP return codes, a bar chart showing response times, and another few graphs and charts detailing various data out of the requests themselves. You could click on, say, the "500" in your return code pie chart, and then every visualization on the page would redraw and show you stats for just requests that that were 500s. (What's unique about the requests that return server errors?) Or turn it around - click on the section of a chart that denotes requests that took longer than 100ms to process, and now you see info about those requests only. (What makes these long-time requests so special?) This was a jaw-droppingly awesome troubleshooting tool, and now it's gone. I hope they return it before Kibana4 gets out of beta!

arafalov 12y ago

> Provide real-time text search over a large corpus (ie, some subset of Project Gutenburg, a bunch of product reviews, etc.) That would be nice. But have you looked at the PG export yet? It's a million of individual files in the semantic web notation with duplication for the author information, etc. I would LOVE for somebody to convert that into the full-fidelity form suitable for import into the search engine. So, please do it for real, not just as a 'possible'. Another dataset with similar problem is Google Mail takeout. Theoretically in an mbox form, there is apparently enough quirks in there to not be parsable by the standalone 3rd party libraries. Somebody said python might be able to do it, but I haven't seen the confirmation specifically for Google Mail. If you manage to do either and document the steps/share the code, I'll personally sing your praises at the search engine workshops/meetup/lectures I do (usually Solr rather ES).

fheisler 12y ago
I love the idea of analyzing your own gmail - will take a look into parsing MBOX data!
- arafalov 12y ago
  Ok, if you are going to do it, here is the bunch of links that got there part of the way and need someone to take further: http://googleappsdeveloper.blogspot.com/2011/11/parsing-exported-mailboxes-using-python.html http://googleappsdeveloper.blogspot.com/2011/11/parsing-expo... https://docs.python.org/3/library/mailbox.html?highlight=mailbox#module-mailbox https://docs.python.org/3/library/mailbox.html?highlight=mai... https://gist.github.com/ptwobrussell/8791064 https://gist.github.com/ptwobrussell/8791064 http://www.kryogenix.org/days/2013/12/19/searching-my-email/ http://www.kryogenix.org/days/2013/12/19/searching-my-email/ http://batleth.sapienti-sat.org/projects/mb2md/ http://batleth.sapienti-sat.org/projects/mb2md/ (if going from mbox to mailbox is easier)
makmanalp 12y ago
I think you can also dump it in json, which would be ideal for this.

mercnet 12y ago

Great article as I now feel the need to research Elasticsearch vs Postgres Full Text Search to see which one would benefit my side project.

chishaku 12y ago
I had the same question and went with Elasticsearch. Also check this re: postgres... http://blog.lostpropertyhq.com/postgres-full-text-search-is-good-enough/#1 http://blog.lostpropertyhq.com/postgres-full-text-search-is-...

troygoode 12y ago

Great idea Fletcher. We <3 ElasticSearch here at Lanetix as well and I think some of our developers that haven't worked with it as much will give your challenge a shot.

fheisler 12y ago
Awesome, send along links to anything they come up with! We're thinking of live streaming the meetup, so that might be a good way to share around as well.
- troygoode 12y ago
  Will do. And we'd love to watch if you live stream it.