3 ms·

> Provide real-time text search over a large corpus (ie, some subset of Project Gutenburg, a bunch of product reviews, etc.) That would be nice. But have you lo

> Provide real-time text search over a large corpus (ie, some subset of Project Gutenburg, a bunch of product reviews, etc.) That would be nice. But have you looked at the PG export yet? It's a million of individual files in the semantic web notation with duplication for the author information, etc. I would LOVE for somebody to convert that into the full-fidelity form suitable for import into the search engine. So, please do it for real, not just as a 'possible'. Another dataset with similar problem is Google Mail takeout. Theoretically in an mbox form, there is apparently enough quirks in there to not be parsable by the standalone 3rd party libraries. Somebody said python might be able to do it, but I haven't seen the confirmation specifically for Google Mail. If you manage to do either and document the steps/share the code, I'll personally sing your praises at the search engine workshops/meetup/lectures I do (usually Solr rather ES).

fheisler 12y ago
I love the idea of analyzing your own gmail - will take a look into parsing MBOX data!
- arafalov 12y ago
  Ok, if you are going to do it, here is the bunch of links that got there part of the way and need someone to take further: http://googleappsdeveloper.blogspot.com/2011/11/parsing-exported-mailboxes-using-python.html http://googleappsdeveloper.blogspot.com/2011/11/parsing-expo... https://docs.python.org/3/library/mailbox.html?highlight=mailbox#module-mailbox https://docs.python.org/3/library/mailbox.html?highlight=mai... https://gist.github.com/ptwobrussell/8791064 https://gist.github.com/ptwobrussell/8791064 http://www.kryogenix.org/days/2013/12/19/searching-my-email/ http://www.kryogenix.org/days/2013/12/19/searching-my-email/ http://batleth.sapienti-sat.org/projects/mb2md/ http://batleth.sapienti-sat.org/projects/mb2md/ (if going from mbox to mailbox is easier)
makmanalp 12y ago
I think you can also dump it in json, which would be ideal for this.
- arafalov 12y ago
  Which one of these can you dump into JSON? Do you have references? Could be nice.
  - grinich 12y ago
    You could use the Inbox APIs! :) More info w/ docs: https://www.inboxapp.com/ https://www.inboxapp.com/ Open-source version: https://github.com/inboxapp/inbox https://github.com/inboxapp/inbox
YousefED 12y ago
I actually just loaded emails from both the GMail API and an mbox file into ES. Haven't gotten around to do advanced analytics yet, mainly using it to load into Mixpanel / Intercom to track events such as "customer has replied to one of our emails". We then use this info to do funnel analysis in Mixpanel, and exclude users we're currently in touch with via email from intercom onboarding emails.
- arafalov 12y ago
  Well, don't just brag. Write a blog post, publish a GitHub repo. Share it somehow.

fheisler 12y ago

I love the idea of analyzing your own gmail - will take a look into parsing MBOX data!

arafalov 12y ago
Ok, if you are going to do it, here is the bunch of links that got there part of the way and need someone to take further: http://googleappsdeveloper.blogspot.com/2011/11/parsing-exported-mailboxes-using-python.html http://googleappsdeveloper.blogspot.com/2011/11/parsing-expo... https://docs.python.org/3/library/mailbox.html?highlight=mailbox#module-mailbox https://docs.python.org/3/library/mailbox.html?highlight=mai... https://gist.github.com/ptwobrussell/8791064 https://gist.github.com/ptwobrussell/8791064 http://www.kryogenix.org/days/2013/12/19/searching-my-email/ http://www.kryogenix.org/days/2013/12/19/searching-my-email/ http://batleth.sapienti-sat.org/projects/mb2md/ http://batleth.sapienti-sat.org/projects/mb2md/ (if going from mbox to mailbox is easier)

makmanalp 12y ago

I think you can also dump it in json, which would be ideal for this.

arafalov 12y ago
Which one of these can you dump into JSON? Do you have references? Could be nice.
- grinich 12y ago
  You could use the Inbox APIs! :) More info w/ docs: https://www.inboxapp.com/ https://www.inboxapp.com/ Open-source version: https://github.com/inboxapp/inbox https://github.com/inboxapp/inbox

YousefED 12y ago

I actually just loaded emails from both the GMail API and an mbox file into ES. Haven't gotten around to do advanced analytics yet, mainly using it to load into Mixpanel / Intercom to track events such as "customer has replied to one of our emails". We then use this info to do funnel analysis in Mixpanel, and exclude users we're currently in touch with via email from intercom onboarding emails.

arafalov 12y ago
Well, don't just brag. Write a blog post, publish a GitHub repo. Share it somehow.