6 ms·

ElasticDump – Import and export tools for ElasticSearch

69 points by evantahler 12y ago

evlapix 12y ago
I haven't used this yet, but it seems like it should make updating mappings much easier. See the "using aliases for greater flexibility" section in this article: http://www.elasticsearch.org/blog/changing-mapping-with-zero-downtime/ http://www.elasticsearch.org/blog/changing-mapping-with-zero... I had to write the scripts to implement this concept myself and it wasn't a quick and easy task. It would had gone along a lot easier if I was able to abstract away some of those queries with a tool like this.
- evantahler 12y ago
  We use index mappings at TaskRabbit, and it does make things a lot easier. It helps with 0-downtime mapping changes (as you point out), and it also lets us combine large indexes. We segment large data collections by time (ie: events_year_month) which we roll up in an alias. This allows us to delete/add large collections without downtime as well. What's cool about the elsasticsearch api (and therefore the elasticdump tool) is that with aliases, you can read/write to and from an index just like you can an alias.
deleted 12y ago
[deleted]
andrewvc 12y ago
This is a neat tool for moving data around, but for serious stuff I'm not sure why you would use this over the official snapshot / restore API (http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html http://www.elasticsearch.org/guide/en/elasticsearch/referenc...). While the API there is a bit more cumbersome, it's one less dependency, and should be more efficient. For the specific use case of backups, the official API is even less cumbersome, esp. if you're using the cloud plugin and the S3 export option.
- schmichael 12y ago
  snapshot/restore doesn't allow applying new mappings. Also I use our similar custom tool* for dev purposes where you might want to kill the copy after a while and just work with a subset of the data... or explicitly apply a filter to copy a subset of data. * https://github.com/lytics/escp https://github.com/lytics/escp
  - Xylakant 12y ago
    on top of that you can apply transformations to the data if you use scroll and bulk, but not with dump/restore. On the flip side, dump/restore allows incremental transfer which is a bit of work to do if you're implementing your own copy script.
schmichael 12y ago
While we don't have a spiffy logo, we made a tool for copying indexes around for testing/dev: https://github.com/lytics/escp https://github.com/lytics/escp
coolsunglasses 12y ago
Replicating this could be a good way for me to exercise the (okay) index and (very poor at the moment) mapping support in https://github.com/bitemyapp/bloodhound https://github.com/bitemyapp/bloodhound
Argorak 12y ago
It really needs some cleanup and some extractions for public use, but we have a set of rake tasks to do all that and a bit more (e.g. reindexing into a new index and flipping the associated alias) https://github.com/asquera/elasticsearch-rake-tasks https://github.com/asquera/elasticsearch-rake-tasks
rhoml 12y ago
So whats the difference from this and the embedded backup restore tools on ES 1.0.1?
- lukeholder 12y ago
  snapshot/restore doesn't allow applying new mappings.

69 points by evantahler 12y ago

I haven't used this yet, but it seems like it should make updating mappings much easier. See the "using aliases for greater flexibility" section in this article: http://www.elasticsearch.org/blog/changing-mapping-with-zero-downtime/ http://www.elasticsearch.org/blog/changing-mapping-with-zero... I had to write the scripts to implement this concept myself and it wasn't a quick and easy task. It would had gone along a lot easier if I was able to abstract away some of those queries with a tool like this.

evantahler 12y ago
We use index mappings at TaskRabbit, and it does make things a lot easier. It helps with 0-downtime mapping changes (as you point out), and it also lets us combine large indexes. We segment large data collections by time (ie: events_year_month) which we roll up in an alias. This allows us to delete/add large collections without downtime as well. What's cool about the elsasticsearch api (and therefore the elasticdump tool) is that with aliases, you can read/write to and from an index just like you can an alias.

deleted 12y ago

[deleted]

andrewvc 12y ago

This is a neat tool for moving data around, but for serious stuff I'm not sure why you would use this over the official snapshot / restore API (http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-snapshots.html http://www.elasticsearch.org/guide/en/elasticsearch/referenc...). While the API there is a bit more cumbersome, it's one less dependency, and should be more efficient. For the specific use case of backups, the official API is even less cumbersome, esp. if you're using the cloud plugin and the S3 export option.

schmichael 12y ago
snapshot/restore doesn't allow applying new mappings. Also I use our similar custom tool* for dev purposes where you might want to kill the copy after a while and just work with a subset of the data... or explicitly apply a filter to copy a subset of data. * https://github.com/lytics/escp https://github.com/lytics/escp
- Xylakant 12y ago
  on top of that you can apply transformations to the data if you use scroll and bulk, but not with dump/restore. On the flip side, dump/restore allows incremental transfer which is a bit of work to do if you're implementing your own copy script.

schmichael 12y ago

While we don't have a spiffy logo, we made a tool for copying indexes around for testing/dev: https://github.com/lytics/escp https://github.com/lytics/escp

coolsunglasses 12y ago

Replicating this could be a good way for me to exercise the (okay) index and (very poor at the moment) mapping support in https://github.com/bitemyapp/bloodhound https://github.com/bitemyapp/bloodhound

Argorak 12y ago

It really needs some cleanup and some extractions for public use, but we have a set of rake tasks to do all that and a bit more (e.g. reindexing into a new index and flipping the associated alias) https://github.com/asquera/elasticsearch-rake-tasks https://github.com/asquera/elasticsearch-rake-tasks

rhoml 12y ago

So whats the difference from this and the embedded backup restore tools on ES 1.0.1?

lukeholder 12y ago
snapshot/restore doesn't allow applying new mappings.