5 ms·

> Elasticsearch supports automatic shard rebalancing, which would let us add new nodes to the cluster, fulfilling the linearly scalable requirement out of the b

by jdc0589 10y ago

> Elasticsearch supports automatic shard rebalancing, which would let us add new nodes to the cluster, fulfilling the linearly scalable requirement out of the box. I'm assuming they mean "I can add a new node to the cluster, and NEW SHARDS FROM NEW INDEXES can be distributed amongst the new nodes".... So far as I know elasticsearch can't rebalance shard location or composition automatically based on cluster membership change events...right? I mean, the cluster reroute api will let you manually move a shard, but thats all I know of.

jhgg 10y ago
ES can indeed rebalance shards to new nodes as they join the cluster. More here: https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html https://www.elastic.co/guide/en/elasticsearch/reference/curr...
- jdc0589 10y ago
  huh, the more you know. not sure how I've missed this. I'll definitely look for some discussions/use cases of how this actually plays out.
  - jazoom 10y ago
    I suspect you were thinking of changing the number of shards, which can be an important part of rebalancing an index. Unfortunately, Elasticsearch can't do that.
    - jonaf 10y ago
      There is a convenient reindex API available in more recent versions, although in most cases where you can rebuild your index anyway, it's faster to do that.
      - jazoom 10y ago
        Ahh. I was wondering when they were going to get around to implementing that. I might have to upgrade to 5.x soon.
        wsgolfer 10y ago
        FYI, the reindex API was added as experimental in 2.3, so it's been there for about a year [1]. Even as experimental it worked great. [1] https://www.elastic.co/guide/en/elasticsearch/reference/2.3/docs-reindex.html https://www.elastic.co/guide/en/elasticsearch/reference/2.3/...
        jazoom 10y ago
        Ha. I just came here to edit my comment. I remember seeing it before and just checked the docs again. In 5.2 it's STILL considered experiential. Craziness. It took me about an hour to build The basic functionality myself.
        jasontedor 10y ago
        Thank you for noticing this, it is not considered experimental anymore. I integrated a change[1] to update the docs; it should percolate through to the site within thirty minutes. [1]: https://github.com/elastic/elasticsearch/pull/23621 https://github.com/elastic/elasticsearch/pull/23621
        jazoom 10y ago
        Great work. It appears to have already percolated.
        jdc0589 10y ago
        FYI, the re-index API is awesome.
pram 10y ago
Yes that's actually one of my favorite ES features. Kafka doesn't do any type of automatic topic partition rebalancing which was quite shocking to me when I first started using it. Seems kind of primitive in comparison.
packetized 10y ago
Tuning cluster rebalance settings can be... challenging. I've done some bad, bad things with a cluster where it would effectively balance things back and forth - forever. Fiddling with cluster.routing.allocation.balance.* can really paint you into a corner quickly.

by jdc0589 10y ago

jhgg 10y ago

ES can indeed rebalance shards to new nodes as they join the cluster. More here: https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html https://www.elastic.co/guide/en/elasticsearch/reference/curr...

jdc0589 10y ago
huh, the more you know. not sure how I've missed this. I'll definitely look for some discussions/use cases of how this actually plays out.
- jazoom 10y ago
  I suspect you were thinking of changing the number of shards, which can be an important part of rebalancing an index. Unfortunately, Elasticsearch can't do that.
  - jonaf 10y ago
    There is a convenient reindex API available in more recent versions, although in most cases where you can rebuild your index anyway, it's faster to do that.
    - jazoom 10y ago
      Ahh. I was wondering when they were going to get around to implementing that. I might have to upgrade to 5.x soon.
      - wsgolfer 10y ago
        FYI, the reindex API was added as experimental in 2.3, so it's been there for about a year [1]. Even as experimental it worked great. [1] https://www.elastic.co/guide/en/elasticsearch/reference/2.3/docs-reindex.html https://www.elastic.co/guide/en/elasticsearch/reference/2.3/...
        jazoom 10y ago
        Ha. I just came here to edit my comment. I remember seeing it before and just checked the docs again. In 5.2 it's STILL considered experiential. Craziness. It took me about an hour to build The basic functionality myself.
        jasontedor 10y ago
        Thank you for noticing this, it is not considered experimental anymore. I integrated a change[1] to update the docs; it should percolate through to the site within thirty minutes. [1]: https://github.com/elastic/elasticsearch/pull/23621 https://github.com/elastic/elasticsearch/pull/23621
        jazoom 10y ago
        Great work. It appears to have already percolated.
        jdc0589 10y ago
        FYI, the re-index API is awesome.

pram 10y ago

Yes that's actually one of my favorite ES features. Kafka doesn't do any type of automatic topic partition rebalancing which was quite shocking to me when I first started using it. Seems kind of primitive in comparison.

packetized 10y ago

Tuning cluster rebalance settings can be... challenging. I've done some bad, bad things with a cluster where it would effectively balance things back and forth - forever. Fiddling with cluster.routing.allocation.balance.* can really paint you into a corner quickly.