3 ms·

hi @mpercy, Thanks for correcting me on the dynamic consensus membership change. Looks like the basic support was indeed there, but several important enhanceme

hi @mpercy, Thanks for correcting me on the dynamic consensus membership change. Looks like the basic support was indeed there, but several important enhancements were needed (for correctness and usability). - To make the "online" piece of the membership change work correctly we added support for LEARNER (PRE VOTER) role (where the new member enters in a non-voting mode till it's caught up). https://github.com/YugaByte/yugabyte-db/commit/909d26e31ecd0ef0f87eb677961dcf238f9d7853 https://github.com/YugaByte/yugabyte-db/commit/909d26e31ecd0.... - Load Balancing (which uses the membership changes) is automatic. (https://github.com/YugaByte/yugabyte-db/commit/e4667eb7ec0e6b870eeb6a8cc34273fe1b9b576b https://github.com/YugaByte/yugabyte-db/commit/e4667eb7ec0e6...) - Remote bootstrap (due to membership changes) also has undergone substantial changes given that YugaByte DB uses a customize/extended version of RocksDB as the storage engine and does a tighter coupling of Raft with RocksDB storage engine. (https://github.com/YugaByte/yugabyte-db/blob/master/docs/extending-rocksdb.md https://github.com/YugaByte/yugabyte-db/blob/master/docs/ext...) - Dynamic Leader Balancing is also new-- it causes leadership to be proactively altered in a running system to ensure each node is the leader for a similar number of tablets. regards, Kannan

mpercy 8y ago
Interesting. Just last year we implemented improved re-replication (https://github.com/apache/kudu/commit/79a255 https://github.com/apache/kudu/commit/79a255) which sounds very similar to what you did with LEARNER roles, and we added manually-triggered rebalancing (https://github.com/apache/kudu/commit/ccdcf6 https://github.com/apache/kudu/commit/ccdcf6 and https://kudu.apache.org/releases/1.8.0/docs/administration.html#rebalancer_tool https://kudu.apache.org/releases/1.8.0/docs/administration.h...). I'm curious if you did anything to prevent automatic rebalancing from being triggered at a "bad time" or have throttled it in some way, or whether moving large amounts of data between servers at arbitrary times was not a concern. I am also curious if you added some type of API using the LEARNER role to support a CDC-type of listener interface using consensus. By the way, we also recently added support for rack/location awareness in a series of patches including https://github.com/apache/kudu/commit/ebb285 https://github.com/apache/kudu/commit/ebb285 We should really start some threads on the dev lists to periodically share this type of information and merge things back and forth to avoid duplicating work where possible. I know the systems are pretty different at the catalog and storage layers but there are still many similarities.
- kmuthukk 8y ago
  Yes - load-balancing is automatic and turned on by default. Two key protections we did there were: - A per-node rate limit for sending/receiving on behalf of remote bootstrap (https://github.com/YugaByte/yugabyte-db/commit/154810cc5f5ddfb4c29633f3bfc4bfe8fa6ab596 https://github.com/YugaByte/yugabyte-db/commit/154810cc5f5dd...) - A global throttle on how many quorum membership changes are in flight cluster wide. On the CDC question - we added an OBSERVER role as well (a non-voting quorum member) to support for features like read-replicas. This is currently not exposed via an external API - but CDC is something we are working on now. Happy to exchange notes with your team! regards, Kannan
- kmuthukk 8y ago
  Forgot to respond to your: << we also recently added support for rack/location awareness in a series of patches including https://github.com/apache/kudu/commit/ebb285 https://github.com/apache/kudu/commit/ebb285 >> Nice! We too ended up adding similar support for multi-AZ/multi-region aware replication configurations starting here (https://github.com/YugaByte/yugabyte-db/commit/c0ced5dd934facfc02f38cd0d98785b48043f00a https://github.com/YugaByte/yugabyte-db/commit/c0ced5dd934fa...) regards, Kannan

mpercy 8y ago

Interesting. Just last year we implemented improved re-replication (https://github.com/apache/kudu/commit/79a255 https://github.com/apache/kudu/commit/79a255) which sounds very similar to what you did with LEARNER roles, and we added manually-triggered rebalancing (https://github.com/apache/kudu/commit/ccdcf6 https://github.com/apache/kudu/commit/ccdcf6 and https://kudu.apache.org/releases/1.8.0/docs/administration.html#rebalancer_tool https://kudu.apache.org/releases/1.8.0/docs/administration.h...). I'm curious if you did anything to prevent automatic rebalancing from being triggered at a "bad time" or have throttled it in some way, or whether moving large amounts of data between servers at arbitrary times was not a concern. I am also curious if you added some type of API using the LEARNER role to support a CDC-type of listener interface using consensus. By the way, we also recently added support for rack/location awareness in a series of patches including https://github.com/apache/kudu/commit/ebb285 https://github.com/apache/kudu/commit/ebb285 We should really start some threads on the dev lists to periodically share this type of information and merge things back and forth to avoid duplicating work where possible. I know the systems are pretty different at the catalog and storage layers but there are still many similarities.

kmuthukk 8y ago
Yes - load-balancing is automatic and turned on by default. Two key protections we did there were: - A per-node rate limit for sending/receiving on behalf of remote bootstrap (https://github.com/YugaByte/yugabyte-db/commit/154810cc5f5ddfb4c29633f3bfc4bfe8fa6ab596 https://github.com/YugaByte/yugabyte-db/commit/154810cc5f5dd...) - A global throttle on how many quorum membership changes are in flight cluster wide. On the CDC question - we added an OBSERVER role as well (a non-voting quorum member) to support for features like read-replicas. This is currently not exposed via an external API - but CDC is something we are working on now. Happy to exchange notes with your team! regards, Kannan
kmuthukk 8y ago
Forgot to respond to your: << we also recently added support for rack/location awareness in a series of patches including https://github.com/apache/kudu/commit/ebb285 https://github.com/apache/kudu/commit/ebb285 >> Nice! We too ended up adding similar support for multi-AZ/multi-region aware replication configurations starting here (https://github.com/YugaByte/yugabyte-db/commit/c0ced5dd934facfc02f38cd0d98785b48043f00a https://github.com/YugaByte/yugabyte-db/commit/c0ced5dd934fa...) regards, Kannan