3 ms·

FWIW, we implemented dynamic consensus membership change in Kudu way back in 2015 (https://github.com/apache/kudu/commit/535dae https://github.com/apache/kudu/c

by mpercy 8y ago

FWIW, we implemented dynamic consensus membership change in Kudu way back in 2015 (https://github.com/apache/kudu/commit/535dae https://github.com/apache/kudu/commit/535dae) but presumably that was after the fork. We still haven't implemented leader leases or distributed transactions in Kudu though due to prioritizing other features. It's very cool that you have implemented those consistency features.

kmuthukk 8y ago
hi @mpercy, Thanks for correcting me on the dynamic consensus membership change. Looks like the basic support was indeed there, but several important enhancements were needed (for correctness and usability). - To make the "online" piece of the membership change work correctly we added support for LEARNER (PRE VOTER) role (where the new member enters in a non-voting mode till it's caught up). https://github.com/YugaByte/yugabyte-db/commit/909d26e31ecd0ef0f87eb677961dcf238f9d7853 https://github.com/YugaByte/yugabyte-db/commit/909d26e31ecd0.... - Load Balancing (which uses the membership changes) is automatic. (https://github.com/YugaByte/yugabyte-db/commit/e4667eb7ec0e6b870eeb6a8cc34273fe1b9b576b https://github.com/YugaByte/yugabyte-db/commit/e4667eb7ec0e6...) - Remote bootstrap (due to membership changes) also has undergone substantial changes given that YugaByte DB uses a customize/extended version of RocksDB as the storage engine and does a tighter coupling of Raft with RocksDB storage engine. (https://github.com/YugaByte/yugabyte-db/blob/master/docs/extending-rocksdb.md https://github.com/YugaByte/yugabyte-db/blob/master/docs/ext...) - Dynamic Leader Balancing is also new-- it causes leadership to be proactively altered in a running system to ensure each node is the leader for a similar number of tablets. regards, Kannan
- mpercy 8y ago
  Interesting. Just last year we implemented improved re-replication (https://github.com/apache/kudu/commit/79a255 https://github.com/apache/kudu/commit/79a255) which sounds very similar to what you did with LEARNER roles, and we added manually-triggered rebalancing (https://github.com/apache/kudu/commit/ccdcf6 https://github.com/apache/kudu/commit/ccdcf6 and https://kudu.apache.org/releases/1.8.0/docs/administration.html#rebalancer_tool https://kudu.apache.org/releases/1.8.0/docs/administration.h...). I'm curious if you did anything to prevent automatic rebalancing from being triggered at a "bad time" or have throttled it in some way, or whether moving large amounts of data between servers at arbitrary times was not a concern. I am also curious if you added some type of API using the LEARNER role to support a CDC-type of listener interface using consensus. By the way, we also recently added support for rack/location awareness in a series of patches including https://github.com/apache/kudu/commit/ebb285 https://github.com/apache/kudu/commit/ebb285 We should really start some threads on the dev lists to periodically share this type of information and merge things back and forth to avoid duplicating work where possible. I know the systems are pretty different at the catalog and storage layers but there are still many similarities.
  - kmuthukk 8y ago
    Yes - load-balancing is automatic and turned on by default. Two key protections we did there were: - A per-node rate limit for sending/receiving on behalf of remote bootstrap (https://github.com/YugaByte/yugabyte-db/commit/154810cc5f5ddfb4c29633f3bfc4bfe8fa6ab596 https://github.com/YugaByte/yugabyte-db/commit/154810cc5f5dd...) - A global throttle on how many quorum membership changes are in flight cluster wide. On the CDC question - we added an OBSERVER role as well (a non-voting quorum member) to support for features like read-replicas. This is currently not exposed via an external API - but CDC is something we are working on now. Happy to exchange notes with your team! regards, Kannan
  - kmuthukk 8y ago
    Forgot to respond to your: << we also recently added support for rack/location awareness in a series of patches including https://github.com/apache/kudu/commit/ebb285 https://github.com/apache/kudu/commit/ebb285 >> Nice! We too ended up adding similar support for multi-AZ/multi-region aware replication configurations starting here (https://github.com/YugaByte/yugabyte-db/commit/c0ced5dd934facfc02f38cd0d98785b48043f00a https://github.com/YugaByte/yugabyte-db/commit/c0ced5dd934fa...) regards, Kannan

by mpercy 8y ago

kmuthukk 8y ago

hi @mpercy, Thanks for correcting me on the dynamic consensus membership change. Looks like the basic support was indeed there, but several important enhancements were needed (for correctness and usability). - To make the "online" piece of the membership change work correctly we added support for LEARNER (PRE VOTER) role (where the new member enters in a non-voting mode till it's caught up). https://github.com/YugaByte/yugabyte-db/commit/909d26e31ecd0ef0f87eb677961dcf238f9d7853 https://github.com/YugaByte/yugabyte-db/commit/909d26e31ecd0.... - Load Balancing (which uses the membership changes) is automatic. (https://github.com/YugaByte/yugabyte-db/commit/e4667eb7ec0e6b870eeb6a8cc34273fe1b9b576b https://github.com/YugaByte/yugabyte-db/commit/e4667eb7ec0e6...) - Remote bootstrap (due to membership changes) also has undergone substantial changes given that YugaByte DB uses a customize/extended version of RocksDB as the storage engine and does a tighter coupling of Raft with RocksDB storage engine. (https://github.com/YugaByte/yugabyte-db/blob/master/docs/extending-rocksdb.md https://github.com/YugaByte/yugabyte-db/blob/master/docs/ext...) - Dynamic Leader Balancing is also new-- it causes leadership to be proactively altered in a running system to ensure each node is the leader for a similar number of tablets. regards, Kannan

mpercy 8y ago
Interesting. Just last year we implemented improved re-replication (https://github.com/apache/kudu/commit/79a255 https://github.com/apache/kudu/commit/79a255) which sounds very similar to what you did with LEARNER roles, and we added manually-triggered rebalancing (https://github.com/apache/kudu/commit/ccdcf6 https://github.com/apache/kudu/commit/ccdcf6 and https://kudu.apache.org/releases/1.8.0/docs/administration.html#rebalancer_tool https://kudu.apache.org/releases/1.8.0/docs/administration.h...). I'm curious if you did anything to prevent automatic rebalancing from being triggered at a "bad time" or have throttled it in some way, or whether moving large amounts of data between servers at arbitrary times was not a concern. I am also curious if you added some type of API using the LEARNER role to support a CDC-type of listener interface using consensus. By the way, we also recently added support for rack/location awareness in a series of patches including https://github.com/apache/kudu/commit/ebb285 https://github.com/apache/kudu/commit/ebb285 We should really start some threads on the dev lists to periodically share this type of information and merge things back and forth to avoid duplicating work where possible. I know the systems are pretty different at the catalog and storage layers but there are still many similarities.
- kmuthukk 8y ago
  Yes - load-balancing is automatic and turned on by default. Two key protections we did there were: - A per-node rate limit for sending/receiving on behalf of remote bootstrap (https://github.com/YugaByte/yugabyte-db/commit/154810cc5f5ddfb4c29633f3bfc4bfe8fa6ab596 https://github.com/YugaByte/yugabyte-db/commit/154810cc5f5dd...) - A global throttle on how many quorum membership changes are in flight cluster wide. On the CDC question - we added an OBSERVER role as well (a non-voting quorum member) to support for features like read-replicas. This is currently not exposed via an external API - but CDC is something we are working on now. Happy to exchange notes with your team! regards, Kannan
- kmuthukk 8y ago
  Forgot to respond to your: << we also recently added support for rack/location awareness in a series of patches including https://github.com/apache/kudu/commit/ebb285 https://github.com/apache/kudu/commit/ebb285 >> Nice! We too ended up adding similar support for multi-AZ/multi-region aware replication configurations starting here (https://github.com/YugaByte/yugabyte-db/commit/c0ced5dd934facfc02f38cd0d98785b48043f00a https://github.com/YugaByte/yugabyte-db/commit/c0ced5dd934fa...) regards, Kannan