4 ms·
Interesting. Just last year we implemented improved re-replication (https://github.com/apache/kudu/commit/79a255 https://github.com/apache/kudu/commit/79a255) w
by mpercy 8y ago
Interesting. Just last year we implemented improved re-replication (https://github.com/apache/kudu/commit/79a255 https://github.com/apache/kudu/commit/79a255) which sounds very similar to what you did with LEARNER roles, and we added manually-triggered rebalancing (https://github.com/apache/kudu/commit/ccdcf6 https://github.com/apache/kudu/commit/ccdcf6 and https://kudu.apache.org/releases/1.8.0/docs/administration.html#rebalancer_tool https://kudu.apache.org/releases/1.8.0/docs/administration.h...).
I'm curious if you did anything to prevent automatic rebalancing from being triggered at a "bad time" or have throttled it in some way, or whether moving large amounts of data between servers at arbitrary times was not a concern.
I am also curious if you added some type of API using the LEARNER role to support a CDC-type of listener interface using consensus.
By the way, we also recently added support for rack/location awareness in a series of patches including https://github.com/apache/kudu/commit/ebb285 https://github.com/apache/kudu/commit/ebb285
We should really start some threads on the dev lists to periodically share this type of information and merge things back and forth to avoid duplicating work where possible. I know the systems are pretty different at the catalog and storage layers but there are still many similarities.
- kmuthukk 8y agoYes - load-balancing is automatic and turned on by default. Two key protections we did there were: - A per-node rate limit for sending/receiving on behalf of remote bootstrap (https://github.com/YugaByte/yugabyte-db/commit/154810cc5f5ddfb4c29633f3bfc4bfe8fa6ab596 https://github.com/YugaByte/yugabyte-db/commit/154810cc5f5dd...) - A global throttle on how many quorum membership changes are in flight cluster wide. On the CDC question - we added an OBSERVER role as well (a non-voting quorum member) to support for features like read-replicas. This is currently not exposed via an external API - but CDC is something we are working on now. Happy to exchange notes with your team! regards, Kannan
- kmuthukk 8y agoForgot to respond to your: << we also recently added support for rack/location awareness in a series of patches including https://github.com/apache/kudu/commit/ebb285 https://github.com/apache/kudu/commit/ebb285 >> Nice! We too ended up adding similar support for multi-AZ/multi-region aware replication configurations starting here (https://github.com/YugaByte/yugabyte-db/commit/c0ced5dd934facfc02f38cd0d98785b48043f00a https://github.com/YugaByte/yugabyte-db/commit/c0ced5dd934fa...) regards, Kannan