4 ms·
The KCL itself handles shard assignment and balancing. Each application has its own dynamodb table where it stores lease information. If you have 64 shards and
by thisone 8y ago
The KCL itself handles shard assignment and balancing. Each application has its own dynamodb table where it stores lease information. If you have 64 shards and want that processed on one machine? Spin up one instance of the application. The KCL will spin up 64 sub processes.
Instead want to spread the load? Spin up as many instances of the application you want, they don't need to be clustered, they will negotiate, using the lease table to take and balance the shards across all instances.
- thisone 8y agoI'll add that the same goes for resharding. The KCL sees and handles the Parent/child shard relationship, getting a lease on the children, but waiting until the parent is "drained" before reading from the children. And it handles this through the shenanigans that occur when resharding in non-multiple/evenly-divisible numbers