4 ms·
We wanted to use the KCL on the receiver side, but figured that it required building some sort of cluster to mediate which node would receive which shard. Trick
by strictfp 8y ago
We wanted to use the KCL on the receiver side, but figured that it required building some sort of cluster to mediate which node would receive which shard. Tricky, especially when resharding. With Lambda, you get that for free. So we opted for lambda, but weren't overly happy. Did you solve that problem?
- thisone 8y agoThe KCL itself handles shard assignment and balancing. Each application has its own dynamodb table where it stores lease information. If you have 64 shards and want that processed on one machine? Spin up one instance of the application. The KCL will spin up 64 sub processes. Instead want to spread the load? Spin up as many instances of the application you want, they don't need to be clustered, they will negotiate, using the lease table to take and balance the shards across all instances.
- thisone 8y agoI'll add that the same goes for resharding. The KCL sees and handles the Parent/child shard relationship, getting a lease on the children, but waiting until the parent is "drained" before reading from the children. And it handles this through the shenanigans that occur when resharding in non-multiple/evenly-divisible numbers