4 ms·
If the OP doesn't mind expanding a little on this bit, I'd be grateful. > If the dedupe worker crashes for any reason or encounters an error from Kafka, when i
by iampims 9y ago
If the OP doesn't mind expanding a little on this bit, I'd be grateful.
> If the dedupe worker crashes for any reason or encounters an error from Kafka, when it re-starts it will first consult the “source of truth” for whether an event was published: the output topic.
Does this mean that "on worker crash" the worker replays the entire output topic and compare it to the rocksdb dataset?
Also, how do you handle scaling up or down the number of workers/partitions?
- ryanworl 9y agoI'm not the OP, but changing the number of Kafka partitions isn't a super graceful operation. You would be wise to add as many as you could reasonably need assuming one consumer thread per partition. But not too many because each one is at least two files on disk!
- tedmiston 9y agoI haven't used Kafka yet, but a Kafka partition is roughly the same as a Kinesis shard, right?
- ryanworl 9y agoYes, but you have to set them up upfront and there is no API to split and merge partitions. If you add more partitions, it doesn't automatically re-shard for you either. I don't think it should automatically re-shard as it would cause a ton of disk and network IO, but just something to be aware of.