4 ms·

One thing which is not clear about kafka or kinesis is when you have multiple consumers for the same topic how will they get the data and in what order , and wh

One thing which is not clear about kafka or kinesis is when you have multiple consumers for the same topic how will they get the data and in what order , and what happens when consumers die down. How do you handle consumers in your data pipeline ?

timclark 12y ago
Apache Kafka describes the ordering of message arrival to consumers in its documentation, I think it is even covered in the introduction (http://kafka.apache.org/documentation.html#introduction http://kafka.apache.org/documentation.html#introduction).
easytiger 12y ago
On the contrary, Ordering is something that is VERY clear in kafka, > By having a notion of parallelism—the partition—within the topics, Kafka is able to provide both ordering guarantees and load balancing over a pool of consumer processes. This is achieved by assigning the partitions in the topic to the consumers in the consumer group so that each partition is consumed by exactly one consumer in the group. By doing this we ensure that the consumer is the only reader of that partition and consumes the data in order. Since there are many partitions this still balances the load over many consumer instances. Note however that there cannot be more consumer instances than partitions. http://kafka.apache.org/documentation.html http://kafka.apache.org/documentation.html
- ewencp 12y ago
  In particular, http://kafka.apache.org/documentation.html#intro_consumers http://kafka.apache.org/documentation.html#intro_consumers addresses the concept of consumer groups and what ordering is guaranteed. One thing that might be worth noting for the grandparent is that Kafka consumers have an offset commit API that gives some control over how failures are handled. If a consumer dies before committing an offset but after reading data from the broker, a fresh consumer that joins the consumer group can see the same data once the system determines the original has died; that ensures all data will be processed, even in the event of consumer failures. Kinesis provides the same ordering guarantees. They use different terminology (Kafka topics == Kinesis streams; Kafka partitions == Kinesis shards) but have the same system interface. The details of the APIs used for consumption differ, but they provide the same basic functionality of Kafka's "consumer groups".

timclark 12y ago

Apache Kafka describes the ordering of message arrival to consumers in its documentation, I think it is even covered in the introduction (http://kafka.apache.org/documentation.html#introduction http://kafka.apache.org/documentation.html#introduction).

easytiger 12y ago

On the contrary, Ordering is something that is VERY clear in kafka, > By having a notion of parallelism—the partition—within the topics, Kafka is able to provide both ordering guarantees and load balancing over a pool of consumer processes. This is achieved by assigning the partitions in the topic to the consumers in the consumer group so that each partition is consumed by exactly one consumer in the group. By doing this we ensure that the consumer is the only reader of that partition and consumes the data in order. Since there are many partitions this still balances the load over many consumer instances. Note however that there cannot be more consumer instances than partitions. http://kafka.apache.org/documentation.html http://kafka.apache.org/documentation.html

ewencp 12y ago
In particular, http://kafka.apache.org/documentation.html#intro_consumers http://kafka.apache.org/documentation.html#intro_consumers addresses the concept of consumer groups and what ordering is guaranteed. One thing that might be worth noting for the grandparent is that Kafka consumers have an offset commit API that gives some control over how failures are handled. If a consumer dies before committing an offset but after reading data from the broker, a fresh consumer that joins the consumer group can see the same data once the system determines the original has died; that ensures all data will be processed, even in the event of consumer failures. Kinesis provides the same ordering guarantees. They use different terminology (Kafka topics == Kinesis streams; Kafka partitions == Kinesis shards) but have the same system interface. The details of the APIs used for consumption differ, but they provide the same basic functionality of Kafka's "consumer groups".