4 ms·
Have you also evaluated Kafka? as it is a common choice feeding Storm .
by yashinovsky 12y ago
Have you also evaluated Kafka? as it is a common choice feeding Storm .
- sargun 12y agoKafka is a log, not a queue. They have every so slightly different semantics, but logs have HOL blocking per partition for processing messages because they use a watermark, compared to queues, which track on a per-message basis.
- rohitnair 12y agoThe article already includes a couple of "not queues" that they use for event stream processing, Kinesis being one of them. Kinesis consumers also maintain a watermark to keep track of how far they've read in a stream. Considering that, a Kafka comparison seems like a reasonable ask.
- itaifrenkel 12y agoSure we have. Kinesis was considered because it was inspired by Kafka, and Kafka is known to work well. The thing is that any distributed stateful service, by design, requires DevOps experience and takes time to manage. We try to stay lean at this stage, and chose Kinesis over Kafka becuase it took very little time to setup, and has no "maintenance costs". Had we have chosen Kafka, we would have enjoyed much shorter latency times.
- eikenberry 12y agoThis is for AWS. Kafka isn't designed to work well in an environment where partitioning occurs with any frequency. AWS is such an environment. https://aphyr.com/posts/293-call-me-maybe-kafka https://aphyr.com/posts/293-call-me-maybe-kafka
- itaifrenkel 12y agoI always wondered if Amazon would really be open about and go public when a split brain in their network caused one of the hosted data services to loose data. Maybe they just throw money at the hardware and hope for the best.
- yashinovsky 12y agoFor that matter, RabbbitMQ isn't either. see https://aphyr.com/posts/315-call-me-maybe-rabbitmq https://aphyr.com/posts/315-call-me-maybe-rabbitmq .