3 ms·

From everything I've read Kafka is a really bad fit for AWS. It is not tolerant of partitioning. They stated this in their own design document where they presen

From everything I've read Kafka is a really bad fit for AWS. It is not tolerant of partitioning. They stated this in their own design document where they present it as a CA system. In his Jepsen post on Kafka, Kyle backed this up with more data. Given this, why do people deploy it to AWS? It seems like an invitation to disaster.

bbrazil 12y ago
What does AWS have to do with it? Partitions aren't the exclusive domain of AWS. http://aphyr.com/posts/288-the-network-is-reliable http://aphyr.com/posts/288-the-network-is-reliable
- eikenberry 12y ago
  No, they are just well known for having plentiful partitioning issues. If you are in your own data centers, you can plan for carefully for this and make it a very low concern.
noelwelsh 12y ago
I think you are vastly overstating the issue. The issue in the Jepsen post (http://aphyr.com/posts/293-call-me-maybe-kafka http://aphyr.com/posts/293-call-me-maybe-kafka) requires a complex failure case to cause data loss. For most people using Kafka, a little data loss very infrequently, while still accepting writes, is acceptable. The alternative is to refuse all writes till the system is back. This in unacceptable for typical use cases. (See http://blog.empathybox.com/post/62279088548/a-few-notes-on-kafka-and-jepsen http://blog.empathybox.com/post/62279088548/a-few-notes-on-k...)
- justinsb 12y ago
  It's a trade-off: C vs A. If you can't lose data, Kafka's current configuration is unacceptable. There's an open issue, but it isn't fixed yet, and this means Kafka is (currently) unsuitable e.g. for use as a transaction log. Kafka is great if your data is lower value (e.g. web analytics). It's a pity, because it would be great to have an open source component for the Kafka architecture, but where data-loss is unacceptable. Hopefully this will be fixed in Kafka itself; I've also been working on something that is based around Raft.
Rapzid 12y ago
For nearly the same price(about 2300/month) you can get 200MB/s worth of Kinesis shards.
- erichmond 12y ago
  Can you explain this a little more? We've been investigating kafka v kinesis, and the severe restrictions on kinesis' API has been giving us pause. http://docs.aws.amazon.com/kinesis/latest/dev/service-sizes-and-limits.html http://docs.aws.amazon.com/kinesis/latest/dev/service-sizes-...
  - Rapzid 12y ago
    Hi erichmond. I'm sorry if that was a bit off-the-cuff; I was simply pointing out the price equivalence of the resources. I can however make a SOLID suggestion that any time somebody sees resource limits on AWS that might hold them back they should get in touch with their account manager(or solutions architect, etc) to see what can be done. There are very few hard limits. Most of the soft-limits are to prevent people's bills from exploding. As far as the API itself though, as is usually the case you might need to get a bit creative to design around AWS imposed constraints due to their design decisions. It might end up not feasible, or not worth it, for your particular use case. But I believe it's at least worth investigating/discussing.

bbrazil 12y ago

What does AWS have to do with it? Partitions aren't the exclusive domain of AWS. http://aphyr.com/posts/288-the-network-is-reliable http://aphyr.com/posts/288-the-network-is-reliable

eikenberry 12y ago
No, they are just well known for having plentiful partitioning issues. If you are in your own data centers, you can plan for carefully for this and make it a very low concern.

noelwelsh 12y ago

I think you are vastly overstating the issue. The issue in the Jepsen post (http://aphyr.com/posts/293-call-me-maybe-kafka http://aphyr.com/posts/293-call-me-maybe-kafka) requires a complex failure case to cause data loss. For most people using Kafka, a little data loss very infrequently, while still accepting writes, is acceptable. The alternative is to refuse all writes till the system is back. This in unacceptable for typical use cases. (See http://blog.empathybox.com/post/62279088548/a-few-notes-on-kafka-and-jepsen http://blog.empathybox.com/post/62279088548/a-few-notes-on-k...)

justinsb 12y ago
It's a trade-off: C vs A. If you can't lose data, Kafka's current configuration is unacceptable. There's an open issue, but it isn't fixed yet, and this means Kafka is (currently) unsuitable e.g. for use as a transaction log. Kafka is great if your data is lower value (e.g. web analytics). It's a pity, because it would be great to have an open source component for the Kafka architecture, but where data-loss is unacceptable. Hopefully this will be fixed in Kafka itself; I've also been working on something that is based around Raft.

Rapzid 12y ago

For nearly the same price(about 2300/month) you can get 200MB/s worth of Kinesis shards.

erichmond 12y ago
Can you explain this a little more? We've been investigating kafka v kinesis, and the severe restrictions on kinesis' API has been giving us pause. http://docs.aws.amazon.com/kinesis/latest/dev/service-sizes-and-limits.html http://docs.aws.amazon.com/kinesis/latest/dev/service-sizes-...
- Rapzid 12y ago
  Hi erichmond. I'm sorry if that was a bit off-the-cuff; I was simply pointing out the price equivalence of the resources. I can however make a SOLID suggestion that any time somebody sees resource limits on AWS that might hold them back they should get in touch with their account manager(or solutions architect, etc) to see what can be done. There are very few hard limits. Most of the soft-limits are to prevent people's bills from exploding. As far as the API itself though, as is usually the case you might need to get a bit creative to design around AWS imposed constraints due to their design decisions. It might end up not feasible, or not worth it, for your particular use case. But I believe it's at least worth investigating/discussing.