4 ms·

Define "small scale". Keep in mind that Kafka was developed by LinkedIn to handle trillions of bytes of message ingest/transfer per day in real time (that's at

by sidlls 9y ago

Define "small scale". Keep in mind that Kafka was developed by LinkedIn to handle trillions of bytes of message ingest/transfer per day in real time (that's at least tens of MB per second across all boundaries, and probably much more during "bursty" periods). Is that you? If not, unless you just want to use Kafka as a learning experience/resume builder it's (probably) not necessary to even use it. If that isn't you and you are still intent on using it, I'd second the opinion to go with "let someone else (e.g. 'cloud provider') administer it". It's probably not really worth the effort to get configuration settings and installation details right unless you genuinely have a serious interest in it.

faizshah 9y ago
I'm looking for a small scale solution (several orders smaller than linkedin) for change data capture into an ordered queue that Flink or Spark can consume continuously. The solution needs to be reliable and highly available, latency needs to be reasonable and relatively consistent to keep aggregates relatively fresh. Do you have a recommendation?
- Xorlev 9y ago
  We "only" had 20K messages/s at my last job, and Kafka worked incredibly well for us. We ran 3 brokers and always had headroom for jobs which read the last N days of the log. We had a mix of realtime (continuous) consumers and batch consumers that consumed directly from Kafka, but also archived that data to S3 for future batch processing. Operations were quite simple for us, but we were also used to running Zookeeper before, so YMMV. If you can get away with Kinesis, it's simpler operationally, but Kafka isn't difficult compared to most systems.
  - faizshah 9y ago
    Thanks for sharing, what was your experience with node failure at that scale?
    - Xorlev 9y ago
      Short unavailability (10-20s) of the partitions whose leadership were assigned to that broker, followed by otherwise uninterrupted service. If it was a clean failure, no big deal. If the disk started having issues and taking a long time for IO to come back, all partitions with replicas or leaders on that node were slow. That's kind of the state of the world with most distributed things though: a bad node is worse than a dead node.
      - anonacct37 9y ago
        Another experience report. I work work a team that handles gigabytes per second and one stage of the pipeline involves sending data from a nsq to Kafka. The nsq nodes take exponentially less resources and empirically have far greater reliability. Odd because in terms of functionality I care about, both systems do the same thing. "no leader for partition" is something I see frequently. In Kafka's defence a better managed cluster might do better. In nsq's defence: nobody has to manage it, it just runs.
        Xorlev 9y ago
        I mean, they're kind of apples and oranges. nsq is a primarily in-memory (minus the fallback to disk if too many messages are in flight) pubsub service fundamentally designed for spreading load and availability. There's no delivery or ordering guarantees, no replication. Kafka is a distributed, partitioned log. Messages published to a partition in a given order will be stored in that order and replicated in that order, and consumed in that same order. Kafka writes all of your data to disk, and waits for replicas to acknowledge it (depending on your producer configuration). Kafka does considerably more work than nsq. I like nsq, it works well, but they aren't at all designed to solve the same problems. I mentioned the issues above because they'd happened, but in years of running Kafka it was never Kafka's fault. > "no leader for partition" is something I see frequently. Something is wrong with your cluster if it's constantly switching leadership. Leadership should be stable -- and furthermore, partitions have preferred leaders, so they should stay with specific nodes until that node goes down. Kafka has a process which shifts leadership back to preferred leaders automatically (unless it's off). Check your logs.
- threeseed 9y ago
  Remember that both Kafka and Kinesis have "at least once" delivery and from experience it's a common scenario. You will receive multiple messages. Two options: (1) store hashes of each row and then deduplicate, (2) use a database. I can't remember if it's an issue for change capture since in theory you have some insert timestamp column anyway.
  - cmatta 9y ago
    Kafka has "exactly once" delivery guarantees since 0.11, I believe this will be the default in Kafka 1.0 (I work for Confluent).

by sidlls 9y ago

faizshah 9y ago

I'm looking for a small scale solution (several orders smaller than linkedin) for change data capture into an ordered queue that Flink or Spark can consume continuously. The solution needs to be reliable and highly available, latency needs to be reasonable and relatively consistent to keep aggregates relatively fresh. Do you have a recommendation?

Xorlev 9y ago
We "only" had 20K messages/s at my last job, and Kafka worked incredibly well for us. We ran 3 brokers and always had headroom for jobs which read the last N days of the log. We had a mix of realtime (continuous) consumers and batch consumers that consumed directly from Kafka, but also archived that data to S3 for future batch processing. Operations were quite simple for us, but we were also used to running Zookeeper before, so YMMV. If you can get away with Kinesis, it's simpler operationally, but Kafka isn't difficult compared to most systems.
- faizshah 9y ago
  Thanks for sharing, what was your experience with node failure at that scale?
  - Xorlev 9y ago
    Short unavailability (10-20s) of the partitions whose leadership were assigned to that broker, followed by otherwise uninterrupted service. If it was a clean failure, no big deal. If the disk started having issues and taking a long time for IO to come back, all partitions with replicas or leaders on that node were slow. That's kind of the state of the world with most distributed things though: a bad node is worse than a dead node.
    - anonacct37 9y ago
      Another experience report. I work work a team that handles gigabytes per second and one stage of the pipeline involves sending data from a nsq to Kafka. The nsq nodes take exponentially less resources and empirically have far greater reliability. Odd because in terms of functionality I care about, both systems do the same thing. "no leader for partition" is something I see frequently. In Kafka's defence a better managed cluster might do better. In nsq's defence: nobody has to manage it, it just runs.
      - Xorlev 9y ago
        I mean, they're kind of apples and oranges. nsq is a primarily in-memory (minus the fallback to disk if too many messages are in flight) pubsub service fundamentally designed for spreading load and availability. There's no delivery or ordering guarantees, no replication. Kafka is a distributed, partitioned log. Messages published to a partition in a given order will be stored in that order and replicated in that order, and consumed in that same order. Kafka writes all of your data to disk, and waits for replicas to acknowledge it (depending on your producer configuration). Kafka does considerably more work than nsq. I like nsq, it works well, but they aren't at all designed to solve the same problems. I mentioned the issues above because they'd happened, but in years of running Kafka it was never Kafka's fault. > "no leader for partition" is something I see frequently. Something is wrong with your cluster if it's constantly switching leadership. Leadership should be stable -- and furthermore, partitions have preferred leaders, so they should stay with specific nodes until that node goes down. Kafka has a process which shifts leadership back to preferred leaders automatically (unless it's off). Check your logs.
threeseed 9y ago
Remember that both Kafka and Kinesis have "at least once" delivery and from experience it's a common scenario. You will receive multiple messages. Two options: (1) store hashes of each row and then deduplicate, (2) use a database. I can't remember if it's an issue for change capture since in theory you have some insert timestamp column anyway.
- cmatta 9y ago
  Kafka has "exactly once" delivery guarantees since 0.11, I believe this will be the default in Kafka 1.0 (I work for Confluent).