5 ms·

> However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype

> However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster. While calling this out on a message board comment section is going to be well-received, asking "do we need this" while working at the company with said architect/tech lead is not well-received. How many of us get paid to work jobs where we're basically told "shut up, this is what we're doing/using, go with it"?

rad_gruchalski 4y ago
And go for mqtt instead? Not a smart choice. Look, it’s going to work for a few hundred, few thousand topics. But as soon as you need resilience, replication, or you outgrow that one broker… good luck. Mqtt is awful to scale horizontally.
- justinclift 4y ago
  Maybe something AMQP related instead?
  - rad_gruchalski 4y ago
    Those are two different technologies. Amqp is all about routing and queues. Kafka is a distributed log, it is not a queue. There’s a significant difference between those two. Kafka: every consumer for a partition within a consumer group will see a message at least once. A queue: it’s possible that a partition has multiple consumers and only one consumer sees a particular message. Kafka is relatively small to medium number of large volume topics. Topics can be larger the a machine due to partitioning. Strict ordering per partition based on message arrival time. Queues (RabbitMQ or anything amqp) are relatively large number or small to medium volume topics. Ordering is an option, a topic must fit within a machine. Thos are orthogonal concepts. They can live next to each other. My first choice is always Kafka because: persistence, replication, scalability. Works fine as a single broker, can always scale horizontally, zookeeper is not that scary, especially with a good operator. If you think that Kafka is difficult to run, wait until you need replication in RabbitMQ. Good luck with leader election in your application layer. No fun.
    - MuffinFlavored 4y ago
      > Kafka is a distributed log When should you use Kafka instead of storing rows in SQL with a timestamp so you can replay them/fetch them if needed? Why do you need a sharded Kafka cluster? Most businesses are going to have Redis, SQL, and probably RabbitMQ. Where/why add Kafka to that stack?
      - manv1 4y ago
        Kafka was designed for places with a huge data/log volume (GB/sec or more). From what I understand it's generally The Thing To Use when you have not very many (ie: thousands) high-volume sources that have good links. You push stuff into the logs and you can take your time chewing through the log. MQTT was designed for shitloads (hundreds of thousands+) of small devices on bad links connecting to the mothership. Neither of them really are a message queue. I mean, they queue messages, but they have different goals than, say, ActiveMQ or RabbitMQ (which both can be used as a backend for MQTT). Using Kafka for a message queue is overkill. It's more likely that MQTT fits your bill.
      - rad_gruchalski 4y ago
        > When should you use Kafka instead of storing rows in SQL with a timestamp so you can replay them/fetch them if needed? Lots to unpack here. The simple answer is: you use Kafka when you need to have the data in order in arrival written to disk but you don't need ad-hoc query. Messages must be persisted, processed at least once by someone, in order. It's common to have Kafka in front of a database as a sort of buffer: when data is in Kafka, it has been acknowledged as seen and will be processed later. Now, why would you store it in Kafka rather than in a table with timestamps. I said in my previous comment that messages are persisted based on the arrival time on the broker. This is somewhat inaccurate. Kafka is not related in a message arrival time. It's interested in message arrival. All messages arriving on a partition are process in order of arrival. Messages are stored on disk as byes, each message is written to a segment file, a partition will often have more than one segment file. Segments are sequential. A message is written at a byte offset within a segment and the format is something like: {metadata, length, message body}, the protocol is documented here: https://kafka.apache.org/protocol https://kafka.apache.org/protocol. Each segment file has an index file. The index file maps the partition message offset to a byte offset within a segment file. To read a message, the broker needs to know the message sequence you want. The broker will identify the segment by looking at (AFAIR) the index file name which contains the first message offset within its respective segment. It will read the index, find the byte offset of the message, check the message length, read the message, and return it. However, as you can imagine, this is a pretty involved scenario because it implies a bunch of random reads from disk. The point of Kafka is sequential reads over complete partitions or segments. So Kafka shines when you need to read large sequences from a partition. It will identify a segment, memory map it and do a sequential read over a large file (usually a segment is something like 1GB). This is different than inserting into an SQL table with timestamps. First of all - you will have duplicated timestamps. The only way to avoid duplication is auto-incremental indexes. These will be similar to the message offset. However, a large number of writers will inevitably lead to a congestion on your auto-incremented sequences. Next, your table will have indexes: as you want ACID, those indexes will have to be updated before data is considered stored (C). Next, you have the data stored but it's an SQL database, so your data is on one node. This is different than Kafka minimum replication factor where the data is considered as stored when at least N replicas confirm it. As for reads: so your client application would store it's own last processed sequence number from your auto-incremented index, and this would work the same as the consumer offset in Kafka. But it you have hundreds of GBs of data on your server, you'll be hitting disk pretty often for index scans, or your indexes will outgrow your RAM. Sure, you can scale to a bigger box (costly) or you go for a DRDBMS (YBDB or Cockroach or similar) but due to how your tables are distributed in those systems, you will hit a latency cliff. Reads are usually better than writes, writes can be atrocious, DRDBMS large batch writes can be 500x+ slower than single node RDBMS. > so you can replay them/fetch them if needed? If you need ad-hoc queries, that's the best way to go. But for volumes of data larger than your SQL server, I'd consider Kafka in front and SQL to store only what needs to be queried. > Why do you need a sharded Kafka cluster? Three reasons: 1. Write performance. I wrote higher that messages are written in order into a sequence file. I have personally worked with clusters ingesting 400Mbps per partition on multiple partitions on a single node. Basically, Kafka can saturate your bandwidth and if you have SSDs, everything what's ingested, will be persisted because Kafka uses zero-copy and memory mapped files. So it's basically like writing to RAM. If you need to ingest more data than a single node can handle, you need more machines. 2. Read performance: data within a topic partition is written in sequence and will be read in sequence. Your typical scenario if using some derivative of a key to establish the partition number. By default kafka uses murmur2 hash. What this basically means. Imagine that you have data for email addresses. The email address is the key. All data for a given email address ends up on the same partition. Thus your consumer will always read from one partition for a given email address. The more partitions you have, the higher is the number of consumers. Of course one there will be multiple email addresses within a single partition because their hash maps to the same partition number. The higher the number of partitions, the higher the granularity of that distribution. However, it's easy to hit a hotspot for a key if you have a lot of data for a hot key. 3. I would argue, the most important one: replication. If you have 3 brokers and your topic has a replication factor of 3, you have one copy of data per physical machine, all data consistent across brokers. That is - each partition and every segment exists in three copies, one copy per physical broker. There will be multiple partitions for the same topic on the same broker and each will be replicated to other brokers. Taking a broker out does not impact your operations. You can still read and write, you will have under-replicated partitions but Kafka will automatically recover once the missing broker comes back. If a broker falls out, and that broker was a leader for a partition, Kafka will elect a new leader from one of the remaining brokers and your consumers and producers will automatically continue writing like nothing happened. > Most businesses are going to have Redis, SQL, and probably RabbitMQ. > Where/why add Kafka to that stack? Kafka is an all-purpose strictly-ordered-within-partition buffer of data with automated replication and leader election. Multiple consumers can read from the same topic partitions at the same time. Each one has its own offset and can consume at its own pace. Each consumer can do different things with those messages at the same time. Those consumers can be located at different org levels. Kafka is great at moving large volumes of data within the organization and facilitating other processes, for example ETL. Man, I sound like I'm some Confluent salesman. I'm not affiliated. I work for Klarrio, we build streaming systems.
  - ilyt 4y ago
    Architecturally AMQP is just bigger brother of MQTT, to the point some queue brokers (like RabbitMQ) can accept both. MQTT 5 made difference smaller too.
    - rad_gruchalski 4y ago
      Ha, but do you want AMQP 0.9.1 or 1.0? Completely different protocols. And do you need a router? A broker? An L7 load balancer? So maybe you need Apache ActiveMQ next to your RabbitMQ with AMQP 1.0 plugin? Or Qpid? Or all three of them? What I liked about a mix of three of those was that I could connect Azure Event Hub with RabbitMQ and ActiveMQ, they would all speak to each other via Qpid.
- jonquark 4y ago
  This comment seems backwards to me. If you're funnelling incoming messages in to hundreds of topics (or less) Kafka is a great "fat pipe" if you need millions (or tens of millions) of topics for IoT devices, MQTT is much more designed for that usecase Disclosure: I'm biased - I've worked on the MQTT spec and I'm the lead for Eclipse Amlen
  - rad_gruchalski 4y ago
    You are absolutely right. That's what I said in my sibling comment so I'm not sure what's backwards. However, if your millions of little topics don't fit on a single machine - what do you do? You need a fat pipe. Hence you put your little topics into the fat pipe, send over the fat pipe to other mqtt brokers that need to disseminate egress. Example - you have 1 topic with 1 producer and 20M consumers. Each consumer is a tcp connection. Say that you can do C1M happily, you still need 20 brokers to serve egress for all your connected clients. Now imagine that you have 100 brokers, 100M connected clients and your connections are randomly distributed over your brokers. You don't want to route every message to every broker because. So you need a fat pipe and some middle man that knows which brokers a message must be routed to because there are currently consumers connected to those brokers waiting subscribed to topics and waiting for messages. As someone who works on MQTT, you for sure understand the problem. I have never heard of Eclipse Amlen. However, I am working with MQTT and Kafka since 2012 and have seen a nation-wide successful MQTT rollouts where MQTT and Kafka worked in tandem to solve exactly the problem you are talking about - millions of little concurrent connections distributed over a large fleet of devices for sub-100ms round trip. It's not a competition, it's a coopoeration.
    - jonquark 4y ago
      I agree there are cases where Kafka and MQTT are often used together. If you have lots of MQTT clients producing messages fanned-in to a small number of backend apps (or consuming a small number of wide fan out messages) people often combine Kafka as a fat pipe behind MQTT brokers (though there are alternatives, consuming messages from the brokers using e.g. MQTTv5 shared subs) In more complicated situations (e.g. "outgoing" persistent messages buffered for individual clients (i.e. each client has a "message inbox"), Kafka is less obviously useful (it's an anti-pattern to try and random-seek the messages from Kafka topics as clients connect). In this kind of pattern, the main architecture pattern I see are clients partitioned across (highly available) MQTT brokers. If the messages come from MQTT clients directly to other clients (e.g. instant messaging (Facebook Messenger uses MQTT), having these broker in a cluster sharing a topic tree is very useful. If the outgoing messages don't get buffered whilst the clients are off-line (e.g. because these are responses to client requests) then you don't need each client to be routed to a "home" broker that buffers the messages - it can connect anywhere. It all depends on the shape of the message flow you're designing the system to support (but to say MQTT is a toy for small numbers of topics as it seemed to me that you argued up-thread seems misguided).
- ilyt 4y ago
  Soooo right around when you have actual business with actual customers and use cases and are in best moment to re-architecture your app properly instead of throwing infrastructure at wall and seeing what sticks Then just shove MQTT topics into kafka and slowly migrate stuff over.
  - rad_gruchalski 4y ago
    I worked on a smart home platform where we did exactly that. But we moved from RabbitMQ to Kafka. This was a Spring Boot microservice architecture. We moved all of it without changing the service consume/publish semantics. The added benefit was that to facilitate the move, we wrote full integration tests on real containers and we have replaced RabbitMQ with Kafka under those tests. The system handled much better under load and the company had full integration tests. The migration took 2 weeks with 8 weeks writing tests for a team of 3.

rad_gruchalski 4y ago

And go for mqtt instead? Not a smart choice. Look, it’s going to work for a few hundred, few thousand topics. But as soon as you need resilience, replication, or you outgrow that one broker… good luck. Mqtt is awful to scale horizontally.

justinclift 4y ago
Maybe something AMQP related instead?
- rad_gruchalski 4y ago
  Those are two different technologies. Amqp is all about routing and queues. Kafka is a distributed log, it is not a queue. There’s a significant difference between those two. Kafka: every consumer for a partition within a consumer group will see a message at least once. A queue: it’s possible that a partition has multiple consumers and only one consumer sees a particular message. Kafka is relatively small to medium number of large volume topics. Topics can be larger the a machine due to partitioning. Strict ordering per partition based on message arrival time. Queues (RabbitMQ or anything amqp) are relatively large number or small to medium volume topics. Ordering is an option, a topic must fit within a machine. Thos are orthogonal concepts. They can live next to each other. My first choice is always Kafka because: persistence, replication, scalability. Works fine as a single broker, can always scale horizontally, zookeeper is not that scary, especially with a good operator. If you think that Kafka is difficult to run, wait until you need replication in RabbitMQ. Good luck with leader election in your application layer. No fun.
  - MuffinFlavored 4y ago
    > Kafka is a distributed log When should you use Kafka instead of storing rows in SQL with a timestamp so you can replay them/fetch them if needed? Why do you need a sharded Kafka cluster? Most businesses are going to have Redis, SQL, and probably RabbitMQ. Where/why add Kafka to that stack?
    - manv1 4y ago
      Kafka was designed for places with a huge data/log volume (GB/sec or more). From what I understand it's generally The Thing To Use when you have not very many (ie: thousands) high-volume sources that have good links. You push stuff into the logs and you can take your time chewing through the log. MQTT was designed for shitloads (hundreds of thousands+) of small devices on bad links connecting to the mothership. Neither of them really are a message queue. I mean, they queue messages, but they have different goals than, say, ActiveMQ or RabbitMQ (which both can be used as a backend for MQTT). Using Kafka for a message queue is overkill. It's more likely that MQTT fits your bill.
    - rad_gruchalski 4y ago
      > When should you use Kafka instead of storing rows in SQL with a timestamp so you can replay them/fetch them if needed? Lots to unpack here. The simple answer is: you use Kafka when you need to have the data in order in arrival written to disk but you don't need ad-hoc query. Messages must be persisted, processed at least once by someone, in order. It's common to have Kafka in front of a database as a sort of buffer: when data is in Kafka, it has been acknowledged as seen and will be processed later. Now, why would you store it in Kafka rather than in a table with timestamps. I said in my previous comment that messages are persisted based on the arrival time on the broker. This is somewhat inaccurate. Kafka is not related in a message arrival time. It's interested in message arrival. All messages arriving on a partition are process in order of arrival. Messages are stored on disk as byes, each message is written to a segment file, a partition will often have more than one segment file. Segments are sequential. A message is written at a byte offset within a segment and the format is something like: {metadata, length, message body}, the protocol is documented here: https://kafka.apache.org/protocol https://kafka.apache.org/protocol. Each segment file has an index file. The index file maps the partition message offset to a byte offset within a segment file. To read a message, the broker needs to know the message sequence you want. The broker will identify the segment by looking at (AFAIR) the index file name which contains the first message offset within its respective segment. It will read the index, find the byte offset of the message, check the message length, read the message, and return it. However, as you can imagine, this is a pretty involved scenario because it implies a bunch of random reads from disk. The point of Kafka is sequential reads over complete partitions or segments. So Kafka shines when you need to read large sequences from a partition. It will identify a segment, memory map it and do a sequential read over a large file (usually a segment is something like 1GB). This is different than inserting into an SQL table with timestamps. First of all - you will have duplicated timestamps. The only way to avoid duplication is auto-incremental indexes. These will be similar to the message offset. However, a large number of writers will inevitably lead to a congestion on your auto-incremented sequences. Next, your table will have indexes: as you want ACID, those indexes will have to be updated before data is considered stored (C). Next, you have the data stored but it's an SQL database, so your data is on one node. This is different than Kafka minimum replication factor where the data is considered as stored when at least N replicas confirm it. As for reads: so your client application would store it's own last processed sequence number from your auto-incremented index, and this would work the same as the consumer offset in Kafka. But it you have hundreds of GBs of data on your server, you'll be hitting disk pretty often for index scans, or your indexes will outgrow your RAM. Sure, you can scale to a bigger box (costly) or you go for a DRDBMS (YBDB or Cockroach or similar) but due to how your tables are distributed in those systems, you will hit a latency cliff. Reads are usually better than writes, writes can be atrocious, DRDBMS large batch writes can be 500x+ slower than single node RDBMS. > so you can replay them/fetch them if needed? If you need ad-hoc queries, that's the best way to go. But for volumes of data larger than your SQL server, I'd consider Kafka in front and SQL to store only what needs to be queried. > Why do you need a sharded Kafka cluster? Three reasons: 1. Write performance. I wrote higher that messages are written in order into a sequence file. I have personally worked with clusters ingesting 400Mbps per partition on multiple partitions on a single node. Basically, Kafka can saturate your bandwidth and if you have SSDs, everything what's ingested, will be persisted because Kafka uses zero-copy and memory mapped files. So it's basically like writing to RAM. If you need to ingest more data than a single node can handle, you need more machines. 2. Read performance: data within a topic partition is written in sequence and will be read in sequence. Your typical scenario if using some derivative of a key to establish the partition number. By default kafka uses murmur2 hash. What this basically means. Imagine that you have data for email addresses. The email address is the key. All data for a given email address ends up on the same partition. Thus your consumer will always read from one partition for a given email address. The more partitions you have, the higher is the number of consumers. Of course one there will be multiple email addresses within a single partition because their hash maps to the same partition number. The higher the number of partitions, the higher the granularity of that distribution. However, it's easy to hit a hotspot for a key if you have a lot of data for a hot key. 3. I would argue, the most important one: replication. If you have 3 brokers and your topic has a replication factor of 3, you have one copy of data per physical machine, all data consistent across brokers. That is - each partition and every segment exists in three copies, one copy per physical broker. There will be multiple partitions for the same topic on the same broker and each will be replicated to other brokers. Taking a broker out does not impact your operations. You can still read and write, you will have under-replicated partitions but Kafka will automatically recover once the missing broker comes back. If a broker falls out, and that broker was a leader for a partition, Kafka will elect a new leader from one of the remaining brokers and your consumers and producers will automatically continue writing like nothing happened. > Most businesses are going to have Redis, SQL, and probably RabbitMQ. > Where/why add Kafka to that stack? Kafka is an all-purpose strictly-ordered-within-partition buffer of data with automated replication and leader election. Multiple consumers can read from the same topic partitions at the same time. Each one has its own offset and can consume at its own pace. Each consumer can do different things with those messages at the same time. Those consumers can be located at different org levels. Kafka is great at moving large volumes of data within the organization and facilitating other processes, for example ETL. Man, I sound like I'm some Confluent salesman. I'm not affiliated. I work for Klarrio, we build streaming systems.
- ilyt 4y ago
  Architecturally AMQP is just bigger brother of MQTT, to the point some queue brokers (like RabbitMQ) can accept both. MQTT 5 made difference smaller too.
  - rad_gruchalski 4y ago
    Ha, but do you want AMQP 0.9.1 or 1.0? Completely different protocols. And do you need a router? A broker? An L7 load balancer? So maybe you need Apache ActiveMQ next to your RabbitMQ with AMQP 1.0 plugin? Or Qpid? Or all three of them? What I liked about a mix of three of those was that I could connect Azure Event Hub with RabbitMQ and ActiveMQ, they would all speak to each other via Qpid.
jonquark 4y ago
This comment seems backwards to me. If you're funnelling incoming messages in to hundreds of topics (or less) Kafka is a great "fat pipe" if you need millions (or tens of millions) of topics for IoT devices, MQTT is much more designed for that usecase Disclosure: I'm biased - I've worked on the MQTT spec and I'm the lead for Eclipse Amlen
- rad_gruchalski 4y ago
  You are absolutely right. That's what I said in my sibling comment so I'm not sure what's backwards. However, if your millions of little topics don't fit on a single machine - what do you do? You need a fat pipe. Hence you put your little topics into the fat pipe, send over the fat pipe to other mqtt brokers that need to disseminate egress. Example - you have 1 topic with 1 producer and 20M consumers. Each consumer is a tcp connection. Say that you can do C1M happily, you still need 20 brokers to serve egress for all your connected clients. Now imagine that you have 100 brokers, 100M connected clients and your connections are randomly distributed over your brokers. You don't want to route every message to every broker because. So you need a fat pipe and some middle man that knows which brokers a message must be routed to because there are currently consumers connected to those brokers waiting subscribed to topics and waiting for messages. As someone who works on MQTT, you for sure understand the problem. I have never heard of Eclipse Amlen. However, I am working with MQTT and Kafka since 2012 and have seen a nation-wide successful MQTT rollouts where MQTT and Kafka worked in tandem to solve exactly the problem you are talking about - millions of little concurrent connections distributed over a large fleet of devices for sub-100ms round trip. It's not a competition, it's a coopoeration.
  - jonquark 4y ago
    I agree there are cases where Kafka and MQTT are often used together. If you have lots of MQTT clients producing messages fanned-in to a small number of backend apps (or consuming a small number of wide fan out messages) people often combine Kafka as a fat pipe behind MQTT brokers (though there are alternatives, consuming messages from the brokers using e.g. MQTTv5 shared subs) In more complicated situations (e.g. "outgoing" persistent messages buffered for individual clients (i.e. each client has a "message inbox"), Kafka is less obviously useful (it's an anti-pattern to try and random-seek the messages from Kafka topics as clients connect). In this kind of pattern, the main architecture pattern I see are clients partitioned across (highly available) MQTT brokers. If the messages come from MQTT clients directly to other clients (e.g. instant messaging (Facebook Messenger uses MQTT), having these broker in a cluster sharing a topic tree is very useful. If the outgoing messages don't get buffered whilst the clients are off-line (e.g. because these are responses to client requests) then you don't need each client to be routed to a "home" broker that buffers the messages - it can connect anywhere. It all depends on the shape of the message flow you're designing the system to support (but to say MQTT is a toy for small numbers of topics as it seemed to me that you argued up-thread seems misguided).
ilyt 4y ago
Soooo right around when you have actual business with actual customers and use cases and are in best moment to re-architecture your app properly instead of throwing infrastructure at wall and seeing what sticks Then just shove MQTT topics into kafka and slowly migrate stuff over.
- rad_gruchalski 4y ago
  I worked on a smart home platform where we did exactly that. But we moved from RabbitMQ to Kafka. This was a Spring Boot microservice architecture. We moved all of it without changing the service consume/publish semantics. The added benefit was that to facilitate the move, we wrote full integration tests on real containers and we have replaced RabbitMQ with Kafka under those tests. The system handled much better under load and the company had full integration tests. The migration took 2 weeks with 8 weeks writing tests for a team of 3.