3 ms·

Thank you for this. I'm currently involved in the design of a large-ish IoT backend having only worked on smaller volume systems using "traditional" brokers (MQ

Thank you for this. I'm currently involved in the design of a large-ish IoT backend having only worked on smaller volume systems using "traditional" brokers (MQTT, RabbitMQ). Your post helped me fit Kafka into my mental image. What do you think about an incremental development where one starts with direct MQTT subscribers and adds Kafka only once the volume goes up?

rad_gruchalski 4y ago
Without knowing much of your exact requirements, I would aim for the following: devices communicate over MQTT, support one protocol, do it well. The problem with MQTT is scaling the broker. It all depends on what is behind "large-ish". Hundreds of thousands of devices? Millions? Dozens of millions? If you can fit all connections on one broker, it's easy, any available solution will handle this. Going past one broker is where the problem starts. Imagine the following scenario: You have 10 brokers, each broker handles 1M connections. You have a TopicA subscriber on broker 9 and 10, someone publishes a message to TopicA on broker 2. Your broker 2 needs to forward the message to brokers 9 and 10 but no others because if you do this for every message, you waste a ton of compute. So you need a method to distribute your connection table: which brokers currently have connections with subscriptions to TopicA. This is reasonably easy to embed in the broker for a relatively small number of brokers but if you need hundreds of brokers, your global connection table will not fit on each broker. Scale this up - millions of topics and millions of connected clients. You need a way to offload the connection table somewhere else, find a reliable method to distribute changes to your connection table. This is where we have successfully used Kafka - distributing the connection table and state changes.
- sally_glance 4y ago
  Got it - I think for the first 1-2 years one broker instance should be plenty. So the decision to up the complexity can probably be postponed to the point where it becomes clear that we hit 10M+/lots of brokers will be required.
  - rad_gruchalski 4y ago
    Definitely. Focus on solving the business problem first, grow the business. When you have the need to make a switch, you will find a way forward by rebuilding your your MQTT infrastructure to fit your scale. If you ever need to.
    - sally_glance 4y ago
      Yeah - business is smart home renewable energy stuff, not lightbulbs. If they don't go international real fast the small oldschool broker cluster solution will probably be plenty for the next 5 years.

rad_gruchalski 4y ago

Without knowing much of your exact requirements, I would aim for the following: devices communicate over MQTT, support one protocol, do it well. The problem with MQTT is scaling the broker. It all depends on what is behind "large-ish". Hundreds of thousands of devices? Millions? Dozens of millions? If you can fit all connections on one broker, it's easy, any available solution will handle this. Going past one broker is where the problem starts. Imagine the following scenario: You have 10 brokers, each broker handles 1M connections. You have a TopicA subscriber on broker 9 and 10, someone publishes a message to TopicA on broker 2. Your broker 2 needs to forward the message to brokers 9 and 10 but no others because if you do this for every message, you waste a ton of compute. So you need a method to distribute your connection table: which brokers currently have connections with subscriptions to TopicA. This is reasonably easy to embed in the broker for a relatively small number of brokers but if you need hundreds of brokers, your global connection table will not fit on each broker. Scale this up - millions of topics and millions of connected clients. You need a way to offload the connection table somewhere else, find a reliable method to distribute changes to your connection table. This is where we have successfully used Kafka - distributing the connection table and state changes.

sally_glance 4y ago
Got it - I think for the first 1-2 years one broker instance should be plenty. So the decision to up the complexity can probably be postponed to the point where it becomes clear that we hit 10M+/lots of brokers will be required.
- rad_gruchalski 4y ago
  Definitely. Focus on solving the business problem first, grow the business. When you have the need to make a switch, you will find a way forward by rebuilding your your MQTT infrastructure to fit your scale. If you ever need to.
  - sally_glance 4y ago
    Yeah - business is smart home renewable energy stuff, not lightbulbs. If they don't go international real fast the small oldschool broker cluster solution will probably be plenty for the next 5 years.