4 ms·

How to Choose the Right Message Queue for Your Microservices

7 points by mertyildiran 5y ago

kentm 5y ago
> A synchronous approach is more error-prone, harder to debug, and harder to recover. Many industry professionals pass this option over. That's the complete opposite of my experience.
- zactato 5y ago
  Yeah I was also surprised by that line. I’ll give the author “harder to recover” but definitely not “harder to debug”
EdwardDiego 5y ago
On Kafka... Message retrieval is push? Nope, definitely pull. Payload is text or avro? Neither, it's just bytes, how you serialise your payload data is up to you. > Client libraries might have a hard-time of keeping up with ever-changing protocol specification. Therefore upgrades might also be challenging. Kafka maintains backwards compatibility for old clients for a very long time. They've only just (Kafka 3.0) deprecated the protocol released in Kafka 0.8. Choose a client library that has active maintainers, you'll be golden. If your client lib of choose > Message size limit is 1MB. It defaults to this, but it can be set much higher if needed. Lastly - Kafka's not really a message queue. Apache Pulsar offers far more MQ like semantics though.
- geoduck14 5y ago
  >> Message size limit is 1MB. It defaults to this, but it can be set much higher if needed. What is it? I was surprised to hear it was this low.
  - EdwardDiego 5y ago
    It's set to 1MB (and some additional bytes for log overhead) because it was found (by LinkedIn? I think?) to be a reasonable default for throughput. Technically you can set it up to anywhere near MAX_INT bytes, would be interesting to see what happens when you send a 1GiB message. :D You'd most likely need to tune other properties depending on how big you're going. And message size, just to be clear, relates to a message from a producer which can contain 1 to N records, often compressed to reduce network contention. The 1MB limit applies to the bytes which are received by the broker, so you can send batches of records which are >1MB if you're compressing them. Compression imposes some CPU cost on producers and consumers, but unless you've specifically configured a topic to use a mismatching compression algorithm (this defaults to "just accept whatever algorithm the producer" used), there's no cost on the broker, it's written as is, and then consumers pull it still compressed.
windows2020 5y ago
A good old fashioned database queue usually does it for me. It's easy for the DBA to monitor, Ops to run reports on, not a new program, nor another service to pay for and doesn't add additional proprietary lock-in. Plus, devs already have it setup and running.

7 points by mertyildiran 5y ago

> A synchronous approach is more error-prone, harder to debug, and harder to recover. Many industry professionals pass this option over. That's the complete opposite of my experience.

zactato 5y ago
Yeah I was also surprised by that line. I’ll give the author “harder to recover” but definitely not “harder to debug”

EdwardDiego 5y ago

On Kafka... Message retrieval is push? Nope, definitely pull. Payload is text or avro? Neither, it's just bytes, how you serialise your payload data is up to you. > Client libraries might have a hard-time of keeping up with ever-changing protocol specification. Therefore upgrades might also be challenging. Kafka maintains backwards compatibility for old clients for a very long time. They've only just (Kafka 3.0) deprecated the protocol released in Kafka 0.8. Choose a client library that has active maintainers, you'll be golden. If your client lib of choose > Message size limit is 1MB. It defaults to this, but it can be set much higher if needed. Lastly - Kafka's not really a message queue. Apache Pulsar offers far more MQ like semantics though.

geoduck14 5y ago
>> Message size limit is 1MB. It defaults to this, but it can be set much higher if needed. What is it? I was surprised to hear it was this low.
- EdwardDiego 5y ago
  It's set to 1MB (and some additional bytes for log overhead) because it was found (by LinkedIn? I think?) to be a reasonable default for throughput. Technically you can set it up to anywhere near MAX_INT bytes, would be interesting to see what happens when you send a 1GiB message. :D You'd most likely need to tune other properties depending on how big you're going. And message size, just to be clear, relates to a message from a producer which can contain 1 to N records, often compressed to reduce network contention. The 1MB limit applies to the bytes which are received by the broker, so you can send batches of records which are >1MB if you're compressing them. Compression imposes some CPU cost on producers and consumers, but unless you've specifically configured a topic to use a mismatching compression algorithm (this defaults to "just accept whatever algorithm the producer" used), there's no cost on the broker, it's written as is, and then consumers pull it still compressed.

windows2020 5y ago

A good old fashioned database queue usually does it for me. It's easy for the DBA to monitor, Ops to run reports on, not a new program, nor another service to pay for and doesn't add additional proprietary lock-in. Plus, devs already have it setup and running.