3 ms·
Kafka is less message queue (rabbitmq, sqs) and more ordered stream/write ahead log (ala kinesis) > The first is that because of the nature of message queues,
by hughesjj 3y ago
Kafka is less message queue (rabbitmq, sqs) and more ordered stream/write ahead log (ala kinesis)
> The first is that because of the nature of message queues, they're either empty or full.
... Wat?
> that is it's a true "fire and forget" kind of request and direct response time is not a concern, then what does the queue gain you over simply posting it to a database table?
Performance is why. The fire and forget aspect is like udp in the sense that you don't need to ensure ordering of messages (packets) or hard persistence to the database. Also, dead letter queues exist for a reason.
Message queues are super useful. The highest performance systems I've seen use the message queue + pool of workers paradigm, as it allows you to better smooth your load (unlike immediate republishing like in sns, which requires hardware available to accept) with minimal guarantees (unlike a write ahead log such as kinesis). The buffer is also great because it allows you a bit more time to scale up both your message queue fleet and worker fleet when you get a load spike.
- thephyber 3y ago> > The first is that because of the nature of message queues, they're either empty or full. > ... Wat? I interpreted it as "they are either trending towards empty or full". The statement doesn't seem well thought through. That might be true (either empty or full) most of the time (maybe) _if you squint_, but the entire point of the Message Queue is to provide buffering from the transient state (somewhere between empty and full) trending toward the empty state.
- Raidion 3y agoYea, I feel like these people never got a chron daily data dump (times N customers). Scaling is dead simple, and there isn't a need (or ability) to instantaneously handle large bursty workloads in like 99% of cases. Longer SLAs mean it's also easier to hit those SLAs. Giving the clients realistic SLAs is super important.