4 ms·
Kafka is at least a bit simpler than RabbitMQ, though both are very square-ish shaped pegs and are usually forced into very round-ish holes. People regularly th
by pas 4y ago
Kafka is at least a bit simpler than RabbitMQ, though both are very square-ish shaped pegs and are usually forced into very round-ish holes. People regularly think they need a message queue, when they really need a job queue, or message bus. Or even all three, but then they try to hack everything on top of Kafka (or RMQ) ... which can be done, of course, but "results may vary".
For tracking state at scale (but still per-job, per-thing) a Cassandra-like system works best (but preferably a better implementation, eg. SkyllaDB or AeroSpike or some other KV store).
- davydog187 4y agoLol to “ Kafka is at least a bit simpler than RabbitMQ”. I’m sorry, what universe does this statement live in?
- Spivak 4y agoTrue, it’s actually a lot simpler than RabbitMQ. People seem to assume “giant ball of enterprisey Java means” that the experience using it will be complicated. Kafka is extremely reliable, simple to cluster, battle tested (I haven’t hit an actual bug in Kafka in ages), the self-healing is turnkey, and has way stronger guarantees for clients. Where it bites people is that it’s not a queue and scaling is harder than just add more consumers.
- rawgabbit 4y agoI am curious if anyone has used confluent.io (kafka as a service). Or is it too cost prohibitive?
- d8tltanc 4y agoJava is a bit simpler than erlang
- CharlieDigital 4y ago> People regularly think they need a message queue, when they really need a job queue, or message bus This is one of the reasons I am a really, really big fan of Google Cloud's Task Queues. It allows the stupidest, simplest temporal execution of HTTP invocations. Currently working on a project in AWS and it's stunning how complicated it is to achieve the same simple need of "I want to execute this HTTP call at this time in the future". It's either AmazonMQ -- using either ActiveMQ or RabbitMQ with plugins -- or hacking around SQS's 15 minute delay limit. In our case, we are going to end up wrapping our messages in an envelope with a delivery time and if it hasn't met the delivery time, we put it back into SQS. GCP is highly underrated for how it simplifies control over execution of code. Pub/Sub and Task Queues both have HTTP delivery built in. Couple that with Google Cloud Run and it is a recipe for building almost any type of execution model with much less complexity and overhead
- grncdr 4y agoIn case you want to avoid an extra envelope, you can also add custom headers to SQS messages. This can be handy if you want to implement that delay hack without parsing message bodies.
- dariusj18 4y agoI looked it up but couldn't really find information, what would you say is the difference between a "message queue", a "job queue" and a "message bus"?
- pas 4y agoI just throw them on the wall based on my experiences, maybe they have some agreed upon precise definitions, but I'm not aware :o message bus: firehose of events, (by default) no ACK. usually multi producer multi consumer. (see also DBus which is more of an RPC layer + service discovery + pub/sub via event listeners) message queue: usually between components, ACK, but no selective ACK, backpressure, might even have support for "dead letters" (letters not ACKed by any consumer) job queue: selective ACK, retry, etc. (there's also the "enterprise service bus", which is similar, but mostly implemented on things like IBM MQ)
- deleted 4y ago[deleted]