4 ms·

The good thing of people using for small problems overkill queue systems like Kafka or RabbitMQ is that they usually have time to react and correct the mistake

The good thing of people using for small problems overkill queue systems like Kafka or RabbitMQ is that they usually have time to react and correct the mistake after burning lots of engineering resources. The sooner they make the mistake, the better.

dtech 8y ago
Is something like RabbitMQ really overkill though? When I started out with a project that needed a durable queue and some inter-process communication I bought it as a SaaS, within 90 minutes the application was happily sending/consuming messages on a durable queue. I mean, is PostgreSQL really overkill just because a CSV file can also do some things?
- faragon 8y ago
  It depends on your definition of "small problem".
- majidazimi 8y ago
  The problem with RabbitMQ is that, you can't rewind back. It's good for messaging. But if you need log-like abstraction you need Kafka which requires zookeeper to start with. I would appreciate single node message brokers (very much like RabbitMQ) that don't do any fault-tolerance and just focus on providing log abstraction.
  - nitrogen 8y ago
    I've seen the ability to rewind cause problems in practice (accidental rewinding causing redelivery of old messages). It's something to be used cautiously, only if you really need it.
- theptip 8y ago
  I agree with this, RabbitMQ is a standard part of the Django stack (with Celery) and it's probably been the easiest component to manage in my experience. I have a RabbitMQ docker image that took 10 mins to stand up, and has been running in GKE for 2.5 years now without any downtime; it Just Works (TM). We don't drive it hard in terms of messages/second, it's just there delegating work from the web server to workers. I strongly dispute the "overkill" narrative for this component, thinking about it like a SQL database is the right framing. It's really easy to set up for a simple app, and you can turn on features to scale when you need them. But you really don't pay much of a cost to use this component no matter how small your project is.

dtech 8y ago

Is something like RabbitMQ really overkill though? When I started out with a project that needed a durable queue and some inter-process communication I bought it as a SaaS, within 90 minutes the application was happily sending/consuming messages on a durable queue. I mean, is PostgreSQL really overkill just because a CSV file can also do some things?

faragon 8y ago
It depends on your definition of "small problem".
majidazimi 8y ago
The problem with RabbitMQ is that, you can't rewind back. It's good for messaging. But if you need log-like abstraction you need Kafka which requires zookeeper to start with. I would appreciate single node message brokers (very much like RabbitMQ) that don't do any fault-tolerance and just focus on providing log abstraction.
- nitrogen 8y ago
  I've seen the ability to rewind cause problems in practice (accidental rewinding causing redelivery of old messages). It's something to be used cautiously, only if you really need it.
theptip 8y ago
I agree with this, RabbitMQ is a standard part of the Django stack (with Celery) and it's probably been the easiest component to manage in my experience. I have a RabbitMQ docker image that took 10 mins to stand up, and has been running in GKE for 2.5 years now without any downtime; it Just Works (TM). We don't drive it hard in terms of messages/second, it's just there delegating work from the web server to workers. I strongly dispute the "overkill" narrative for this component, thinking about it like a SQL database is the right framing. It's really easy to set up for a simple app, and you can turn on features to scale when you need them. But you really don't pay much of a cost to use this component no matter how small your project is.