3 ms·

Fortunately all our jobs are idempotent, and I hope to keep it that way. Postgres keeps track of what's currently "in-flight" so to speak, so my plan for failur

Fortunately all our jobs are idempotent, and I hope to keep it that way. Postgres keeps track of what's currently "in-flight" so to speak, so my plan for failures is just to re-queue stuff. The good thing about sneakers is we're not locked into any particular routing scheme or message structure so it'll be easy to move away from once we eventually experience problems. So, community, am I being naive? Will the idempotency thing not save me? (Also, lobster_johnson when you say terrible in a virtualisation environment, are you referring to AWS-ish shared VPSs or to a containers-on-colocated-boxes solution, which is what we favor?)

lobster_johnson 12y ago
If you can determine what messages need to be re-queued, and easily re-queue them, then you should be fine. (That does raise the question of what you need a queue for, if your Postgres database keeps your state. Postgres can act as a pretty efficient queue — for one app we do this extensively — especially if you use listen/notify. It won't scale to thousands of requests/sec, but it's simpler to maintain than RabbitMQ, and has the benefit of being fully ACID.) Still, the RabbitMQ partitioning does exist and requires manual intervention whenever it happens. You can tell RabbitMQ to run in a mode where it will automatically recover from a partition by using the node which has the most connections, and discarding the data on the other conflicting nodes; but the problem in your case would be that you might not be able to tell when to re-queue your messages.
- JonnieCache 12y ago
  The database doesn't keep all the state, it just knows about which domain objects are currently "awaiting data," which are "processing," which are "failed" and which are "finished." Within the processing state there are several different things that go on, performed by various chained rabbit tasks. These push the data into elasticsearch, where the magic actually happens. My plan in a complete rabbitmq queue loss scenario is to simply fire the whole process off from the beginning for every row which is in the processing or failed state. Because we have unique deterministic IDs (we just hash the content with blake2b) I'm pretty confident this will work. I should probably do better modelling of how long this would take if it goes down while fully loaded though. I guess I thought we wanted rabbit for the powerful routing and rerouting logic, but now I think about it again, you could do that pretty elegantly in SQL. I do like postgres. What I really want to avoid at this stage is the process/thread/socket management for the actual workers, which sneakers handles sanely.
  - lobster_johnson 12y ago
    Sounds like a good plan.
  - sitkack 12y ago
    My recommendation would be to put your whole system on a test jig and bend it until it breaks. https://github.com/aphyr/jepsen https://github.com/aphyr/jepsen Until then, you won't know.
- joevandyk 12y ago
  https://github.com/chanks/que/blob/master/README.md https://github.com/chanks/que/blob/master/README.md is a postgresql queue that had been benchmarked at 10k/a on AWS.

lobster_johnson 12y ago

If you can determine what messages need to be re-queued, and easily re-queue them, then you should be fine. (That does raise the question of what you need a queue for, if your Postgres database keeps your state. Postgres can act as a pretty efficient queue — for one app we do this extensively — especially if you use listen/notify. It won't scale to thousands of requests/sec, but it's simpler to maintain than RabbitMQ, and has the benefit of being fully ACID.) Still, the RabbitMQ partitioning does exist and requires manual intervention whenever it happens. You can tell RabbitMQ to run in a mode where it will automatically recover from a partition by using the node which has the most connections, and discarding the data on the other conflicting nodes; but the problem in your case would be that you might not be able to tell when to re-queue your messages.

JonnieCache 12y ago
The database doesn't keep all the state, it just knows about which domain objects are currently "awaiting data," which are "processing," which are "failed" and which are "finished." Within the processing state there are several different things that go on, performed by various chained rabbit tasks. These push the data into elasticsearch, where the magic actually happens. My plan in a complete rabbitmq queue loss scenario is to simply fire the whole process off from the beginning for every row which is in the processing or failed state. Because we have unique deterministic IDs (we just hash the content with blake2b) I'm pretty confident this will work. I should probably do better modelling of how long this would take if it goes down while fully loaded though. I guess I thought we wanted rabbit for the powerful routing and rerouting logic, but now I think about it again, you could do that pretty elegantly in SQL. I do like postgres. What I really want to avoid at this stage is the process/thread/socket management for the actual workers, which sneakers handles sanely.
- lobster_johnson 12y ago
  Sounds like a good plan.
- sitkack 12y ago
  My recommendation would be to put your whole system on a test jig and bend it until it breaks. https://github.com/aphyr/jepsen https://github.com/aphyr/jepsen Until then, you won't know.
joevandyk 12y ago
https://github.com/chanks/que/blob/master/README.md https://github.com/chanks/que/blob/master/README.md is a postgresql queue that had been benchmarked at 10k/a on AWS.