4 ms·

Here's what I'm building my new queue with: https://github.com/jondot/sneakers https://github.com/jondot/sneakers It's a ruby library that implements a Resque

Here's what I'm building my new queue with: https://github.com/jondot/sneakers https://github.com/jondot/sneakers It's a ruby library that implements a Resque-like interface on top of RabbitMQ. Crucially, it doesn't have any opinion on the message contents. It doesn't even deserialize them for you. Most of what it does is manage a unicorn-like process/thread pool plus the associated worker objects, and consumes messages in a thread-safe way. That and the usual configuration boilerplate. Also check out this branch which implements a max-retries strategy using dead letter exchanges, and is kept up to date with the upstream: https://github.com/Yesware/sneakers https://github.com/Yesware/sneakers I imagine it will be merged soon. The thread/process management is done by the serverengine gem: https://github.com/frsyuki/serverengine https://github.com/frsyuki/serverengine

lobster_johnson 12y ago
Sneakers is nice, but keep in mind that RabbitMQ has a huge problem with partitions -- which can occur even on flawless networks, but will certainly occur frequently on VPS systems like AWS and Digital Ocean. See [1] for an excellent analysis. Unfortunately, once you have a partition there is no way to merge the isolated nodes back into the cluster; since queues "live" on the node they were initially created on, this could be extremely disruptive. The only way to recover is to discard queue data. I love how RabbitMQ works, but it's terrible in a virtualization environment. Unfortunately there aren't any worthwhile competitors. Kafka comes closest, but it lacks support for topic fanouts, which means every producer has to know about every client, which isn't really acceptable. [1] http://aphyr.com/posts/315-call-me-maybe-rabbitmq http://aphyr.com/posts/315-call-me-maybe-rabbitmq
- JonnieCache 12y ago
  Fortunately all our jobs are idempotent, and I hope to keep it that way. Postgres keeps track of what's currently "in-flight" so to speak, so my plan for failures is just to re-queue stuff. The good thing about sneakers is we're not locked into any particular routing scheme or message structure so it'll be easy to move away from once we eventually experience problems. So, community, am I being naive? Will the idempotency thing not save me? (Also, lobster_johnson when you say terrible in a virtualisation environment, are you referring to AWS-ish shared VPSs or to a containers-on-colocated-boxes solution, which is what we favor?)
  - lobster_johnson 12y ago
    If you can determine what messages need to be re-queued, and easily re-queue them, then you should be fine. (That does raise the question of what you need a queue for, if your Postgres database keeps your state. Postgres can act as a pretty efficient queue — for one app we do this extensively — especially if you use listen/notify. It won't scale to thousands of requests/sec, but it's simpler to maintain than RabbitMQ, and has the benefit of being fully ACID.) Still, the RabbitMQ partitioning does exist and requires manual intervention whenever it happens. You can tell RabbitMQ to run in a mode where it will automatically recover from a partition by using the node which has the most connections, and discarding the data on the other conflicting nodes; but the problem in your case would be that you might not be able to tell when to re-queue your messages.
    - JonnieCache 12y ago
      The database doesn't keep all the state, it just knows about which domain objects are currently "awaiting data," which are "processing," which are "failed" and which are "finished." Within the processing state there are several different things that go on, performed by various chained rabbit tasks. These push the data into elasticsearch, where the magic actually happens. My plan in a complete rabbitmq queue loss scenario is to simply fire the whole process off from the beginning for every row which is in the processing or failed state. Because we have unique deterministic IDs (we just hash the content with blake2b) I'm pretty confident this will work. I should probably do better modelling of how long this would take if it goes down while fully loaded though. I guess I thought we wanted rabbit for the powerful routing and rerouting logic, but now I think about it again, you could do that pretty elegantly in SQL. I do like postgres. What I really want to avoid at this stage is the process/thread/socket management for the actual workers, which sneakers handles sanely.
      - lobster_johnson 12y ago
        Sounds like a good plan.
      - sitkack 12y ago
- kylequest 12y ago
  The statement about Kafka sounds a bit strange... Producers don't need to know about the consumers/clients. They just post messages to the brokers.
  - lobster_johnson 12y ago
    Kafka's topics are also queues, unlike RabbitMQ where a queue is bound to an exchange, and you publish by posting messages to the exchange, not the queue; the exchange determines which bound queues get the message. This binding uses a routing key, which is a path that supports wildcards. For example, a consumer can bind its queue to an exchange "foo" (representing an app's events) using the routing key "create.photo.*". It will get "create.photo.5378" and so on. It won't get "update.photo.5378". It's a really nice, flexible system that allows you to separate publishers from consumers without incurring much of a performance hit; if nobody is listening to a particular key, nothing happens, and if multiple groups of consumers listen to overlapping keys, they still get the messages in the correct order, because each message goes into a single queue. Kafka's topics, being queues, doesn't allow this. Publishing is done to a topic, and consumers must also consume from that topic. One message, one topic. There are three options: 1. Use a single topic for different (heterogenous) messages. For example, "foo" could be the topic, and each message would contain a field called "path" or similar (eg., "photo.5378"). Each consumer would filter out messages it's not interested in based on this path. This puts a potentially expensive burden on the consumer. 2. Create several topics, and let consumers listen to each one: "foo.create", perhaps (all create events about different objects), or "foo.photo" (all events about photos) or even "foo.create.photo". This adds complexity and will potentially explode the number of queues needed for many apps, object types, events, etc. Moreover, since message delivery order guarantees are per topic, messages across different types can come out of order: deletes before creates, for example. 3. Invert the relationship; a producer must know every possible consumer's range of interest, and post to each of their topics. Foo must know that app "bar" is only interested in photos, and that app "baz" wants events about all types of objects. Of course, this breaks separation of concerns (although the routing table could go into something like ZooKeeper) Kafka could implement a similar routing system if it wanted to, of course.

lobster_johnson 12y ago

Sneakers is nice, but keep in mind that RabbitMQ has a huge problem with partitions -- which can occur even on flawless networks, but will certainly occur frequently on VPS systems like AWS and Digital Ocean. See [1] for an excellent analysis. Unfortunately, once you have a partition there is no way to merge the isolated nodes back into the cluster; since queues "live" on the node they were initially created on, this could be extremely disruptive. The only way to recover is to discard queue data. I love how RabbitMQ works, but it's terrible in a virtualization environment. Unfortunately there aren't any worthwhile competitors. Kafka comes closest, but it lacks support for topic fanouts, which means every producer has to know about every client, which isn't really acceptable. [1] http://aphyr.com/posts/315-call-me-maybe-rabbitmq http://aphyr.com/posts/315-call-me-maybe-rabbitmq

JonnieCache 12y ago
Fortunately all our jobs are idempotent, and I hope to keep it that way. Postgres keeps track of what's currently "in-flight" so to speak, so my plan for failures is just to re-queue stuff. The good thing about sneakers is we're not locked into any particular routing scheme or message structure so it'll be easy to move away from once we eventually experience problems. So, community, am I being naive? Will the idempotency thing not save me? (Also, lobster_johnson when you say terrible in a virtualisation environment, are you referring to AWS-ish shared VPSs or to a containers-on-colocated-boxes solution, which is what we favor?)
- lobster_johnson 12y ago
  If you can determine what messages need to be re-queued, and easily re-queue them, then you should be fine. (That does raise the question of what you need a queue for, if your Postgres database keeps your state. Postgres can act as a pretty efficient queue — for one app we do this extensively — especially if you use listen/notify. It won't scale to thousands of requests/sec, but it's simpler to maintain than RabbitMQ, and has the benefit of being fully ACID.) Still, the RabbitMQ partitioning does exist and requires manual intervention whenever it happens. You can tell RabbitMQ to run in a mode where it will automatically recover from a partition by using the node which has the most connections, and discarding the data on the other conflicting nodes; but the problem in your case would be that you might not be able to tell when to re-queue your messages.
  - JonnieCache 12y ago
    The database doesn't keep all the state, it just knows about which domain objects are currently "awaiting data," which are "processing," which are "failed" and which are "finished." Within the processing state there are several different things that go on, performed by various chained rabbit tasks. These push the data into elasticsearch, where the magic actually happens. My plan in a complete rabbitmq queue loss scenario is to simply fire the whole process off from the beginning for every row which is in the processing or failed state. Because we have unique deterministic IDs (we just hash the content with blake2b) I'm pretty confident this will work. I should probably do better modelling of how long this would take if it goes down while fully loaded though. I guess I thought we wanted rabbit for the powerful routing and rerouting logic, but now I think about it again, you could do that pretty elegantly in SQL. I do like postgres. What I really want to avoid at this stage is the process/thread/socket management for the actual workers, which sneakers handles sanely.
    - lobster_johnson 12y ago
      Sounds like a good plan.
    - sitkack 12y ago
kylequest 12y ago
The statement about Kafka sounds a bit strange... Producers don't need to know about the consumers/clients. They just post messages to the brokers.
- lobster_johnson 12y ago
  Kafka's topics are also queues, unlike RabbitMQ where a queue is bound to an exchange, and you publish by posting messages to the exchange, not the queue; the exchange determines which bound queues get the message. This binding uses a routing key, which is a path that supports wildcards. For example, a consumer can bind its queue to an exchange "foo" (representing an app's events) using the routing key "create.photo.*". It will get "create.photo.5378" and so on. It won't get "update.photo.5378". It's a really nice, flexible system that allows you to separate publishers from consumers without incurring much of a performance hit; if nobody is listening to a particular key, nothing happens, and if multiple groups of consumers listen to overlapping keys, they still get the messages in the correct order, because each message goes into a single queue. Kafka's topics, being queues, doesn't allow this. Publishing is done to a topic, and consumers must also consume from that topic. One message, one topic. There are three options: 1. Use a single topic for different (heterogenous) messages. For example, "foo" could be the topic, and each message would contain a field called "path" or similar (eg., "photo.5378"). Each consumer would filter out messages it's not interested in based on this path. This puts a potentially expensive burden on the consumer. 2. Create several topics, and let consumers listen to each one: "foo.create", perhaps (all create events about different objects), or "foo.photo" (all events about photos) or even "foo.create.photo". This adds complexity and will potentially explode the number of queues needed for many apps, object types, events, etc. Moreover, since message delivery order guarantees are per topic, messages across different types can come out of order: deletes before creates, for example. 3. Invert the relationship; a producer must know every possible consumer's range of interest, and post to each of their topics. Foo must know that app "bar" is only interested in photos, and that app "baz" wants events about all types of objects. Of course, this breaks separation of concerns (although the routing table could go into something like ZooKeeper) Kafka could implement a similar routing system if it wanted to, of course.