7 ms·

A Fast, Simple, Queue Built on MongoDB

31 points by BenjaminCoe 15y ago

rbranson 15y ago
Developer aversion to learning new tools is astounding sometimes. Background workers and message queues have to be the most reinvented wheel of all time. 99% of the time, something free and great will already do what you want, and they've thought of things you haven't. Why not redirect some of this energy into using a tool like Celery[1]? Surely it's simpler to learn a tool than it is to build a new one from scratch? * What happens when job's crash, fail, or get stuck? * What about when the machines running the workers die or are isolated from the network? * What happens if the backing store dies? * How do you start, stop, restart, grow, and shrink your worker pool? [1] http://ask.github.com/celery/getting-started/introduction.html http://ask.github.com/celery/getting-started/introduction.ht...
- BenjaminCoe 15y ago
  * The number of messages we pass into the queue are very minimal, the dangers of the queue backing up are very low. * We are a Ruby/JavaScript/Python shop, so a library like Celery is not an optimal solution. * MongoDB is a core part of our infrastructure if it dies and is unavailable, we have bigger fish to fry. * Every piece of architecture in a stack is conceptual and maintenance overhead. I am certainly not averse to learning new technologies, but for the time being using MongoDB for this makes sense. I should underscore that most of the actual queue implementation is simply taking advantage of the behaviour of capped collections in MongoDB. Karait is a convenient API on top of this.
  - rbranson 15y ago
    I'm talking about individual failure recovery, not full queues. What happens when a worker fails a job? What happens when a server dies? Is your code bug free? How do you recover? IMHO, you're making a pretty big assumption here anyway with the fixed queue depth. The idea with a message queue is that it delegates the responsibility of ensuring message delivery to another component, so that enqueues always work, and that messages always get delivered. Perhaps it's ok in your use case that some tasks fail to get delivered, but I'd prefer not to voluntarily give that up for some reason that I may have missed from the original post. In the linked .NET example, they used capped collections because it was a chatty message bus, not a worker queue, and guaranteed message delivery wasn't important. I've nearly filled the drives up on a RabbitMQ box with messages and it was still able to insert and pop them off at thousands per second, with full durability and persistence. Celery can read and write JSON to a multitude of backends. Last time I checked, both Ruby and JavaScript have pretty solid JSON support. DaemonKit on Ruby supports building a robust worker pool over AMQP. Every wheel reinvented is also conceptual and maintenance overhead, and I'd argue that, most of the time, is probably far worse than writing tools from scratch. There's a highly likelihood reinvention will introduce a bug or poorly-handled "edge" case in a from-scratch tool than if an existing, mature tool is used. When "bad things happen," a mature tool will give "a way out" that a custom tool probably won't have. Much like the honey badger, when scale and disaster strikes, it simply does not give a shit about how minimalistic your stack is. Good tooling can make short work of navigating out of these messes.
    - BenjaminCoe 15y ago
      MongoDB uses capped collection internally for the oplog, which is very similar to my use-case. I have built an API to make this easier, i.e., there are no wheels being re-invented here. In my use-case I am simply passing a message to a worker to indicate that it it must terminate. There are no adverse consequences if the 'job' fails. I am most certainly not advocating that Karait solves all the world's queueing problems. It solves a specific problem I have, and I am confident in this assertion. BTW, do you get some sort of fiduciary kick-back from the makers Celery ;)
      - rbranson 15y ago
        Ok, so actually, you're not using it for "Dispatching work to crawlers" like the original post says?
      - BenjaminCoe 15y ago
        No, as a matter of fact, I'm not yet. I plan on adding in something resembling a visibility timeout in SQS, before I do this. So that I can ensure that messages always get delivered. I was going to continue using SQS until this functionality is well tested. Having said that, this is a good temporary solution for me to shutdown the crawlers. And I see absolutely no harm in abstracting the use of capped collections into an easy to use library. Is it as well developed as a mature AMQP queueing solution? Of course not, I don't claim that it is. Having said that, I trust MongoDB in my stack as a means to communicate between processes, and am confident using it for the use-cases that I am. As long as I'm doing this, I see no harm in open-sourcing the helper libraries I create for doing so. Am I a huge advocate of MongoDB? No, but if you happen to have it in your stack, as I do, It's handy to be able to use it in a similar manner to the way that Resque leverages Redis.
- tptacek 15y ago
  Your point is well taken, but doesn't Celery want RabbitMQ? That's an awful lot of machinery for what is (in many apps) a very small problem. We get to look at lots of people's Rails apps, and it appears that Resque is emerging as the de facto answer to this problem. A nice attribute of Resque is that for all the machinery Redis provides, it also solves the "memcache" problem and is probably a better utility player than $X-MQ. If you have an app that already wants $X-MQ, Celery sounds very sensible. And having apps that want $X-MQ is a good thing, too. And, not reinventing this wheel also makes sense.
  - ctide 15y ago
    I was going to ask about Resque as well. There's also a port of it (I don't know how well maintained it is, but it exists) that uses mongo as the backend instead of redis if you want to avoid adding another platform to your app.
    - deleted 15y ago
      [deleted]
    - BenjaminCoe 15y ago
      I'm still leaning on SQS for most things. If it comes time to tear this piece of infrastructure out completely, I will likely go with Resque, or RabbitMQ. Karait is a simple abstraction layer on top of MongoDB capped collections, good for a certain class of problems, certainly not a capital 'M' message queue. An aside: I've worked with RabbitMQ in a large production environment. It is far from trivial to keep on its feet. It would be hard to get me excited about adding an extra layer of abstraction on top of it like Celery (I'd probably lean towards just using an AMQP library).
  - rbranson 15y ago
    Celery is best on RabbitMQ, but supports tons of backing stores, not that you necessarily should[1]. How is RabbitMQ a lot of machinery? Compared to Redis, AMQP and Rabbit have the complexity of a bologna sandwich. It's actually a quite simple system, and very easy to install. There is almost always no configuration necessary. Developers can install it in one command with MacPorts or Homebrew or apt or yum and don't even have to create users or "vhosts" to get started. Create a named exchange, subscribe to it, you're now AMQPing. It's well-packaged and well-documented. It's also so light on it's feet that it rarely actually needs a dedicated box; just stick it on your database server to get started. I think Resque is a good solution for Rails, for sure. I've built stuff around it, and it works well, and definitely battle tested. It certainly beats the existing AMQP-based solutions. The idea that your application code probably sucks and/or is in a constant state of flux, so handle failure gracefully is rather excellent. The web interface is particularly nice and useful. While I don't necessarily agree 100% with Redis as the backing store, the actual mechanics of the worker queue implementation (plus tools like resque-pool) work so well that for most projects, it's perfectly fine. I actually had a problem specific to using Redis materialize when it ran out of memory while queueing a large batch of per-user-generated weekly emails. A cron job would spin up a script that would cache a large amount of database data in RAM, using it to generate per-user content. The rate at which email could be sent became a major bottleneck, and we had Resque, so we just enqueue'd the entire email and had a small job that connected to the email server and send them. This worked reasonably well and for many months, the high watermark for RAM usage was well below what the box could hold. Unexpectedly, the amount of content contained within these emails quadrupled. Redis ate through all the RAM on the box and it started going to disk using Redis' VM mechanism. This was configured to "theoretically" prevent an unexpected OOM from making the queues unavailable. The unfortunate part is that the Resque stores the entire queue in a single key, and when the key gets swapped onto disk, Redis must read this entire key (and therefore the entire queue) into RAM before it's able to read any data from it. This wasn't Resque's fault really. It was my fault. I did not take the time to understand how Resque and Redis worked well enough to make a truly informed decision. I did not test the OOM condition. I should have tried to fill up the queue on a test box with a billion items just to see what happened. All lessons learned. [1] http://www.rabbitmq.com/blog/2011/01/20/rabbitmq-backing-stores-databases-and-disks/ http://www.rabbitmq.com/blog/2011/01/20/rabbitmq-backing-sto...
xal 15y ago
Doing a queue system where the workers wait in a busy-loop is pretty insane. I'm saying that as the author of Delayed::Job so i'm pretty much responsible for those shenanigans.
- BenjaminCoe 15y ago
  Oh, I agree 100%. That's not how I advocate using this (it's just an example I whipped together of the API). I use it in a distributed environment where the queue is checked periodically by subprocesses (mainly for shutdown events).
- LeafStorm 15y ago
  Out of curiosity, what strategy would you recommend instead? Something like Redis' B[LR]POP, where the blocking can be handled on the network layer? Or a different strategy entirely?
  - Todd 15y ago
    If you put this behind, say a REST API vs using MongoDB directly, the clients could use long polling. If you look at Twitter's Kestrel, it was implemented with a similar mechanism, albeit leveraging some capabilities that were intrinsic to memcached. I built a little queuing prototype using long polling and it works very well. If you build it using an async server, you can run many clients while only using a few threads. There's still the question of how to block and/or wake on an enqueue event to MongoDB vs. polling. The best approach may be to implement queueing using your REST API, leveraging all of the advantages of the async server and use MongoDB for a backing store or journal. This is similar to how Kestrel does it. My prototype was very responsive, although I only got about 500 messages per second of throughput on my MBA.
    - BenjaminCoe 15y ago
      I see what you're getting at. Behind the scenes, I am ultimately using an event loop :) Having said that, since MongoDB is a persistent connection, does long pulling over HTTP make sense for the client? I was thinking of doing a proof of concept in Node, that would at least behave like it's getting pushed messages rather than pulling ... But, I couldn't think of a way other than repeated setTimeouts. Any thoughts?
josephruscio 15y ago
Perhaps I'm assuming too much, but it looks like you might be confusing notification with queueing. As you found with SQS, queueing systems can make for poor notification mechanisms. Did you try using AWS SNS (http://aws.amazon.com/sns/ http://aws.amazon.com/sns/) for your first two use cases? We have similar notification use-cases internally, and SNS has worked great.
- BenjaminCoe 15y ago
  I don't know that this is quite what I'm looking for, a worthwhile read though. Never ceases to amaze me how many things Amazon can prefix with Simple.

31 points by BenjaminCoe 15y ago

Developer aversion to learning new tools is astounding sometimes. Background workers and message queues have to be the most reinvented wheel of all time. 99% of the time, something free and great will already do what you want, and they've thought of things you haven't. Why not redirect some of this energy into using a tool like Celery[1]? Surely it's simpler to learn a tool than it is to build a new one from scratch? * What happens when job's crash, fail, or get stuck? * What about when the machines running the workers die or are isolated from the network? * What happens if the backing store dies? * How do you start, stop, restart, grow, and shrink your worker pool? [1] http://ask.github.com/celery/getting-started/introduction.html http://ask.github.com/celery/getting-started/introduction.ht...

BenjaminCoe 15y ago
* The number of messages we pass into the queue are very minimal, the dangers of the queue backing up are very low. * We are a Ruby/JavaScript/Python shop, so a library like Celery is not an optimal solution. * MongoDB is a core part of our infrastructure if it dies and is unavailable, we have bigger fish to fry. * Every piece of architecture in a stack is conceptual and maintenance overhead. I am certainly not averse to learning new technologies, but for the time being using MongoDB for this makes sense. I should underscore that most of the actual queue implementation is simply taking advantage of the behaviour of capped collections in MongoDB. Karait is a convenient API on top of this.
- rbranson 15y ago
  I'm talking about individual failure recovery, not full queues. What happens when a worker fails a job? What happens when a server dies? Is your code bug free? How do you recover? IMHO, you're making a pretty big assumption here anyway with the fixed queue depth. The idea with a message queue is that it delegates the responsibility of ensuring message delivery to another component, so that enqueues always work, and that messages always get delivered. Perhaps it's ok in your use case that some tasks fail to get delivered, but I'd prefer not to voluntarily give that up for some reason that I may have missed from the original post. In the linked .NET example, they used capped collections because it was a chatty message bus, not a worker queue, and guaranteed message delivery wasn't important. I've nearly filled the drives up on a RabbitMQ box with messages and it was still able to insert and pop them off at thousands per second, with full durability and persistence. Celery can read and write JSON to a multitude of backends. Last time I checked, both Ruby and JavaScript have pretty solid JSON support. DaemonKit on Ruby supports building a robust worker pool over AMQP. Every wheel reinvented is also conceptual and maintenance overhead, and I'd argue that, most of the time, is probably far worse than writing tools from scratch. There's a highly likelihood reinvention will introduce a bug or poorly-handled "edge" case in a from-scratch tool than if an existing, mature tool is used. When "bad things happen," a mature tool will give "a way out" that a custom tool probably won't have. Much like the honey badger, when scale and disaster strikes, it simply does not give a shit about how minimalistic your stack is. Good tooling can make short work of navigating out of these messes.
  - BenjaminCoe 15y ago
    MongoDB uses capped collection internally for the oplog, which is very similar to my use-case. I have built an API to make this easier, i.e., there are no wheels being re-invented here. In my use-case I am simply passing a message to a worker to indicate that it it must terminate. There are no adverse consequences if the 'job' fails. I am most certainly not advocating that Karait solves all the world's queueing problems. It solves a specific problem I have, and I am confident in this assertion. BTW, do you get some sort of fiduciary kick-back from the makers Celery ;)
    - rbranson 15y ago
      Ok, so actually, you're not using it for "Dispatching work to crawlers" like the original post says?
    - BenjaminCoe 15y ago
      No, as a matter of fact, I'm not yet. I plan on adding in something resembling a visibility timeout in SQS, before I do this. So that I can ensure that messages always get delivered. I was going to continue using SQS until this functionality is well tested. Having said that, this is a good temporary solution for me to shutdown the crawlers. And I see absolutely no harm in abstracting the use of capped collections into an easy to use library. Is it as well developed as a mature AMQP queueing solution? Of course not, I don't claim that it is. Having said that, I trust MongoDB in my stack as a means to communicate between processes, and am confident using it for the use-cases that I am. As long as I'm doing this, I see no harm in open-sourcing the helper libraries I create for doing so. Am I a huge advocate of MongoDB? No, but if you happen to have it in your stack, as I do, It's handy to be able to use it in a similar manner to the way that Resque leverages Redis.
tptacek 15y ago
Your point is well taken, but doesn't Celery want RabbitMQ? That's an awful lot of machinery for what is (in many apps) a very small problem. We get to look at lots of people's Rails apps, and it appears that Resque is emerging as the de facto answer to this problem. A nice attribute of Resque is that for all the machinery Redis provides, it also solves the "memcache" problem and is probably a better utility player than $X-MQ. If you have an app that already wants $X-MQ, Celery sounds very sensible. And having apps that want $X-MQ is a good thing, too. And, not reinventing this wheel also makes sense.
- ctide 15y ago
  I was going to ask about Resque as well. There's also a port of it (I don't know how well maintained it is, but it exists) that uses mongo as the backend instead of redis if you want to avoid adding another platform to your app.
  - deleted 15y ago
    [deleted]
  - BenjaminCoe 15y ago
    I'm still leaning on SQS for most things. If it comes time to tear this piece of infrastructure out completely, I will likely go with Resque, or RabbitMQ. Karait is a simple abstraction layer on top of MongoDB capped collections, good for a certain class of problems, certainly not a capital 'M' message queue. An aside: I've worked with RabbitMQ in a large production environment. It is far from trivial to keep on its feet. It would be hard to get me excited about adding an extra layer of abstraction on top of it like Celery (I'd probably lean towards just using an AMQP library).
- rbranson 15y ago
  Celery is best on RabbitMQ, but supports tons of backing stores, not that you necessarily should[1]. How is RabbitMQ a lot of machinery? Compared to Redis, AMQP and Rabbit have the complexity of a bologna sandwich. It's actually a quite simple system, and very easy to install. There is almost always no configuration necessary. Developers can install it in one command with MacPorts or Homebrew or apt or yum and don't even have to create users or "vhosts" to get started. Create a named exchange, subscribe to it, you're now AMQPing. It's well-packaged and well-documented. It's also so light on it's feet that it rarely actually needs a dedicated box; just stick it on your database server to get started. I think Resque is a good solution for Rails, for sure. I've built stuff around it, and it works well, and definitely battle tested. It certainly beats the existing AMQP-based solutions. The idea that your application code probably sucks and/or is in a constant state of flux, so handle failure gracefully is rather excellent. The web interface is particularly nice and useful. While I don't necessarily agree 100% with Redis as the backing store, the actual mechanics of the worker queue implementation (plus tools like resque-pool) work so well that for most projects, it's perfectly fine. I actually had a problem specific to using Redis materialize when it ran out of memory while queueing a large batch of per-user-generated weekly emails. A cron job would spin up a script that would cache a large amount of database data in RAM, using it to generate per-user content. The rate at which email could be sent became a major bottleneck, and we had Resque, so we just enqueue'd the entire email and had a small job that connected to the email server and send them. This worked reasonably well and for many months, the high watermark for RAM usage was well below what the box could hold. Unexpectedly, the amount of content contained within these emails quadrupled. Redis ate through all the RAM on the box and it started going to disk using Redis' VM mechanism. This was configured to "theoretically" prevent an unexpected OOM from making the queues unavailable. The unfortunate part is that the Resque stores the entire queue in a single key, and when the key gets swapped onto disk, Redis must read this entire key (and therefore the entire queue) into RAM before it's able to read any data from it. This wasn't Resque's fault really. It was my fault. I did not take the time to understand how Resque and Redis worked well enough to make a truly informed decision. I did not test the OOM condition. I should have tried to fill up the queue on a test box with a billion items just to see what happened. All lessons learned. [1] http://www.rabbitmq.com/blog/2011/01/20/rabbitmq-backing-stores-databases-and-disks/ http://www.rabbitmq.com/blog/2011/01/20/rabbitmq-backing-sto...

xal 15y ago

Doing a queue system where the workers wait in a busy-loop is pretty insane. I'm saying that as the author of Delayed::Job so i'm pretty much responsible for those shenanigans.

BenjaminCoe 15y ago
Oh, I agree 100%. That's not how I advocate using this (it's just an example I whipped together of the API). I use it in a distributed environment where the queue is checked periodically by subprocesses (mainly for shutdown events).
LeafStorm 15y ago
Out of curiosity, what strategy would you recommend instead? Something like Redis' B[LR]POP, where the blocking can be handled on the network layer? Or a different strategy entirely?
- Todd 15y ago
  If you put this behind, say a REST API vs using MongoDB directly, the clients could use long polling. If you look at Twitter's Kestrel, it was implemented with a similar mechanism, albeit leveraging some capabilities that were intrinsic to memcached. I built a little queuing prototype using long polling and it works very well. If you build it using an async server, you can run many clients while only using a few threads. There's still the question of how to block and/or wake on an enqueue event to MongoDB vs. polling. The best approach may be to implement queueing using your REST API, leveraging all of the advantages of the async server and use MongoDB for a backing store or journal. This is similar to how Kestrel does it. My prototype was very responsive, although I only got about 500 messages per second of throughput on my MBA.
  - BenjaminCoe 15y ago
    I see what you're getting at. Behind the scenes, I am ultimately using an event loop :) Having said that, since MongoDB is a persistent connection, does long pulling over HTTP make sense for the client? I was thinking of doing a proof of concept in Node, that would at least behave like it's getting pushed messages rather than pulling ... But, I couldn't think of a way other than repeated setTimeouts. Any thoughts?

josephruscio 15y ago

Perhaps I'm assuming too much, but it looks like you might be confusing notification with queueing. As you found with SQS, queueing systems can make for poor notification mechanisms. Did you try using AWS SNS (http://aws.amazon.com/sns/ http://aws.amazon.com/sns/) for your first two use cases? We have similar notification use-cases internally, and SNS has worked great.

BenjaminCoe 15y ago
I don't know that this is quite what I'm looking for, a worthwhile read though. Never ceases to amaze me how many things Amazon can prefix with Simple.