4 ms·

* The number of messages we pass into the queue are very minimal, the dangers of the queue backing up are very low. * We are a Ruby/JavaScript/Python shop, so

* The number of messages we pass into the queue are very minimal, the dangers of the queue backing up are very low. * We are a Ruby/JavaScript/Python shop, so a library like Celery is not an optimal solution. * MongoDB is a core part of our infrastructure if it dies and is unavailable, we have bigger fish to fry. * Every piece of architecture in a stack is conceptual and maintenance overhead. I am certainly not averse to learning new technologies, but for the time being using MongoDB for this makes sense. I should underscore that most of the actual queue implementation is simply taking advantage of the behaviour of capped collections in MongoDB. Karait is a convenient API on top of this.

rbranson 15y ago
I'm talking about individual failure recovery, not full queues. What happens when a worker fails a job? What happens when a server dies? Is your code bug free? How do you recover? IMHO, you're making a pretty big assumption here anyway with the fixed queue depth. The idea with a message queue is that it delegates the responsibility of ensuring message delivery to another component, so that enqueues always work, and that messages always get delivered. Perhaps it's ok in your use case that some tasks fail to get delivered, but I'd prefer not to voluntarily give that up for some reason that I may have missed from the original post. In the linked .NET example, they used capped collections because it was a chatty message bus, not a worker queue, and guaranteed message delivery wasn't important. I've nearly filled the drives up on a RabbitMQ box with messages and it was still able to insert and pop them off at thousands per second, with full durability and persistence. Celery can read and write JSON to a multitude of backends. Last time I checked, both Ruby and JavaScript have pretty solid JSON support. DaemonKit on Ruby supports building a robust worker pool over AMQP. Every wheel reinvented is also conceptual and maintenance overhead, and I'd argue that, most of the time, is probably far worse than writing tools from scratch. There's a highly likelihood reinvention will introduce a bug or poorly-handled "edge" case in a from-scratch tool than if an existing, mature tool is used. When "bad things happen," a mature tool will give "a way out" that a custom tool probably won't have. Much like the honey badger, when scale and disaster strikes, it simply does not give a shit about how minimalistic your stack is. Good tooling can make short work of navigating out of these messes.
- BenjaminCoe 15y ago
  MongoDB uses capped collection internally for the oplog, which is very similar to my use-case. I have built an API to make this easier, i.e., there are no wheels being re-invented here. In my use-case I am simply passing a message to a worker to indicate that it it must terminate. There are no adverse consequences if the 'job' fails. I am most certainly not advocating that Karait solves all the world's queueing problems. It solves a specific problem I have, and I am confident in this assertion. BTW, do you get some sort of fiduciary kick-back from the makers Celery ;)
  - rbranson 15y ago
    Ok, so actually, you're not using it for "Dispatching work to crawlers" like the original post says?
  - BenjaminCoe 15y ago
    No, as a matter of fact, I'm not yet. I plan on adding in something resembling a visibility timeout in SQS, before I do this. So that I can ensure that messages always get delivered. I was going to continue using SQS until this functionality is well tested. Having said that, this is a good temporary solution for me to shutdown the crawlers. And I see absolutely no harm in abstracting the use of capped collections into an easy to use library. Is it as well developed as a mature AMQP queueing solution? Of course not, I don't claim that it is. Having said that, I trust MongoDB in my stack as a means to communicate between processes, and am confident using it for the use-cases that I am. As long as I'm doing this, I see no harm in open-sourcing the helper libraries I create for doing so. Am I a huge advocate of MongoDB? No, but if you happen to have it in your stack, as I do, It's handy to be able to use it in a similar manner to the way that Resque leverages Redis.

rbranson 15y ago

I'm talking about individual failure recovery, not full queues. What happens when a worker fails a job? What happens when a server dies? Is your code bug free? How do you recover? IMHO, you're making a pretty big assumption here anyway with the fixed queue depth. The idea with a message queue is that it delegates the responsibility of ensuring message delivery to another component, so that enqueues always work, and that messages always get delivered. Perhaps it's ok in your use case that some tasks fail to get delivered, but I'd prefer not to voluntarily give that up for some reason that I may have missed from the original post. In the linked .NET example, they used capped collections because it was a chatty message bus, not a worker queue, and guaranteed message delivery wasn't important. I've nearly filled the drives up on a RabbitMQ box with messages and it was still able to insert and pop them off at thousands per second, with full durability and persistence. Celery can read and write JSON to a multitude of backends. Last time I checked, both Ruby and JavaScript have pretty solid JSON support. DaemonKit on Ruby supports building a robust worker pool over AMQP. Every wheel reinvented is also conceptual and maintenance overhead, and I'd argue that, most of the time, is probably far worse than writing tools from scratch. There's a highly likelihood reinvention will introduce a bug or poorly-handled "edge" case in a from-scratch tool than if an existing, mature tool is used. When "bad things happen," a mature tool will give "a way out" that a custom tool probably won't have. Much like the honey badger, when scale and disaster strikes, it simply does not give a shit about how minimalistic your stack is. Good tooling can make short work of navigating out of these messes.

BenjaminCoe 15y ago
MongoDB uses capped collection internally for the oplog, which is very similar to my use-case. I have built an API to make this easier, i.e., there are no wheels being re-invented here. In my use-case I am simply passing a message to a worker to indicate that it it must terminate. There are no adverse consequences if the 'job' fails. I am most certainly not advocating that Karait solves all the world's queueing problems. It solves a specific problem I have, and I am confident in this assertion. BTW, do you get some sort of fiduciary kick-back from the makers Celery ;)
- rbranson 15y ago
  Ok, so actually, you're not using it for "Dispatching work to crawlers" like the original post says?
- BenjaminCoe 15y ago
  No, as a matter of fact, I'm not yet. I plan on adding in something resembling a visibility timeout in SQS, before I do this. So that I can ensure that messages always get delivered. I was going to continue using SQS until this functionality is well tested. Having said that, this is a good temporary solution for me to shutdown the crawlers. And I see absolutely no harm in abstracting the use of capped collections into an easy to use library. Is it as well developed as a mature AMQP queueing solution? Of course not, I don't claim that it is. Having said that, I trust MongoDB in my stack as a means to communicate between processes, and am confident using it for the use-cases that I am. As long as I'm doing this, I see no harm in open-sourcing the helper libraries I create for doing so. Am I a huge advocate of MongoDB? No, but if you happen to have it in your stack, as I do, It's handy to be able to use it in a similar manner to the way that Resque leverages Redis.