4 ms·
MongoDB uses capped collection internally for the oplog, which is very similar to my use-case. I have built an API to make this easier, i.e., there are no wheel
by BenjaminCoe 15y ago
MongoDB uses capped collection internally for the oplog, which is very similar to my use-case. I have built an API to make this easier, i.e., there are no wheels being re-invented here.
In my use-case I am simply passing a message to a worker to indicate that it it must terminate. There are no adverse consequences if the 'job' fails.
I am most certainly not advocating that Karait solves all the world's queueing problems. It solves a specific problem I have, and I am confident in this assertion.
BTW, do you get some sort of fiduciary kick-back from the makers Celery ;)
- rbranson 15y agoOk, so actually, you're not using it for "Dispatching work to crawlers" like the original post says?
- BenjaminCoe 15y agoNo, as a matter of fact, I'm not yet. I plan on adding in something resembling a visibility timeout in SQS, before I do this. So that I can ensure that messages always get delivered. I was going to continue using SQS until this functionality is well tested. Having said that, this is a good temporary solution for me to shutdown the crawlers. And I see absolutely no harm in abstracting the use of capped collections into an easy to use library. Is it as well developed as a mature AMQP queueing solution? Of course not, I don't claim that it is. Having said that, I trust MongoDB in my stack as a means to communicate between processes, and am confident using it for the use-cases that I am. As long as I'm doing this, I see no harm in open-sourcing the helper libraries I create for doing so. Am I a huge advocate of MongoDB? No, but if you happen to have it in your stack, as I do, It's handy to be able to use it in a similar manner to the way that Resque leverages Redis.