6 ms·
Wow, someone else using Beanstalkd; my hero! Beanstalkd may be one of my favourite but most neglected pieces of software.
by robotmay 14y ago
Wow, someone else using Beanstalkd; my hero! Beanstalkd may be one of my favourite but most neglected pieces of software.
- pimeys 14y agoIt is a blast. Although we had our concerns because you can't see what's inside a Beanstalkd queue (duh, because it's a queue), but building a system for monitoring the failures was not a big deal. And of course the tubes rock. Now I can build a separate tube for each retry, I can set different priorities for the tubes (so the more fail prone tubes won't block all workers), I can set a deadline for a job to finish and if the worker dies in the middle the job will be returned for the other workers. And best of all, I can monitor all tubes nicely with Graphite.
- fullmoon 14y agoWhy do you feel that a queue is inherently unpeekable?
- pimeys 14y agoYou can peek, but it's not built for looking through all the items in it. It is doable, but not very efficient.
- ozataman 14y agoDo you have a custom solution for monitoring via Graphite? If so, is it something you can share?
- pimeys 14y agoIt's pretty straightforward. Mostly we have custom data not related to Beanstalkd to monitor, but basically there's two scenarios: 1. When doing something, increment a counter in Statsd, which will aggregate the values to Graphite. Works well for custom stuff. 2. Write a rake task to poll Beanstalkd with `stats` command, reduce the data and send it straight to Graphite. I'm really planning to open source the queuing system we're using because it's so simple. It might need a prettier admin view and some nice way of plugging in more monitoring systems before I put it to public.
- forsaken 14y agoWe also use beanstalk a ton where I work. We have put multiple billions of jobs through it over the past few years. It's rock solid and one of my favorite pieces of software.