6 ms·
Gearman, an open source framework for distributed processing
- zrail 15y agoMy problem with Gearman is that the queues are completely opaque. There's no way to inspect them, except in so far as if you use a database backend you can see the opaque jobs and get a count.
- jbert 15y agoI don't think you can see individual job data, but you (and/or your monitoring daemon) can telnet to the port and issue admin commands. e.g. 'status' will list all job types, with #queued, #running and #workers. http://gearman.org/?id=protocol http://gearman.org/?id=protocol (search for 'Administrative Protocol' - 2nd hit)
- strags 15y agoHow does this compare to resque? (Ah... answered my own question - resque seems to be ruby only).
- zrail 15y agoIn short, Gearman is multi-language, multiple different backends, possible to do RPC-style synchronous processing. Resque and friends[1] are single-language, exclusively async, but also extremely transparent because they're just JSON blobs in redis. Resque has a huge following as well. [1]: https://github.com/defunkt/resque/wiki/alternate-implementations https://github.com/defunkt/resque/wiki/alternate-implementat...
- gizzlon 15y agoDoesn't Gearman do async as well?
- evmar 15y agoIt's funny to see how bradfitz's projects other than memcached have fared over time. Didn't realize this one was still used by anyone.
- jemeshsu 15y agoCan never count Gearman out, and it is winning new user such as Instagram. Great for simple work queue. I recently switch from Celery to Gearman to reduce headache.
- scorpioxy 15y agoCan you share what sort of headaches you got from Celery? I've used both and find them equally powerful.
- dpres 15y agoI was just using Gearman::Client and Gearman::Worker today... it's still used by at least me.
- enigmabomb 15y agoWe tried to use this in our last project and ended up writing the mother ship and workers from scratch because there didn't seem to be a community behind gearman. As soon as the nda runs out, I'm going to open source it as a competing framework.
- rubashov 15y agoWhat was your problem with it? I have tried to use Condor for a couple things recently and was annoyed enough by it I concluded Gearman was probably the better approach, though I haven't used Gearman. For the record, Condor probably works fine for its primary use case, which is long running single threaded processes on Unix that eat an input file and spit out an output file before terminating. I was trying to use it to have a bunch of processes on Windows that also occasionally write to a database. Was also trying to tag "slots" on specific machines to use certain installed dependencies. I advise against trying to use Condor and Windows, or on trying to use any of its advertised advanced features beyond the basic "standard" or "universe" model. Have had no end of trouble with the "stored credentials" feature that lets you run a process under a specific account.
- mst 15y agoI think a part of the problem there may be that a substantial percentage of the original user community are perl hackers, and tend to discuss their Gearman usage in the context of whatever else they're using - I see Gearman discussions reasonably often on the Catalyst IRC channel, for example. Another reason is quite possibly that for the vast majority of tasks, Gearman "just works" - if the obvious way to use a piece of software works first time, talking to other users isn't particularly necessary (there's not really a GNU cat community either, for example :) I'm not entirely sure what you'd do to fix that though (or even if there's anything except perception to be fixed)
- pan69 15y agoTried it about a year ago. Couldn't get it installed on Ubuntu.
- maxklein 15y agoI used gearman to replace cron. It's much more efficient, particular for tasks like queueing email to be sent out.