5 ms·
Introducing Resque
- bjclark 17y agoRedis is almost perfect for a queue because of it's lists and atomic operations. And, most of all, because it's persistent. It's amazing how many queues aren't persistent. I'm glad someone finally did this, it could be a great solution. I'm looking forward to digging into this.
- mrduncan 17y agoAnd, most of all, because it's persistent. Just a slight caution, since Redis stores it's data in memory and asynchronously writes out to disk it is possible to lose small amounts of data in a crash. In most cases this is an acceptable trade off for the speed of an in-memory database though. There is also plenty of good info on the project page - http://code.google.com/p/redis/ http://code.google.com/p/redis/
- antirez 17y agoSince Redis 1.1 (currently in beta - don't use it in production for now) there is a new append-only journal storage system alternative to the main one (that is, snapshotting), so starting from 1.1 Resque will be more reliable as well if needed, because in this new (optional) configuration Redis can't lose data. It's also possible to configure if it should fsync() never, everysec, or after every write. It's up to you to select the tradeoff between speed and safety :)
- xal 17y agoI wrote DJ. This looks amazing for high volume sites. At Shopify DJ is barely able to keep up these days. My biggest regret with DJ is actually that I added numerical priorities instead of choosing the same tag based approach that Resque implemented. I'll likely release a DJ2 at some point which backports some of the ideas.
- trevorturk 17y agoFWIW the author of DJ posted in more detail on this subject in this Google Group: http://groups.google.com/group/delayed_job/msg/9e2987caa177ba2d http://groups.google.com/group/delayed_job/msg/9e2987caa177b...
- mattyb 17y agoFWIW, xal (who you replied to) == the author of DJ. http://github.com/tobi/delayed_job http://github.com/tobi/delayed_job
- deleted 17y ago[deleted]
- tptacek 17y agoThe Resque code aside --- and I want to switch to Resque, like, tomorrow --- this is the best article on Ruby job queues I've ever read. It's Github's review of pretty much every mainstream job queue out there. Excellent.
- adriand 17y agoI've also been through a lot of different background processing libraries for Rails, and so far, if you have simple needs, I'm finding that Spawn is really awesome. http://github.com/tra/spawn/ http://github.com/tra/spawn/ I think something like Resque will be very helpful to us over the long term, but if you're looking for something quick that just works to send a big batch of emails, import stuff via a web service, whatever, check out Spawn. Not at all appropriate for a site like Github, but works well for our vastly smaller websites.
- dacort 17y agoReally looking forward to trying this - I've given backgroundrb, BJ, DJ, and workling a go and there's always been something that's frustrated me. It's amazing the number of different things I'm considering using Redis for as well.
- richcollins 17y agoAlso see VertexDB (http://github.com/stevedekorte/vertexdb http://github.com/stevedekorte/vertexdb). It is a high performance persistent graph database with filesystem like semantics (will soon support FUSED). It presents an API for an atomic queue pop operation, which allows for synchronization of jobs.
- jbyers 17y agoNice. We built a mysql-backed queue with nearly identical properties but it's not especially efficient. It does 10+ jobs per second per worker, but with locking this starts to cause InnoDB concurrency problems. Not insurmountable, but moving to a Redis-backed approach seems like a great option when our current queue can no longer keep up. Really glad to have this approach proven by GitHub.
- paraschopra 17y agoSlightly off-topic, but is there any stable background job queue available for PHP (and perhaps MySQL)? I have been using custom hacked solution on top of tokyo tyrant but I will prefer if someone has a mature library.
- callmeed 17y agoJeeze, I just started migrating from BJ to DJ ... what should I do now?