7 ms·

Announcing DJJob, a PHP port of delayed_job

38 points by ericwaller 16y ago

thibaut_barrere 16y ago
If you like Resque, too: http://github.com/chrisboulton/php-resque http://github.com/chrisboulton/php-resque
dkoch 16y ago
Also the php PECL gearman driver (http://pecl.php.net/package/gearman http://pecl.php.net/package/gearman) works well with the C-based gearmand server (http://gearman.org/ http://gearman.org/). We do some asynchronous image processing with PHP gearman workers, and it's been a very stable solution for us.
- nphase 16y ago
  Gearman is especially handy if you expose your application to it. I've got a wrapper that allows the app to push any function call out to the workers for async handling. So your image processing (or whatever heavy/blocking task) can be dispatched from within your standard libraries without having to maintain a separate set of code.
- horacegrant 16y ago
  And the post here sounds very NIH for not even mentioning Gearman in its survey of options.
joeuser12 16y ago
Yup, Gearman does this very well and it definitely isn't complicated.
bl4k 16y ago
you need to strip out the implementation specific classes such as propel and the mysql-specific calls to make this useful to anybody else for eg. an in-memory sqlite db would be ideal for this purpose (and a lot faster)
- ericwaller 16y ago
  Absolutely, dropping Propel is TODO #1. I just figured I'd throw it on github and see if there was any interest first.
happybuy 16y ago
In your example, does the call: $worker = new DJWorker($options); $worker->start(); Actually: 1. block until all jobs stored in the database have been completed 2. simply run a single job and return 3. return immediately and run all the jobs on a separate thread in the background I'm wondering where in a web application would be the best place to call $worker->start()?
- aonic 16y ago
  I think the point is to run the worker outside of a web request and using a cron job
  - andrewjshults 16y ago
    Or using the command line version of PHP and GNU Screen (I haven't fully studied the code, but it looks similar to a queue system that I hacked together a couple of days ago where the worker is wrapped in a while(TRUE) loop). In the interest of some outside figures, how have you found this to scale? I implemented a similar solution as more of a stopgap until I had the time to get RabbitMQ installed with a "proper" worker dispatcher (more moving parts but less stress on the DB). I mostly ask because we implemented it for our notification system where we had to generate a send large batches of custom emails which was bogging down the page (and hard to debug where in the chain we were getting failed messages).
    - aonic 16y ago
      Not sure who your question was directed to, but I've implemented a similar system in the past for a website with 10m+ users and 15+ web servers. It scaled just fine, we were sending close to 150K emails a day, among other background jobs for reporting, data cleanup, automated spam checks, etc
xal 16y ago
I'm the author of Delayed Job. The popularity of it really surprises me to no end, and always brings a smile to my face. DJ took about 15 minutes to code up originally and is a product of one of those software rage moments ("why don't people queue like this??!"). I put it on GitHub because Jeremy Kemper really wanted to have a look at it. I thought the idea behind it was so naive that it would have occurred to everyone who ever needed a queue and had discarded the approach for some reason. Little did I know that it was a fairly novel approach and that it would become one of my most popular libraries, right after ActiveMerchant and Liquid. Thank you for the PHP port!
- ericwaller 16y ago
  Great to hear from you. I remember being impressed by how short and straightforward delayed_job was when I first explored job queues. That's actually one of the only reasons I even considered building/porting something myself. You're welcome, and thank you for delayed_job!
mikeytown2 16y ago
Made something very similar to this in PHP about a year ago. It's buried in the Drupal project called Boost. It's the multi process crawler. It supports up to 8 processes crawling your site getting the cache warm. Gets call on cron & then 15 seconds after drupal's cron finished it starts up all 8 "threads". I've used it for image processing as well. Very stable will run for days if you give it a big enough queue (millions of pages to crawl). http://drupal.org/project/boost http://drupal.org/project/boost

38 points by ericwaller 16y ago

If you like Resque, too: http://github.com/chrisboulton/php-resque http://github.com/chrisboulton/php-resque

dkoch 16y ago

Also the php PECL gearman driver (http://pecl.php.net/package/gearman http://pecl.php.net/package/gearman) works well with the C-based gearmand server (http://gearman.org/ http://gearman.org/). We do some asynchronous image processing with PHP gearman workers, and it's been a very stable solution for us.

nphase 16y ago
Gearman is especially handy if you expose your application to it. I've got a wrapper that allows the app to push any function call out to the workers for async handling. So your image processing (or whatever heavy/blocking task) can be dispatched from within your standard libraries without having to maintain a separate set of code.
horacegrant 16y ago
And the post here sounds very NIH for not even mentioning Gearman in its survey of options.

joeuser12 16y ago

Yup, Gearman does this very well and it definitely isn't complicated.

bl4k 16y ago

you need to strip out the implementation specific classes such as propel and the mysql-specific calls to make this useful to anybody else for eg. an in-memory sqlite db would be ideal for this purpose (and a lot faster)

ericwaller 16y ago
Absolutely, dropping Propel is TODO #1. I just figured I'd throw it on github and see if there was any interest first.

happybuy 16y ago

In your example, does the call: $worker = new DJWorker($options); $worker->start(); Actually: 1. block until all jobs stored in the database have been completed 2. simply run a single job and return 3. return immediately and run all the jobs on a separate thread in the background I'm wondering where in a web application would be the best place to call $worker->start()?

aonic 16y ago
I think the point is to run the worker outside of a web request and using a cron job
- andrewjshults 16y ago
  Or using the command line version of PHP and GNU Screen (I haven't fully studied the code, but it looks similar to a queue system that I hacked together a couple of days ago where the worker is wrapped in a while(TRUE) loop). In the interest of some outside figures, how have you found this to scale? I implemented a similar solution as more of a stopgap until I had the time to get RabbitMQ installed with a "proper" worker dispatcher (more moving parts but less stress on the DB). I mostly ask because we implemented it for our notification system where we had to generate a send large batches of custom emails which was bogging down the page (and hard to debug where in the chain we were getting failed messages).
  - aonic 16y ago
    Not sure who your question was directed to, but I've implemented a similar system in the past for a website with 10m+ users and 15+ web servers. It scaled just fine, we were sending close to 150K emails a day, among other background jobs for reporting, data cleanup, automated spam checks, etc

xal 16y ago

I'm the author of Delayed Job. The popularity of it really surprises me to no end, and always brings a smile to my face. DJ took about 15 minutes to code up originally and is a product of one of those software rage moments ("why don't people queue like this??!"). I put it on GitHub because Jeremy Kemper really wanted to have a look at it. I thought the idea behind it was so naive that it would have occurred to everyone who ever needed a queue and had discarded the approach for some reason. Little did I know that it was a fairly novel approach and that it would become one of my most popular libraries, right after ActiveMerchant and Liquid. Thank you for the PHP port!

ericwaller 16y ago
Great to hear from you. I remember being impressed by how short and straightforward delayed_job was when I first explored job queues. That's actually one of the only reasons I even considered building/porting something myself. You're welcome, and thank you for delayed_job!

mikeytown2 16y ago

Made something very similar to this in PHP about a year ago. It's buried in the Drupal project called Boost. It's the multi process crawler. It supports up to 8 processes crawling your site getting the cache warm. Gets call on cron & then 15 seconds after drupal's cron finished it starts up all 8 "threads". I've used it for image processing as well. Very stable will run for days if you give it a big enough queue (millions of pages to crawl). http://drupal.org/project/boost http://drupal.org/project/boost