3 ms·
Ask HN: How do you do batch processing?
I'm curious to to hear what technologies others use to process large volumes of batch jobs.
We've found it useful to be able to offload more process-intensive jobs from an end-user execution, and pass it into a queue ( which looks something like this http://abhinavsingh.com/blog/2010/02/memq-fast-queue-implementation-using-memcached-and-php-only/ )
What do you use?
- maxdemarzi 15y agoResque. https://github.com/blog/542-introducing-resque https://github.com/blog/542-introducing-resque I use the Ruby version, but there is a php version too. https://github.com/chrisboulton/php-resque https://github.com/chrisboulton/php-resque
- andrewjshults 15y agoWe're using a combo of the PHP version and the Ruby version and it's been working great for us (we migrated over from a custom queue on top of MySQL. The Sinatra based front-end is awesome for being able to monitor everything that's going on and with a little bit of work you can get pretty much any of the Ruby extensions to work with the PHP version (we're using the resque-scheduler https://github.com/bvandenbos/resque-scheduler https://github.com/bvandenbos/resque-scheduler to do delayed jobs). There are a few inconsistencies between how the jobs get handled between the two (it's not a huge difference and I've been meaning to submit it as a patch, I need to separate out some of the changes I made to make the PHP version compatible with resque-scheduler). If you have questions about implementing the PHP version, feel free to ping me.
- gresrun 15y agoThere is also a Java version that I wrote a few months ago if that's more your cup of tea: https://github.com/gresrun/jesque https://github.com/gresrun/jesque https://github.com/gresrun/jesque-web https://github.com/gresrun/jesque-web