3 ms·
Ask YC: Which queue/background worker solution?
I want to distribute tasks across several servers and processes. I've found several solutions, but couldn't find any useful third party reviews. So can anyone recommend a good solution from their own experience?
Currently, I'm looking at:
- Scarling/Kestrel http://github.com/robey/kestrel/tree/master
- Gearman http://www.danga.com/gearman/
- RabbitMQ or any other AMQP solution (but that looks as overkill)
- rolling my own with REST or MySQL
- RobGR 18y agoI have done this via writing my own system. At one workplace I had perl scripts that ran on every Unix machine there at night (about 60 Sun's on engineer's desktops) that checked with a central repository, locked a task, and downloaded it and ran it and filed the results. I have since done the same with a mysql "task" database, and the client checked in and picked up a task and ran it. In both cases the system was not general and specialized to the particular jobs at hand. It really depends on what type of tasks you have, and what resources you have in terms of computers. Without those details, the best I can say is that rolling your own is not that hard, and you should not be intimidated by it.