3 ms·
Ask HN: What do you use to test job queues?
Wondering what people out there use to test parallel processing of jobs through a queue of workers? I spent 3 hours debugging this one bug because I had to rely on logging statements rather than a more programmatic approach.
My stack is python with celery+redis forming the workers & queue.
- patio11 15y ago1) The queues execute model methods. Unit test model methods as normal. 2) Test that methods which should add things to the queue actually do, using mocks or what have you. 3) Build monitoring infrastructure so that if e.g. the queue worker goes down hard in production it restarts automatically and, if this doesn't solve it, you get woken up ASAP. This is far and away my most common infrastructural issue over the last ~3 years.
- dchuk 15y agoto add to this, distribute your worker processes onto multiple machines. That way if you lose a machine, your queue will still process jobs (albeit slightly crippled).