3 ms·
Celery is neat. Something that sort of bothers me with it though is -- and this is maybe totally something that is my fault, I just can't figure it out -- somet
by methodover 10y ago
Celery is neat. Something that sort of bothers me with it though is -- and this is maybe totally something that is my fault, I just can't figure it out -- sometimes workers will just stop processing things. For unknown reasons. I've set up some alerts to automatically slack me when this happens, so I can go in and see why they're stuck and restart them. Still haven't been able to figure it out. Happens maybe once every month or two.
- harel 10y agoPut a hard time out on the worker and it will be killed and restarted if hanging for X period of time.
- jvoorhis 10y agoIn my experience, celery takes some hand holding to stay alive. I generally configure it to recycle workers after N tasks, in addition to a hard timeout on tasks. I've also found in my configuration, using RabbitMQ as the bus, workers can become non-responsive after receiving too large a message.
- methodover 10y agoYeah, I've thought about it. I guess I've been holding out hope that I could figure out the underlying cause. What's wierd is, it always affects just one type of worker -- the workers that handle sending an email. And it happens on all workers simultaneously, on all servers, of that type. Just that type. Meh. I should just put a harakiri timeout like you suggest.