3 ms·
Fair enough and very honest. > But this is a global parameter across all queues so this means we essentially loose the one good feature of celery for small tas
by BiteCode_dev 6y ago
Fair enough and very honest.
> But this is a global parameter across all queues so this means we essentially loose the one good feature of celery for small tasks which is retrying lost tasks within some acceptable timeframe.
Oh, for this you just setup two celery deamon, each one with their own queues and config. I usually don't want my long running task on the same instance than the short ones anyway.
> We were also excited to use it's chaining and chord features but went into a series of bugs we couldn't dig ourselves out of when tasks crashed inside a chord (went into permanent loops). I just declared bankruptcy on these features and we implemented chaining ourselves.
Granted on that one, they not the best part of celery.
Just out of curiosity, which broken and result backend did you use for celery?
I mostly use Redis as I had plenty of problems with Rabbit MQ, and wonder if you didn't have those because of it.
- ramraj07 6y agoOur use case was that the timescale of any of our tasks (depending how complex a query the user makes) can go from 1 minute to 45 minutes. We demoed a new task that occassionally went over the 1 hour mark. It's definitely annoying to have separate queues for these tasks but that might be what we need to do! We use redis. FWIW within the narrow limits of the task properties it's remarkably stable,so no complaints on that!