5 ms·
I'm curious as to what was so difficult with picking up Celery? I found it very straightforward, not appreciably more complex than what you've laid out. Not t
by Steeeve 9y ago
I'm curious as to what was so difficult with picking up Celery? I found it very straightforward, not appreciably more complex than what you've laid out.
Not that Celery is the only way to go. This is good work - I'm just interested in the why.
- omegote 9y agoThe number of options celery has is overwhelming. Also, the concepts used by rabbitmq and celery (the most common setup) are not straightforward to understand: when you mix queues, exchanges and routing keys you'd better stop and get a good grasp of those elements before going any further.
- whalesalad 9y agoYep. Celery + RabbitMQ is a pain in the ass for very simple cases. RabbitMQ is more sophisticated than most basic projects require.
- deleted 9y ago[deleted]
- koolba 9y agoI have yet to encounter a legit use case for RabbitMQ. I'm sure they exist in theory, but it's overkill in every situation that I've seen it deployed.
- fernandotakai 9y agowell, depends on your load! at work, we have millions of message per minute (our whole infrastructure is based on rabbitmq). rabbitmq can handle that load without any issues. also, rabbitmq has an amazing admin interface, monitoring is really easy (most things you use for monitoring support it) and the whole thing has really sane defaults (imho). all in all, it's one of my favorites pieces of software ever!
- scardine 9y agoCelery + Redis is way simpler.
- fernandotakai 9y agohonestly, i always found quite easy to setup. do you remember your issue? (btw, i remember people having problems with celery because of they way they imported the tasks. if you always do full imports (as in, from a.b.c import tasks instead of from . import tasks) you shouldn't have any issues).
- adamcharnock 9y agoI actually found the RabbitMQ tutorial docs[1] to be an excellent introduction to the concepts, some of the best I've seen. Half an hour to an hour of reading and I had a decent grasp of what was going on. I think it would be overkill for a single application, but with multiple applications it provides some useful concepts and features. It also comes with a web UI which can be handy. [1] https://www.rabbitmq.com/getstarted.html https://www.rabbitmq.com/getstarted.html PS. I'll stop evangelising soon, I promise.
- aidos 9y agoWe used Celery and it was a major pain point for us. At the time (it was a couple of years back) there was no way to stop it from taking 2 tasks at a time from the queue. We have a smaller number of long running tasks and it made it very difficult to scale out the work. Also, the monitoring story was pretty rubbish and debugging was hard. Now we use RQ, which is mostly pretty good but we've been slowly adjusting parts of it's behaviour to make it more suitable for us. Recently, we did some work around how it does it's TTL so we could more quickly see where workers had terminated abnormally. We've also changed it so a given worker instance will become increasingly unlikely to take on new work as the machine it's running on (AWS) nears the end of the hour since it was started. As the work load lowers, we can remove workers from the system in the most cost effective way.
- cjhanks 9y agoCertainly, the EC2 instance time-until-nearest-hour should be integrated into virtually every service which queues or has priority.