25 ms·

Oh wow, I was so close to embarking on a Python task queue the other day because Celery was driving me nuts. I just wanted various apps/processes to be able to

Oh wow, I was so close to embarking on a Python task queue the other day because Celery was driving me nuts. I just wanted various apps/processes to be able to fire and listen for events, something that AMQP with a topic exchange seemed great for[1]. The thing that really tripped me up was the overlap in terminology between the AMQP protocol and Celery. Sometimes a queue means a queue, but sometimes it means an exchange. How do task names and routing keys interact? Why does the celery worker die when you listen to more than 3 queues on python 3.6 [2]? Don't get me wrong, for basic stuff celery is great, albeit with a bit of a learning curve for the beginner. (And who knows, maybe I'm just missing something when it comes to more complex applications.) I also think the devs have done a great job and are super responsive. I think they also made a wise choice to chop features a little while back. What I would really love to see (create?) is a cut-down python task queue which deals solely and specifically with some kind of AMQP (or similar) broker. Also with a task scheduler with a UI, that is something I'm loathed to lose. Maybe this seems ridiculous, I get the impression that everyone is talking about microservices, ZeroMQ, service discovery, billions of messages per picosecond etc. I'm talking about medium size projects where a maybe a dozen apps just need to coordinate a bit. I feel like this middle ground lacks solutions. Sorry for the ranty comment. I suspect I'm just trying to talk myself into creating this at some point. [1] https://www.rabbitmq.com/tutorials/tutorial-five-python.html https://www.rabbitmq.com/tutorials/tutorial-five-python.html [2] https://github.com/celery/kombu/issues/675 https://github.com/celery/kombu/issues/675

intev 9y ago
Couldn't agree more. I wanted to create a task for my web app to run every 15 minutes and I didn't realize how complicated it was going to be. I'm using AWS beanstalk and I wanted the task to run on 0,15,30,45. The "simplest" way to do that was to add a worker box and create a cron job (it uses AWS SQS as an intermediary). I want to make sending email tasks as well but that means I'm going to have to move deeper into SQS integration and I don't want to be stuck with AWS. My only other option seems to be setting up and maintaining rabbitmq/celery and that looks like a nightmare. :/
- Mayzie 9y ago
  RabbitMQ is super simple to setup and use (from my experience). I've been quite pleased with it.
  - adamcharnock 9y ago
    That has been my experience too. I've setup a small RabbitMQ cluster on kubernetes and it was pretty straightforward. It also comes with a handy web UI which was more than I expected. The hardest part seems to be answering the "Why isn't X receiving messages from Y?" question. Although it didn't help that I got '*' and '#' switched around in my head (seriously, '#' is the wildcard?!)
- mabcat 9y ago
  The other "simplest" way is to run the cron job on the web workers, and put something in the job to abort unless this host is the first web-worker instance. (Otherwise your jobs would run multiply on each instance.) That's what we're doing to kick off scheduled tasks for Django and it works well. It'd be very messy if the scheduled tasks created load because backend load in the web layer would really confuse autoscaling, but ours don't. I'm also running little celery instances on each web worker, using Django as the backend. No SQS/Redis/RabbitMQ, no huge AWS lock in, no having to set up a backend worker layer. In general it's been fine for a year. You don't get good visibility into processing queue length etc but we usually have four bone-idle web workers so there is no queue length. If that might be a fit for your situation I could bang together a blog post or something.
  - intev 9y ago
    Oh interesting. I also recently came across Huey and TaskTiger that seem to be doing similar things. Have you looked into those?
- goliatone 9y ago
  You could create a stepped lambda function that emits topic every minute with the date and time in it, something like YY.MM.DD.h.m and then register one function handler for multiple topics, I.e. .0, .15,.30, .45. I use this concept extensively in a few edge devices for which I don't want to build dedicated/centralized infrastructure. It works rather well. There seems to be some drift and the timing is not 100% accurate but is ok in my setup

intev 9y ago

Couldn't agree more. I wanted to create a task for my web app to run every 15 minutes and I didn't realize how complicated it was going to be. I'm using AWS beanstalk and I wanted the task to run on 0,15,30,45. The "simplest" way to do that was to add a worker box and create a cron job (it uses AWS SQS as an intermediary). I want to make sending email tasks as well but that means I'm going to have to move deeper into SQS integration and I don't want to be stuck with AWS. My only other option seems to be setting up and maintaining rabbitmq/celery and that looks like a nightmare. :/

Mayzie 9y ago
RabbitMQ is super simple to setup and use (from my experience). I've been quite pleased with it.
- adamcharnock 9y ago
  That has been my experience too. I've setup a small RabbitMQ cluster on kubernetes and it was pretty straightforward. It also comes with a handy web UI which was more than I expected. The hardest part seems to be answering the "Why isn't X receiving messages from Y?" question. Although it didn't help that I got '*' and '#' switched around in my head (seriously, '#' is the wildcard?!)
mabcat 9y ago
The other "simplest" way is to run the cron job on the web workers, and put something in the job to abort unless this host is the first web-worker instance. (Otherwise your jobs would run multiply on each instance.) That's what we're doing to kick off scheduled tasks for Django and it works well. It'd be very messy if the scheduled tasks created load because backend load in the web layer would really confuse autoscaling, but ours don't. I'm also running little celery instances on each web worker, using Django as the backend. No SQS/Redis/RabbitMQ, no huge AWS lock in, no having to set up a backend worker layer. In general it's been fine for a year. You don't get good visibility into processing queue length etc but we usually have four bone-idle web workers so there is no queue length. If that might be a fit for your situation I could bang together a blog post or something.
- intev 9y ago
  Oh interesting. I also recently came across Huey and TaskTiger that seem to be doing similar things. Have you looked into those?
goliatone 9y ago
You could create a stepped lambda function that emits topic every minute with the date and time in it, something like YY.MM.DD.h.m and then register one function handler for multiple topics, I.e. .0, .15,.30, .45. I use this concept extensively in a few edge devices for which I don't want to build dedicated/centralized infrastructure. It works rather well. There seems to be some drift and the timing is not 100% accurate but is ok in my setup