4 ms·

Couldn't agree more. I wanted to create a task for my web app to run every 15 minutes and I didn't realize how complicated it was going to be. I'm using AWS be

by intev 9y ago

Couldn't agree more. I wanted to create a task for my web app to run every 15 minutes and I didn't realize how complicated it was going to be. I'm using AWS beanstalk and I wanted the task to run on 0,15,30,45. The "simplest" way to do that was to add a worker box and create a cron job (it uses AWS SQS as an intermediary). I want to make sending email tasks as well but that means I'm going to have to move deeper into SQS integration and I don't want to be stuck with AWS. My only other option seems to be setting up and maintaining rabbitmq/celery and that looks like a nightmare. :/

Mayzie 9y ago
RabbitMQ is super simple to setup and use (from my experience). I've been quite pleased with it.
- adamcharnock 9y ago
  That has been my experience too. I've setup a small RabbitMQ cluster on kubernetes and it was pretty straightforward. It also comes with a handy web UI which was more than I expected. The hardest part seems to be answering the "Why isn't X receiving messages from Y?" question. Although it didn't help that I got '*' and '#' switched around in my head (seriously, '#' is the wildcard?!)
mabcat 9y ago
The other "simplest" way is to run the cron job on the web workers, and put something in the job to abort unless this host is the first web-worker instance. (Otherwise your jobs would run multiply on each instance.) That's what we're doing to kick off scheduled tasks for Django and it works well. It'd be very messy if the scheduled tasks created load because backend load in the web layer would really confuse autoscaling, but ours don't. I'm also running little celery instances on each web worker, using Django as the backend. No SQS/Redis/RabbitMQ, no huge AWS lock in, no having to set up a backend worker layer. In general it's been fine for a year. You don't get good visibility into processing queue length etc but we usually have four bone-idle web workers so there is no queue length. If that might be a fit for your situation I could bang together a blog post or something.
- intev 9y ago
  Oh interesting. I also recently came across Huey and TaskTiger that seem to be doing similar things. Have you looked into those?
goliatone 9y ago
You could create a stepped lambda function that emits topic every minute with the date and time in it, something like YY.MM.DD.h.m and then register one function handler for multiple topics, I.e. .0, .15,.30, .45. I use this concept extensively in a few edge devices for which I don't want to build dedicated/centralized infrastructure. It works rather well. There seems to be some drift and the timing is not 100% accurate but is ok in my setup
- intev 9y ago
  I can't, or rather, I don't want to use lambda because my functions need access to both the DB and the open internet. The only way I can get that working is to add a NAT Gateway to my VPC which has a relatively high fixed monthly cost. :/
  - goliatone 9y ago
    You emit a topic from the lambda function, your handlers can be hosted anywhere and can access anything they need. I think you misunderstood
crdoconnor 9y ago
>I wanted to create a task for my web app to run every 15 minutes Cron would probably be a better bet then. I love celery but I'm not a fan of celery beat.

by intev 9y ago

Mayzie 9y ago

RabbitMQ is super simple to setup and use (from my experience). I've been quite pleased with it.

adamcharnock 9y ago
That has been my experience too. I've setup a small RabbitMQ cluster on kubernetes and it was pretty straightforward. It also comes with a handy web UI which was more than I expected. The hardest part seems to be answering the "Why isn't X receiving messages from Y?" question. Although it didn't help that I got '*' and '#' switched around in my head (seriously, '#' is the wildcard?!)

mabcat 9y ago

The other "simplest" way is to run the cron job on the web workers, and put something in the job to abort unless this host is the first web-worker instance. (Otherwise your jobs would run multiply on each instance.) That's what we're doing to kick off scheduled tasks for Django and it works well. It'd be very messy if the scheduled tasks created load because backend load in the web layer would really confuse autoscaling, but ours don't. I'm also running little celery instances on each web worker, using Django as the backend. No SQS/Redis/RabbitMQ, no huge AWS lock in, no having to set up a backend worker layer. In general it's been fine for a year. You don't get good visibility into processing queue length etc but we usually have four bone-idle web workers so there is no queue length. If that might be a fit for your situation I could bang together a blog post or something.

intev 9y ago
Oh interesting. I also recently came across Huey and TaskTiger that seem to be doing similar things. Have you looked into those?

goliatone 9y ago

You could create a stepped lambda function that emits topic every minute with the date and time in it, something like YY.MM.DD.h.m and then register one function handler for multiple topics, I.e. .0, .15,.30, .45. I use this concept extensively in a few edge devices for which I don't want to build dedicated/centralized infrastructure. It works rather well. There seems to be some drift and the timing is not 100% accurate but is ok in my setup

intev 9y ago
I can't, or rather, I don't want to use lambda because my functions need access to both the DB and the open internet. The only way I can get that working is to add a NAT Gateway to my VPC which has a relatively high fixed monthly cost. :/
- goliatone 9y ago
  You emit a topic from the lambda function, your handlers can be hosted anywhere and can access anything they need. I think you misunderstood

crdoconnor 9y ago

>I wanted to create a task for my web app to run every 15 minutes Cron would probably be a better bet then. I love celery but I'm not a fan of celery beat.