5 ms·

A Simple Pattern for Jobs and Crons on AWS

48 points by douglasfshearer 10y ago

krallja 10y ago
You could also use `sqsd` (an Elastic Beanstalk worker environment) and avoid having to write any queue reading code.
deleted 10y ago
[deleted]
WaxProlix 10y ago
Something similar, and perhaps even simpler: Make a few SNS topics that lambdas can subscribe to - either one per method or one per method type, and then filter more in code - and push notifications to those from your service layer as needed to invoke async workflows. I've been doing this recently and it's pretty pain-free. Been meaning to look at Step Functions for a lot of this async work stuff, but haven't had time yet (plus it's nice to let these things mature a bit before diving in).
luhn 10y ago
I've used SQS for a simple job queue before. It works well and it's nice not to have to worry about more infrastructure. To make things even better, Celery now has an SQS broker, which I'd like to try out. I've also had success with an HTTP subscription in SNS for simple async jobs. Having your tasks be just another HTTP endpoint is really convenient. It's not actually a task queue, but it's often a good-enough substitute. One frustration I've had with it is that Amazon signs SNS messages, but I can't find a Python library for validating the signature. Not even AWS's flagship API client (boto3) has the functionality.
- Ysx 10y ago
  > [...] I can't find a Python library for validating the signature. Not even AWS's flagship API client (boto3) has the functionality. I wrote validatesns [1] for this, give it a try! (Not sure why there's an official PHP validator but not Python...) [1] https://github.com/nathforge/validatesns https://github.com/nathforge/validatesns
Zombieball 10y ago
An alternative would be to just use AWS lambda: > You can create a Lambda function and direct AWS Lambda to execute it on a regular schedule. You can specify a fixed rate (for example, execute a Lambda function every hour or 15 minutes), or you can specify a Cron expression. http://docs.aws.amazon.com/lambda/latest/dg/with-scheduled-events.html http://docs.aws.amazon.com/lambda/latest/dg/with-scheduled-e... It would remove the need of infrastructure for "task workers". Although, depending on your stack / setup, it of course may be easier to have your application tier double as a task worker due to complicated dependencies or startup times. Additionally, I am not sure if everyone is aware but SQS does offer FIFO delivery with guaranteed once-only "processing". There are some interesting caveats to this, definitely worth looking into though :)
- coredog64 10y ago
  Yes, but... AWS Lambda has a limitation on the number of functions you can attach to a scheduled event. What this means is that you wind up with a shitload of 'every 5 minutes' schedulers that all need unique names. WTF Amazon?
  - idbehold 10y ago
    Hook up just one Lambda function to the scheduled event. And have it invoke other Lambda functions.
- deleted 10y ago
  [deleted]
nodesocket 10y ago
Can't wait until Google Functions[1] adds the ability to schedule and define recurring function calls. [1] - https://cloud.google.com/functions/ https://cloud.google.com/functions/
glumpyfish 10y ago
For jobs that take longer than Lambda will allow, I use Datapipeline to run a ShellCommandActivity, which can run a bash script on an EC2 instance.
- bdcravens 10y ago
  Running an ECS task from Lambda is another option.
revicon 10y ago
As simple as this article is trying to make things, it still feels like it's trying to re-create what elastic beanstalk workers already do out of the box... http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-...

krallja 10y ago

You could also use `sqsd` (an Elastic Beanstalk worker environment) and avoid having to write any queue reading code.

deleted 10y ago

[deleted]

WaxProlix 10y ago

Something similar, and perhaps even simpler: Make a few SNS topics that lambdas can subscribe to - either one per method or one per method type, and then filter more in code - and push notifications to those from your service layer as needed to invoke async workflows. I've been doing this recently and it's pretty pain-free. Been meaning to look at Step Functions for a lot of this async work stuff, but haven't had time yet (plus it's nice to let these things mature a bit before diving in).

luhn 10y ago

I've used SQS for a simple job queue before. It works well and it's nice not to have to worry about more infrastructure. To make things even better, Celery now has an SQS broker, which I'd like to try out. I've also had success with an HTTP subscription in SNS for simple async jobs. Having your tasks be just another HTTP endpoint is really convenient. It's not actually a task queue, but it's often a good-enough substitute. One frustration I've had with it is that Amazon signs SNS messages, but I can't find a Python library for validating the signature. Not even AWS's flagship API client (boto3) has the functionality.

Ysx 10y ago
> [...] I can't find a Python library for validating the signature. Not even AWS's flagship API client (boto3) has the functionality. I wrote validatesns [1] for this, give it a try! (Not sure why there's an official PHP validator but not Python...) [1] https://github.com/nathforge/validatesns https://github.com/nathforge/validatesns

Zombieball 10y ago

An alternative would be to just use AWS lambda: > You can create a Lambda function and direct AWS Lambda to execute it on a regular schedule. You can specify a fixed rate (for example, execute a Lambda function every hour or 15 minutes), or you can specify a Cron expression. http://docs.aws.amazon.com/lambda/latest/dg/with-scheduled-events.html http://docs.aws.amazon.com/lambda/latest/dg/with-scheduled-e... It would remove the need of infrastructure for "task workers". Although, depending on your stack / setup, it of course may be easier to have your application tier double as a task worker due to complicated dependencies or startup times. Additionally, I am not sure if everyone is aware but SQS does offer FIFO delivery with guaranteed once-only "processing". There are some interesting caveats to this, definitely worth looking into though :)

coredog64 10y ago
Yes, but... AWS Lambda has a limitation on the number of functions you can attach to a scheduled event. What this means is that you wind up with a shitload of 'every 5 minutes' schedulers that all need unique names. WTF Amazon?
- idbehold 10y ago
  Hook up just one Lambda function to the scheduled event. And have it invoke other Lambda functions.
deleted 10y ago
[deleted]

nodesocket 10y ago

Can't wait until Google Functions[1] adds the ability to schedule and define recurring function calls. [1] - https://cloud.google.com/functions/ https://cloud.google.com/functions/

glumpyfish 10y ago

For jobs that take longer than Lambda will allow, I use Datapipeline to run a ShellCommandActivity, which can run a bash script on an EC2 instance.

bdcravens 10y ago
Running an ECS task from Lambda is another option.

revicon 10y ago

As simple as this article is trying to make things, it still feels like it's trying to re-create what elastic beanstalk workers already do out of the box... http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-...