4 ms·
Instead of enquing a job you’d just write to an SQS queue and have a lambda as your “job handler” using the queue as an event source. Boom, no need for Active
by ryanmarsh 8y ago
Instead of enquing a job you’d just write to an SQS queue and have a lambda as your “job handler” using the queue as an event source.
Boom, no need for Active Job.
Welcome to Serverless.
There’s going to be a lot of people trying to do things the “rails way” on serverless and it’s going to be a little confusing for a while. Many will write it off, some will creat innovative new gems. And of course the Ruby community will come up with the best abstraction for SAM templates and deployment, because we like nice tools.
- otterley 8y agoNo need to use SQS here, assuming you don't need strict ordering behavior, since SNS can use Lambda functions as notification targets. The nice thing about SNS is that it can automatically fan out to multiple Lambda functions in parallel.
- ryanmarsh 8y agoIndeed that is true, SNS can fan out to Lambda. It’s my understanding (I welcome a correction) that ActiveJob is more sequential like SQS + Lambda.
- scarface74 8y agoMore like SQS -> Multiple lambda instances Unless you set the concurrency to one. But as always when working with any queue, don’t depend on only once processing or in order processing. AWS does have FIFO queues but they don’t work with lambda directly.
- freedomben 8y agoBe careful the if there is possibility for a job to fail and need retry. SQS visibility and automatic dead lettering can give you this basically for free. I don't think SNS will.
- otterley 8y agoLambda functions can be configured to use SNS for dead letter queues as well. See https://docs.aws.amazon.com/lambda/latest/dg/retries-on-errors.html https://docs.aws.amazon.com/lambda/latest/dg/retries-on-erro... (event sources that aren't stream-based).
- freedomben 8y agoAh, very nice. Thanks for the correction.
- scarface74 8y agoUsing SQS as an event source, it will also fanout with the advantage of being able to process up to 10 messages at once, dead letter queues for SQS make a lot more sense than lambda DLQs etc. But, I prefer to use SNS for producers and SQS for consumers and assign the SQS queue to a topic. With SNS+SQS you get a lot more flexibility.
- social_quotient 8y agoCurious a/your use case or two for the “fan out” capability here. Do you mind sharing?
- scarface74 8y ago1. You can have one SNS topic, send messages to it with attributes and subscribe to different targets based on the attribute. 2. You can have one SNS message go to different targets. There may be more than one consumer interested in an event. 3. “Priority” processing. I can have the same lambda function subscribed to both an SQS queue and an SNS topic. We do some things in batch where the message will go to SNS -> SQS -> lambda. But with messages with a priority of “high” they will go directly from SNS -> Lambda but they will also go through the queue. High priority messages are one off triggered by a user action.
- otterley 8y agoBy “fan out” I mean that SNS will execute as many Lambda functions as needed to handle the published events as they occur. This gives you much greater concurrency than a typical pub/sub architecture where you typically have a ceiling on the subscriber count or are forced to wait on consumers to process the earlier enqueued messages before processing subsequent ones (i.e., the head-of-line blocking problem).
- philwelch 8y agoAnd then what if your Lambda function fails because a service dependency is down? The answer is, it retries twice and then you can have it go to a deadletter queue. If you want to have any configuration control over your retry or deadletter behavior, SQS is really handy. It's also a useful pattern to, instead of configuring the SQS event itself as your Lambda trigger, have a scheduled trigger and then perform health checks on your service dependencies before pulling messages from the queue. This technique saves you wasted retries and even gives you more fine-grained control over deleting messages (the SQS mechanism for saying, "this is done and you don't need to retry it after the timeout). The SQS trigger, in contrast, deletes the entire batch of messages if the Lambda exits cleanly and does not delete any of them if the Lambda does not exit cleanly, which is a bit messy. Also, you can have an SQS queue subscribe to multiple SNS topics, which means you can set up separate SNS topics for backfill or other use cases. This is especially useful in cross-account use cases, where often your online events are coming from an SNS from a different account, but you want to be able to inject your own testing and backfill events.
- scarface74 8y agoI usually don’t let the lambda SQS back end control message deletion for just that reason. I delete individual messages as I process them and throw an exception if any fail.
- otterley 8y agoWhat problems have you encountered using the same SNS topic as the dead letter queue for the Lambda function?
- blairanderson 8y agothis is the truth! excited to see what people come up with!