4 ms·
I'm currently working on a project that's implementing this whole pattern using AWS. 1.) Requests are written to a DynamoDB table 2.) a Lambda subscribes to the
by 0x445442 8y ago
I'm currently working on a project that's implementing this whole pattern using AWS. 1.) Requests are written to a DynamoDB table 2.) a Lambda subscribes to the Dynamo event stream 3.) each event from the stream causes the Lambda to make an async call to an external system 4.) A message (with a delay) is sent to SQS from the Lambda that the async call has been made 5.) Another Lambda reads from the SQS queue and updates the original DynamoDB record completing the loop.
The delay in the queue simulates a polling interval to the external system and when a successful response is returned from the external system the Dynamo record is updated with a status telling the Lambda subscribing to the event stream that "the loop" is done.
This setup has to run at fairly significant peak loads and we're relying on Amazon to provision resources reliably enough to pull this off. None of us know if this will work but in terms of abstracting away the complexities of async/concurrent processing it's a novel approach.
- greglindahl 8y agoIt's similar to how you use Linda (1986), which wasn't novel when it was invented, either.
- Matthias247 8y agoThe challenge with that kind of system is flow control. Essentially everything is coupled by queues without a feedback loop. And once the system gets under load, those queues might run fuller and fuller until the whole system fails. It’s actually similar than an in-program model where you just asynchronously queue void functions in threadpools without knowing how many actually run. The fact that you are only running on top of managed services and that AWS can scale really well might mitigate the problem. And if you have to process all events and can not fail early or move backpressure on the initial caller there might not really be a better solution anyway. But you should be aware about it.