4 ms·
Unfortunately I cannot provice much detail about the nature of the application, however I can say that we collect and process various types of events and perfor
by leetbulb 9y ago
Unfortunately I cannot provice much detail about the nature of the application, however I can say that we collect and process various types of events and perform automated actions based on them. The ingress begins at API Gateway, runs through a pipeline of Lambda functions, ends up in a Kinesis firehose, and ultimately in S3 for further processing. The primary cost (~60%) comes from API Gateway acting as the event ingress point, which was a design mistake...though at the time it did make things easier.
We're currently working on migrating the event collectors to a pure CloudFront based solution (inspired by SnowPlow) where the events will be submitted to CloudFront via signed GET requests and the CF logs will be streamed to the same Lambda-based processing pipeline. Doing this will eliminate almost all of the overhead of API Gateway (still required for certain events that cannot be collected via HTTP GET).