4 ms·
Never done any API/HTTP stuff with lambda, but I've built many ETL pipelines using it. A few years ago we tried the Kinesis -> Lambda thing, but it failed dur
by djhworld 8y ago
Never done any API/HTTP stuff with lambda, but I've built many ETL pipelines using it.
A few years ago we tried the Kinesis -> Lambda thing, but it failed during a large traffic event. This was due to the way the Kinesis poller works, and was made worse by the fact that we had two lambdas running off the same Kinesis stream.
The main issue was you can't control the Kinesis poller - meaning we ran into resource contention during this high traffic event and the iterator age fell behind quite drastically. So we abandoned it in favour of EMR + Flink + Apache Beam.
Other then that though, the S3 -> Lambda stuff works perfectly, and has been running for 3+ years with no issues.
- jonathan-kosgei 8y agoWe also found that using lambda to process kinesis logs was incredibly slow and definitely not the best use case for lambda. We had much better results running a server with a couple of kinesis consumers and redis. This setup was able to process hundreds of millions of records in near real time.