I built an ETL on top of AWS Lambda. I had to pull data from Facebook/Instagram, YouTube, Twitter, and Pinterest on a schedule - once an hour. The pipeline too

I built an ETL on top of AWS Lambda. I had to pull data from Facebook/Instagram, YouTube, Twitter, and Pinterest on a schedule - once an hour. The pipeline took about 3 minutes to run. I used a CloudEvent cron to kick it off, a few lambdas, stored the raw data in S3, had another lambda massage that data, stored the final data into S3, and put a message into SQS letting my OLTP system know that it could bring it in. Worked great. Were there other ways to build it? Sure. Could have done it on the same hosts that my OLTP system was running on. But the amount of memory it used for that 3 minutes doubled the amount that was needed on the host normally. Yes I could have optimized it. But my time was better spent figuring out other issues (I was sole developer on the project). So I just broke it into a bunch of different parts and kicked up to Lambda. Worked great. Never had any issues with it. Never worried about not having resources. I think it cost .2 to .8 USD each run? So ~600 USD a month at the high end. And it took me less than a day to break it up and get it running vs who knows how long to optimize. So it definitely solved a problem I had in a very efficient way, and allowed me to speed it up just by spending money.