3 ms·

Hi, thank you for taking the time. I have a couple questions: * Why is the DB scrape written as json instead of directly proto/avro/parquet? Isn't it a lot more

Hi, thank you for taking the time. I have a couple questions: * Why is the DB scrape written as json instead of directly proto/avro/parquet? Isn't it a lot more costly to store and to handle? * How many events can aws lambda scale to in this kind of architecture?

iamsomewalrus 8y ago
Did we just become best friends with our “I am the Walrus” references? The DB scrape uses a template from Data Pipeline that under the hood uses the Dynamo DB Scan API. Not really surprising, but that API uses JSON. I wanted to use as much off the shelf software as I could to get data into Athena. In this architecture Lambda is only used to listen to the SNS topic that fires when the Data Pipeline job succeeds or fails so we’re not pushing the limits of Lambda at all. You’d probably hit an EC2 limit Witt Data Pipeline before hitting your Lambda limit on the account.