4 ms·
We have something overly dumb and simple. We generate about 400GB of logs a day through kinesis to lambda to s3. And a second lambda that reads of s3 and forwar
by etxm 9y ago
We have something overly dumb and simple. We generate about 400GB of logs a day through kinesis to lambda to s3. And a second lambda that reads of s3 and forwards the stuff to wherever using an s3:objectCreated event ...
If we need to replay something we just move the group of files in and out of the folder on s3 (bucket/logs -> bucket/temp -> bucket/logs) and it triggers a new objectCreated and fires the lambda.
We execute the 2 move operations with the AWS cli tool.
The one other thing we do is we have a config file in bucket/config.json that’s says event type X should go to redshift, pg, es, where ever. So we can tweak that before a move to send data to additional data stores.
We follow ELT instead of ETL for streaming into data stores.
- connormcarthur 9y agoDo you find that you have to replay data often? I'd imagine that 99% or more of that 400GB/day gets thrown away before ending up in Redshift.
- odammit 9y agoRarely. We visualize a lot of it in Kibana and our kibana indexes are pruned after N days (configured for each event type). We usually only replay when we want to restore data for querying stuff outside the pruning range. We keep all that data forever.
- angersock 9y agoWhat are you doing that makes a half-terabyte of logs per day? That seems kinda nuts.
- odammit 9y agoAds ads ads! That’s just one system. Our Nginx access logs alone are about 250GB per day (they use that system).