natebutler

3 ms·

1. 1.▲
   by natebutler 7y ago

   We are in the unfortunate circumstance where we have high traffic but are budget conscious. S3 is cheap and Athena is as well. We significantly reduced cost moving from a fairly large elk cluster on ec2 to a handful of flume machines runnin
2. 2.▲
   by natebutler 7y ago

   Yah, with the s3a connector
3. 3.▲
   by natebutler 7y ago

   We use flume forwarding to s3 and then athena to query the logs. Flume processes each logfile with morphline (which is akin to logstash) and parses each rawlog into json before pushing to s3. We used to run an elk stack but hit a bottleneck