3 ms·
We use flume forwarding to s3 and then athena to query the logs. Flume processes each logfile with morphline (which is akin to logstash) and parses each rawlog
by natebutler 7y ago
We use flume forwarding to s3 and then athena to query the logs. Flume processes each logfile with morphline (which is akin to logstash) and parses each rawlog into json before pushing to s3.
We used to run an elk stack but hit a bottleneck crunching logs with logstash. We found flume's morphline to be performant enough and the nice property of flume is that you can fanout and write to multiple datasources.
It's ironic, but because Athena is kind of a flaky product (lots of weird Hive Exceptions querying lots of data) and because it's really only good at searching if you know what you're looking for, we're considering having flume write to an elasticsearch cluster (but still persisting logs to long-term storage on s3).
- bagels 7y agoWhat are you using to connect flume to s3? The HDFS sink?
- natebutler 7y agoYah, with the s3a connector
- RhodesianHunter 7y agoI've always wondered why companies for whom logs are important but not their core focus / product bother implementing stuff like this themselves. Surely a saas service that does just logs can do it cheaper and with more features? Is it compliance?
- natebutler 7y agoWe are in the unfortunate circumstance where we have high traffic but are budget conscious. S3 is cheap and Athena is as well. We significantly reduced cost moving from a fairly large elk cluster on ec2 to a handful of flume machines running morphline. We’ve looked at datadog and scalyr and even went so far as to implement a flume sink to scalyr but the scalyr quote was way too high.