4 ms·
At our company, we use logrotate to put all logs into an S3 bucket by hostname (with a PutObject only key of course). We then have a service running in EC2 that
by aftbit 3y ago
At our company, we use logrotate to put all logs into an S3 bucket by hostname (with a PutObject only key of course). We then have a service running in EC2 that does a cached get of those logs (based on hostname and timestamp range) then runs ripgrep over it and returns the lines that match.
It works well and requires very little maintenance, and it costs a tiny fraction of something more full-featured like Papertrail, NewRelic, etc.
- diarrhea 3y agoWhat’s “cached” referring to here, in the case of the get?
- aftbit 3y agoWe keep recently fetched files on the EC2 instance's EBS volume for a few days (or until they are evicted to make space), as this is faster than re-reading them from S3 each time someone does a search, and it's common to do a number of searches over the same time period when debugging an issue.
- askIoT 3y agoHow does this work if the dev's want to do live debugging etc.
- aftbit 3y agoWe don't handle that case in our S3 upload system. We only upload upon rotation, which adds a latency of around a day (less for very noisy log files). For live debugging, we either replicate the problem in local environment (if possible), or the dev logs into the production server and runs `tail` on the log file.