Since 2017, at two different companies, I’ve sent logs via UDP to Sumo Logic, via their collector hosted in our cluster. Sumo Logic is reasonably priced, super

Since 2017, at two different companies, I’ve sent logs via UDP to Sumo Logic, via their collector hosted in our cluster. Sumo Logic is reasonably priced, super powerful, easy to use, and really flexible. Can’t recommend it enough. We do log collection and per service log rate limiting via https://github.com/NinesStack/logtailer https://github.com/NinesStack/logtailer to make sure we don’t blow out the budget because someone deployed with debug logging enabled. Fluentbit doesn’t support that per service. Logs are primarily for debugging and we send metrics separately. Rate limiting logs encourages good logging practices as well, because people want to be sure they have the valuable logs when they need them. We dashboard which services are hitting the rate limit. This usually indicates something more deeply wrong that otherwise didn’t get caught. This logging setup gives us everything we’ve needed in seven years of production on two stacks.