I spent over 15 years of the start of my career at startups, and was typically the one woken up by alerts and/or the operations team (when there was one) to hel

I spent over 15 years of the start of my career at startups, and was typically the one woken up by alerts and/or the operations team (when there was one) to help decode alerts. That really shaped my logging philosophy around log levels. FATAL = The system is basically non-functional, even in some partially-working way. Alerts fire. ERROR = Something went wrong in some part of the system. The system may be mostly or partially working, but an important thing is broken and should be fixed. Depending on the where the error is, it may need to be immediately addressed or at least very soon. WARNING = Something isn't working right, but it's possibly transient and the system has was a handling this via retry or other workarounds. This should be addressed at some point if possible, but shouldn't require immediate intervention. INFO = This is the log level that the system is designed to run at. It provides enough human-readable context to see how the system is running. Most importantly, it provides just enough information provide context to WARNINGS/ERRORS/FATALS that may occur somewhere in a given request. The goal is to balance information w/performance. DEBUG = All the things, all the time. Default mode for developers, and when diagnosing issues that require much more depth of information than provided at INFO level. It is VERY RARE to run at this level in production, as INFO logs should provide enough request context for a developer to reproduce most issues, but can be enabled for short bursts in production on one server in a cluster, or some other limited form to capture necessary information. A guiding philosophy on all logs is to persist them in a central location (and locally w/rollover just in case persistence is broken), and is to flow some request or context ID through the system so logs from a single request can be easily identified and related log statements correlated with each other. At my last startup, it was almost a year effort to clean up the logs. Such things like app pool crash on startup were logged at DEBUG, and the application starting successfully was logged at ERROR. Combined with an effort to improve overall quality, hit a first milestone of going 3 months without the ops team requiring escalation to developers outside of business hours (i.e. they could identify and fix things based on logs). Eventually, out of business hours alerts because a "I don't remember the last one" thing.