When I implement a new service, I have to make an executive decision about which metrics to collect (operation counts, durations, storage usage, etc.). Before t

When I implement a new service, I have to make an executive decision about which metrics to collect (operation counts, durations, storage usage, etc.). Before the first incident, I cannot know with 100% certainty which metrics are useful. Only after the incident can I look at the metrics and see which ones did weird things before and during the incident, and that helps drive my decisions on what to alert on. I don't see how I could do it differently. I can only truly know which metrics are useful from experience with past incidents, but I can only gain this experience when the metrics were already collected during the past incident.