3 ms·

People do not understand the value of classifying alerts as useful after the fact. At Netflix we built a feature into our alert systems that added a simple but

People do not understand the value of classifying alerts as useful after the fact. At Netflix we built a feature into our alert systems that added a simple button at the top of every alert that said, "Was this alert useful?". Then we would send the alert owners reports about what percent of people found their alert useful. It really let us narrow in on which alerts were most useful so that others could subscribe the them, and which were noise, so they could be tuned or shut off. That one button alone made a huge difference in people's happiness with being on call.

tracker1 2y ago
I worked on a small team that covered a relatively big site where there were so many alerts it was simply hard to track... They were all sent over email to a group list and most would just delete. I spent about 3 months, each day trying to triage into buckets based on activity and dealing with whatever was causing the most alerts each day. Some came down to just tamping out classes of 4xx errors that should never have been in the email/alert system to begin with. Others came down to indexes to reduce load/locking/contention on some db tables. Others still were much harder to dig into. Will say at the end of the 3 months, there was only a trickle of emails a day and the notifications were taken much more seriously after not being so overwhelming as to being ignored altogether. edit: This was just the first thing I did each day was deal with one problem, then moving to new feature work... It wasn't assigned, as the company would always prioritize new feature work, it was just something I did for my own sanity.
aray07 2y ago
Yeah, that was one of the goals we had. We try to classify when an alert comes up and let the engineer give us feedback. We use that to generate a report so that teams have visibility into which alerts are causing the most amount of noise.
Jolter 2y ago
If each and every alert has an owner, you’ve solved half of the cultural problem already. Good on you!

tracker1 2y ago

I worked on a small team that covered a relatively big site where there were so many alerts it was simply hard to track... They were all sent over email to a group list and most would just delete. I spent about 3 months, each day trying to triage into buckets based on activity and dealing with whatever was causing the most alerts each day. Some came down to just tamping out classes of 4xx errors that should never have been in the email/alert system to begin with. Others came down to indexes to reduce load/locking/contention on some db tables. Others still were much harder to dig into. Will say at the end of the 3 months, there was only a trickle of emails a day and the notifications were taken much more seriously after not being so overwhelming as to being ignored altogether. edit: This was just the first thing I did each day was deal with one problem, then moving to new feature work... It wasn't assigned, as the company would always prioritize new feature work, it was just something I did for my own sanity.

aray07 2y ago

Yeah, that was one of the goals we had. We try to classify when an alert comes up and let the engineer give us feedback. We use that to generate a report so that teams have visibility into which alerts are causing the most amount of noise.

Jolter 2y ago

If each and every alert has an owner, you’ve solved half of the cultural problem already. Good on you!