We just had something similar earlier this year on one of our systems where it sends alerts to clients, but it failed to send them (in our case though the log s

We just had something similar earlier this year on one of our systems where it sends alerts to clients, but it failed to send them (in our case though the log showed they sent). The solution for us took on three unique steps: 1. Make sure unit tests exist for all permutations of failure (network, bad keys etc) and are mandatory to validate that all alerts are operational when new code is being created. Add a commit hook if needed. 2. We created a specific new internal demo account where upon new code being deployed we run a partially automated process (someone needs to push a button right now) where data that triggers specific different types of alerts is inserted into our system to force an alert to be sent out (this is done in production). We did this because we found that unit testing covered one scenario, but a number of other scenarios exist which can be unique to production, e.g. network access, permissions, different keys etc. 3. Add daily checks (or more/less often depending on the need) that all third parties (even internal) your code depends on for alerts are up and operational. We have one where we do hourly checks on the system now to insure it has not failed, another one is only done weekly because the risk of not getting the alert is super low and not client facing. We found in some cases we have one vendor who was sending success even though the alert was never delivered (it basically ate our request), which was found through these automated checks because say 95% went fine but we would miss an hour here or there -- that allowed us to go back through the logs and get with the vendor to see wtf. All of this seems pretty straight forward and obvious honestly, and we had most of it in place but found it just wasn't robust enough in some cases and we weren't doing #2 & #3 with enough granularity.