3 ms·
Same with Twilio. We have an internal server that does system alerts. We recently moved it to an IPv6 only host, and a few weeks went by and noticed there were
by jermaustin1 6mo ago
Same with Twilio. We have an internal server that does system alerts. We recently moved it to an IPv6 only host, and a few weeks went by and noticed there were no longer receiving alerts.
Turns out we could not connect to Twilio's API which is IPv4 only.
- tbrownaw 6mo agoSo zero validation after that change?
- vlovich123 6mo agoZero observability and alerting too. Seems like they’re planning to be a productive future member of that team.
- jermaustin1 6mo agoWho? The infrastructure team that did the move didn't even tell anyone. They were decommissioning old servers, and moving the VMs to new hardware. I'm just a lowly developer that had to troubleshoot why SMSes stopped going out.
- vlovich123 6mo agoObservability and alerting is pretty standard devops. Both the dev and infrastructure teams dropped the ball here. But at least as part of remediating this you added alerting to make sure you’d notice when your twillio connection fails in the future, right?
- jermaustin1 6mo agoEven better, infrastructure enabled IPv6, and the issue was closed. In corporate software development, we work the tickets assigned, and keep our KPIs up so that we don't face the wrath of the bean counters.
- jermaustin1 6mo agoCouldn't tell you. I'm not part of the infrastructure team. I wasn't even aware the alerting service was moving. QA found it a couple weeks later when they were testing alerting, and SMSes weren't coming through.