6 ms·
If you're doing a slow rollout, it's not always easy to tell whether the thing you're rolling out is the culprit. I've been on the other side of this outage whe
by Karrot_Kream 5y ago
If you're doing a slow rollout, it's not always easy to tell whether the thing you're rolling out is the culprit. I've been on the other side of this outage where we had an outage and suspected a slow change we had been rolling out, especially because we opted something new into it minutes before an incident, only to realize later when the dust settled that it was completely unrelated. When you're running at high scale like Roblox and have lots of monitoring in place and multiple pieces of infrastructure at multiple levels of slow-rollout, outages like this one don't quickly point to a smoking gun.