4 ms·
I've worked on a number of systems where code was pushed to a staging environment (a persistent functional replica of production where integration tests happen)
by ak217 6y ago
I've worked on a number of systems where code was pushed to a staging environment (a persistent functional replica of production where integration tests happen) and sat there for a week before being allowed in production. A staging setup might have prevented this scenario, since the quota enforcement grace period would expire in staging a week before prod and give the team a week to notice and push an emergency fix.
- deleted 6y ago[deleted]
- papito 6y agoStaging is never hammered with production traffic, often not exposing problems. It's a sanity check for developers and QA, essentially. On the scale of Google, you test in production, but with careful staged canary deployments. Even a 1% rollout is more than most of us have ever dealt with.
- shadowgovt 6y agoIn this specific case, the problem they were into is that the quota system had been different for three months, but the differences were not being enforced. It's very unclear how they would have gone about Canary that period really, the canary should have been done on the quota side. The quota system should have enabled enforcement to 1% of the quota clients. but it turns out it's actually hard to configure that sort of thing. The ways you can slice subsets of Google infrastructure are absolutely holographic, and it costs engineering time to change the slices.