4 ms·
In this specific case, the problem they were into is that the quota system had been different for three months, but the differences were not being enforced. It
by shadowgovt 6y ago
In this specific case, the problem they were into is that the quota system had been different for three months, but the differences were not being enforced.
It's very unclear how they would have gone about Canary that period really, the canary should have been done on the quota side. The quota system should have enabled enforcement to 1% of the quota clients.
but it turns out it's actually hard to configure that sort of thing. The ways you can slice subsets of Google infrastructure are absolutely holographic, and it costs engineering time to change the slices.