4 ms·

I guess the team has Q1 OKR to meet and/or error budget to spend? Either way, it's bad. I am not saying the bug would have been prevented if the team operates

I guess the team has Q1 OKR to meet and/or error budget to spend? Either way, it's bad. I am not saying the bug would have been prevented if the team operates differently. Brown paper bag release happens. However, the way this incident blew up suggests that the bad release made it to too many users in too short of a time, exactly the behavior incentivized by meeting (or exceeding) an OKR while having some spare error budget.

gowld 6y ago
What do you think "error budget" means? (Hint, it doesn't mean "amount of intentionally deployed errors we allow".) Do you believe WebView should never update because it's never OK to spend error budget?
- room500 6y ago
  That is a strawman argument. Of course WebView should update. The question is whether the structure of incentives contributed to a risky release. If you "play the game", an error budget causes your releases to become riskier and riskier as the budget grows. This is optimal because a riskier release allows you to accelerate changes, allowing you to hit other OKR. The issue is that this will eventually cause very risky releases (where you would slow down otherwise). In a sense, an error budget does eventually become "amount of intentionally deployed errors" because you are incentivized to increase risk until an error does appear.
- topher200 6y ago
  Interestingly, the Google SRE book[1] mentions scenarios where they practice exactly this: > The solution to this Chubby scenario is interesting: SRE makes sure that global Chubby meets, but does not significantly exceed, its service level objective. In any given quarter, if a true failure has not dropped availability below the target, a controlled outage will be synthesized by intentionally taking down the system. I doubt there are any customer-facing systems that practice this technique. Just super interesting that it exists. [1]: https://sre.google/sre-book/service-level-objectives/#xref_risk-management_global-chubby-planned-outage https://sre.google/sre-book/service-level-objectives/#xref_r...

gowld 6y ago

What do you think "error budget" means? (Hint, it doesn't mean "amount of intentionally deployed errors we allow".) Do you believe WebView should never update because it's never OK to spend error budget?

room500 6y ago
That is a strawman argument. Of course WebView should update. The question is whether the structure of incentives contributed to a risky release. If you "play the game", an error budget causes your releases to become riskier and riskier as the budget grows. This is optimal because a riskier release allows you to accelerate changes, allowing you to hit other OKR. The issue is that this will eventually cause very risky releases (where you would slow down otherwise). In a sense, an error budget does eventually become "amount of intentionally deployed errors" because you are incentivized to increase risk until an error does appear.
topher200 6y ago
Interestingly, the Google SRE book[1] mentions scenarios where they practice exactly this: > The solution to this Chubby scenario is interesting: SRE makes sure that global Chubby meets, but does not significantly exceed, its service level objective. In any given quarter, if a true failure has not dropped availability below the target, a controlled outage will be synthesized by intentionally taking down the system. I doubt there are any customer-facing systems that practice this technique. Just super interesting that it exists. [1]: https://sre.google/sre-book/service-level-objectives/#xref_risk-management_global-chubby-planned-outage https://sre.google/sre-book/service-level-objectives/#xref_r...