5 ms·

How does something like this make it to production? I couldn't reply to any work emails for the greater part of the evening yesterday. Adds more evidence to th

How does something like this make it to production? I couldn't reply to any work emails for the greater part of the evening yesterday. Adds more evidence to the case of "never use Google for your business".

throwdbaaway 6y ago
I guess the team has Q1 OKR to meet and/or error budget to spend? Either way, it's bad. I am not saying the bug would have been prevented if the team operates differently. Brown paper bag release happens. However, the way this incident blew up suggests that the bad release made it to too many users in too short of a time, exactly the behavior incentivized by meeting (or exceeding) an OKR while having some spare error budget.
- gowld 6y ago
  What do you think "error budget" means? (Hint, it doesn't mean "amount of intentionally deployed errors we allow".) Do you believe WebView should never update because it's never OK to spend error budget?
  - room500 6y ago
    That is a strawman argument. Of course WebView should update. The question is whether the structure of incentives contributed to a risky release. If you "play the game", an error budget causes your releases to become riskier and riskier as the budget grows. This is optimal because a riskier release allows you to accelerate changes, allowing you to hit other OKR. The issue is that this will eventually cause very risky releases (where you would slow down otherwise). In a sense, an error budget does eventually become "amount of intentionally deployed errors" because you are incentivized to increase risk until an error does appear.
  - topher200 6y ago
    Interestingly, the Google SRE book[1] mentions scenarios where they practice exactly this: > The solution to this Chubby scenario is interesting: SRE makes sure that global Chubby meets, but does not significantly exceed, its service level objective. In any given quarter, if a true failure has not dropped availability below the target, a controlled outage will be synthesized by intentionally taking down the system. I doubt there are any customer-facing systems that practice this technique. Just super interesting that it exists. [1]: https://sre.google/sre-book/service-level-objectives/#xref_risk-management_global-chubby-planned-outage https://sre.google/sre-book/service-level-objectives/#xref_r...
meibo 6y ago
You never made a mistake that made it through your test suite to production?
- aaomidi 6y ago
  Not on something that everyone depends on that increased crashes by an order of magnitude? Even some basic testing on a real android device would've caught this
- carstenhag 6y ago
  Google is known to A/B test about everything, also, they very often release new updates to core features like this to 0.01% of users first. So it is very weird that they released this so (apparently?) a big chunk of their users
- heavenlyblue 6y ago
  Google products don't really need to move "fast and break" things anymore, they just need to work.
jeffbee 6y ago
Maybe it didn’t make it to production? The affected users might have been in the canary population?
bitmapbrother 6y ago
Do you apply the same philosophy to the thousands of system breaking changes introduced to Windows by Microsoft or is this just targeted?

throwdbaaway 6y ago

I guess the team has Q1 OKR to meet and/or error budget to spend? Either way, it's bad. I am not saying the bug would have been prevented if the team operates differently. Brown paper bag release happens. However, the way this incident blew up suggests that the bad release made it to too many users in too short of a time, exactly the behavior incentivized by meeting (or exceeding) an OKR while having some spare error budget.

gowld 6y ago
What do you think "error budget" means? (Hint, it doesn't mean "amount of intentionally deployed errors we allow".) Do you believe WebView should never update because it's never OK to spend error budget?
- room500 6y ago
  That is a strawman argument. Of course WebView should update. The question is whether the structure of incentives contributed to a risky release. If you "play the game", an error budget causes your releases to become riskier and riskier as the budget grows. This is optimal because a riskier release allows you to accelerate changes, allowing you to hit other OKR. The issue is that this will eventually cause very risky releases (where you would slow down otherwise). In a sense, an error budget does eventually become "amount of intentionally deployed errors" because you are incentivized to increase risk until an error does appear.
- topher200 6y ago
  Interestingly, the Google SRE book[1] mentions scenarios where they practice exactly this: > The solution to this Chubby scenario is interesting: SRE makes sure that global Chubby meets, but does not significantly exceed, its service level objective. In any given quarter, if a true failure has not dropped availability below the target, a controlled outage will be synthesized by intentionally taking down the system. I doubt there are any customer-facing systems that practice this technique. Just super interesting that it exists. [1]: https://sre.google/sre-book/service-level-objectives/#xref_risk-management_global-chubby-planned-outage https://sre.google/sre-book/service-level-objectives/#xref_r...

meibo 6y ago

You never made a mistake that made it through your test suite to production?

aaomidi 6y ago
Not on something that everyone depends on that increased crashes by an order of magnitude? Even some basic testing on a real android device would've caught this
carstenhag 6y ago
Google is known to A/B test about everything, also, they very often release new updates to core features like this to 0.01% of users first. So it is very weird that they released this so (apparently?) a big chunk of their users
heavenlyblue 6y ago
Google products don't really need to move "fast and break" things anymore, they just need to work.

jeffbee 6y ago

Maybe it didn’t make it to production? The affected users might have been in the canary population?

bitmapbrother 6y ago

Do you apply the same philosophy to the thousands of system breaking changes introduced to Windows by Microsoft or is this just targeted?