4 ms·

That was an extremely high risk change on GCP's part, reminds me of the App Engine days when you'd wake up to find a totally healthy program spamming 500s becau

by galosh 5y ago

That was an extremely high risk change on GCP's part, reminds me of the App Engine days when you'd wake up to find a totally healthy program spamming 500s because they'd make a breaking change without any announcement. It's shocking they're still pulling stuff like this in 2022

agilob 5y ago
It reminds me how YouTube enforced new codec with a few days notice knowing that FireFox doesn't support it, so FF couldn't play most YT videos for over a week.
- faeyanpiraat 5y ago
  Did yt back up or ff updated to fix issue?
  - agilob 5y ago
    Firefox got the codecs working in the next release
- TheGoddessInari 5y ago
  > enforced new codec YouTube turned off flash player as the default in 2015, and VP9 was supported in Firefox at the same time. YouTube still serves h.264, vp9, and av1. I was trying to figure out what this could be referring to.
  - agilob 5y ago
    Not all codecs were ported at the same time, and then not enabled by default, and when enabled by default it was platform dependent, for others where it worked FF was eating all possible CPU resources and videos were glitching. I remember this as I was using Debian and I was active on /r/firefox, where this [1] link was posted 10 times every day [1] https://www.youtube.com/html5 https://www.youtube.com/html5
kevingadd 5y ago
Feels like an especially severe version of the consistent Google pattern of only testing stuff on Chrome, so new updates/features ship in a way that is some degree of broken on Firefox/Safari. For a significant amount of time YouTube had bad performance on Firefox because they chose to use Web Components by default with a horrible polyfill instead of using the old (still working!) html5 version that ran great.
- loulouxiv 5y ago
  Putting in place an infrastructure to test this kind on changes on the 5-10 most popular browser would be, I think, very cheap for a company like Google. I can't help thinking these may be deliberate moves to eat the little market shares of Chrome concurrents. I remember reading here on HN an article written by an ex-Mozilla insider relating the dissonance between the "friendly" Mozilla-Google employees exchanges and the year-long track record of very oddly recurrent "unfortunate mistakes" from Google degrading the Firefox compatibility.
  - magicalist 5y ago
    > Putting in place an infrastructure to test this kind on changes on the 5-10 most popular browser would be, I think, very cheap for a company like Google. The problem wasn't some web server, it's the Firefox backend services running on GCP.
  - loulouxiv 5y ago
    I am trying to find the link, but for the moment I only find comment making, I think, references to it : https://news.ycombinator.com/item?id=19815348 https://news.ycombinator.com/item?id=19815348 https://news.ycombinator.com/item?id=28495546 https://news.ycombinator.com/item?id=28495546 Does anybody here remember enough keywords to find it out ? Edit: I guess it was this Twitter thread https://mobile.twitter.com/johnath/status/1116871231792455686 https://mobile.twitter.com/johnath/status/111687123179245568... Edit 2: The associated HN thread https://news.ycombinator.com/item?id=19662852 https://news.ycombinator.com/item?id=19662852
  - deleted 5y ago
    [deleted]
charcircuit 5y ago
Would a warning have even helped that much? Since HTTP/3 was expected to be working there wouldn't be a cause to worry.
- Semaphor 5y ago
  > Would a warning have even helped that much? Since HTTP/3 was expected to be working there wouldn't be a cause to worry. It might (as mentioned in TFA) have made them think to run some extra tests, which could have caught the bug. But it also would have made the response faster, as they would have known what changed far sooner.
- 0xbadcafebee 5y ago
  The key word there is "expected". When you do Operations for a living, the only thing you can expect is the unexpected. That's why even after you think you've tested a change, you carefully and slowly roll it out a bit at a time, monitoring golden metrics so you can detect a problem, stop the roll-out, and roll back. It sounds like somebody just flipped a giant switch and never checked error rates, connection metrics, anything. Check out this graph: https://hacks.mozilla.org/files/2022/01/crashes-foxstuck2-2048x946.png https://hacks.mozilla.org/files/2022/01/crashes-foxstuck2-20... Think maybe that would indicate somebody needs to roll back the last change? The problem here, as usual, is a disconnect between stakeholders. Google has this service (it seems like the load balancer for their customer?) it wants to change for one reason or another. The customers may or may not have planned for the change Google is making. Google makes the change, but it isn't a stakeholder of the customer (they basically don't care what happens to the customer). So there is no direct feedback loop for the customer to tell Google something is wrong. If Google was at risk of losing business from its customers going down, it would have a strong relationship with those customers and have a way to quickly help diagnose problems and roll back changes if needed. This is a great lesson for all customers to take away: don't depend on people who you don't have a close relationship with.
brabel 5y ago
The client, Firefox, said it supported HTTP/3 though. Otherwise it wouldn't get to use that. I don't think that's as bad as you try to make it... if the client says it supports something then it breaks when it uses it, it's the fault of the client, not the server.
- alisonkisk 5y ago
- throwaway984393 5y ago
  No SRE in the world that is halfway decent at their job would think that way. You never make assumptions about any kind of change, much less a global change to a completely different protocol. Doesn't matter whose fault it is. You just don't introduce any change that has a chance of unexpected behavior without rigorous testing, and you roll it out g r a d u a l l y, and you stop when error rates increase. Google literally wrote the books on SRE. For them to not know better is absurd.
notyourday 5y ago
> That was an extremely high risk change on GCP's part, reminds me of the App Engine days when you'd wake up to find a totally healthy program spamming 500s because they'd make a breaking change without any announcement. It's shocking they're still pulling stuff like this in 2022 reply Lay with the dogs, wake up with the fleas. Google is a shitty company producing shitty products. When you select to do business with Google you select to do business with a shitty company producing shitty products and treating its customers like shit. Hence I fail to understand the Surprised Pikachu face when something like this happens.

by galosh 5y ago

agilob 5y ago

It reminds me how YouTube enforced new codec with a few days notice knowing that FireFox doesn't support it, so FF couldn't play most YT videos for over a week.

faeyanpiraat 5y ago
Did yt back up or ff updated to fix issue?
- agilob 5y ago
  Firefox got the codecs working in the next release
TheGoddessInari 5y ago
> enforced new codec YouTube turned off flash player as the default in 2015, and VP9 was supported in Firefox at the same time. YouTube still serves h.264, vp9, and av1. I was trying to figure out what this could be referring to.
- agilob 5y ago
  Not all codecs were ported at the same time, and then not enabled by default, and when enabled by default it was platform dependent, for others where it worked FF was eating all possible CPU resources and videos were glitching. I remember this as I was using Debian and I was active on /r/firefox, where this [1] link was posted 10 times every day [1] https://www.youtube.com/html5 https://www.youtube.com/html5

kevingadd 5y ago

Feels like an especially severe version of the consistent Google pattern of only testing stuff on Chrome, so new updates/features ship in a way that is some degree of broken on Firefox/Safari. For a significant amount of time YouTube had bad performance on Firefox because they chose to use Web Components by default with a horrible polyfill instead of using the old (still working!) html5 version that ran great.

loulouxiv 5y ago
Putting in place an infrastructure to test this kind on changes on the 5-10 most popular browser would be, I think, very cheap for a company like Google. I can't help thinking these may be deliberate moves to eat the little market shares of Chrome concurrents. I remember reading here on HN an article written by an ex-Mozilla insider relating the dissonance between the "friendly" Mozilla-Google employees exchanges and the year-long track record of very oddly recurrent "unfortunate mistakes" from Google degrading the Firefox compatibility.
- magicalist 5y ago
  > Putting in place an infrastructure to test this kind on changes on the 5-10 most popular browser would be, I think, very cheap for a company like Google. The problem wasn't some web server, it's the Firefox backend services running on GCP.
- loulouxiv 5y ago
  I am trying to find the link, but for the moment I only find comment making, I think, references to it : https://news.ycombinator.com/item?id=19815348 https://news.ycombinator.com/item?id=19815348 https://news.ycombinator.com/item?id=28495546 https://news.ycombinator.com/item?id=28495546 Does anybody here remember enough keywords to find it out ? Edit: I guess it was this Twitter thread https://mobile.twitter.com/johnath/status/1116871231792455686 https://mobile.twitter.com/johnath/status/111687123179245568... Edit 2: The associated HN thread https://news.ycombinator.com/item?id=19662852 https://news.ycombinator.com/item?id=19662852
- deleted 5y ago
  [deleted]

charcircuit 5y ago

Would a warning have even helped that much? Since HTTP/3 was expected to be working there wouldn't be a cause to worry.

Semaphor 5y ago
> Would a warning have even helped that much? Since HTTP/3 was expected to be working there wouldn't be a cause to worry. It might (as mentioned in TFA) have made them think to run some extra tests, which could have caught the bug. But it also would have made the response faster, as they would have known what changed far sooner.
0xbadcafebee 5y ago
The key word there is "expected". When you do Operations for a living, the only thing you can expect is the unexpected. That's why even after you think you've tested a change, you carefully and slowly roll it out a bit at a time, monitoring golden metrics so you can detect a problem, stop the roll-out, and roll back. It sounds like somebody just flipped a giant switch and never checked error rates, connection metrics, anything. Check out this graph: https://hacks.mozilla.org/files/2022/01/crashes-foxstuck2-2048x946.png https://hacks.mozilla.org/files/2022/01/crashes-foxstuck2-20... Think maybe that would indicate somebody needs to roll back the last change? The problem here, as usual, is a disconnect between stakeholders. Google has this service (it seems like the load balancer for their customer?) it wants to change for one reason or another. The customers may or may not have planned for the change Google is making. Google makes the change, but it isn't a stakeholder of the customer (they basically don't care what happens to the customer). So there is no direct feedback loop for the customer to tell Google something is wrong. If Google was at risk of losing business from its customers going down, it would have a strong relationship with those customers and have a way to quickly help diagnose problems and roll back changes if needed. This is a great lesson for all customers to take away: don't depend on people who you don't have a close relationship with.

brabel 5y ago

The client, Firefox, said it supported HTTP/3 though. Otherwise it wouldn't get to use that. I don't think that's as bad as you try to make it... if the client says it supports something then it breaks when it uses it, it's the fault of the client, not the server.

alisonkisk 5y ago
throwaway984393 5y ago
No SRE in the world that is halfway decent at their job would think that way. You never make assumptions about any kind of change, much less a global change to a completely different protocol. Doesn't matter whose fault it is. You just don't introduce any change that has a chance of unexpected behavior without rigorous testing, and you roll it out g r a d u a l l y, and you stop when error rates increase. Google literally wrote the books on SRE. For them to not know better is absurd.

notyourday 5y ago

> That was an extremely high risk change on GCP's part, reminds me of the App Engine days when you'd wake up to find a totally healthy program spamming 500s because they'd make a breaking change without any announcement. It's shocking they're still pulling stuff like this in 2022 reply Lay with the dogs, wake up with the fleas. Google is a shitty company producing shitty products. When you select to do business with Google you select to do business with a shitty company producing shitty products and treating its customers like shit. Hence I fail to understand the Surprised Pikachu face when something like this happens.