4 ms·

Google is tracking the incident here: https://status.search.google.com/incidents/hySMmncEDZ7Xpaf9i32C https://status.search.google.com/incidents/hySMmncEDZ7Xpaf

Google is tracking the incident here: https://status.search.google.com/incidents/hySMmncEDZ7Xpaf9i32C https://status.search.google.com/incidents/hySMmncEDZ7Xpaf9i... Disclosure: I work at Google but not on this. This was linked from the Bugzilla bug.

robin_reala 3y ago
What I don’t understand is why this isn’t just a rollback, or at worst a revert commit and redeploy. I can forgive an issue with a slightly obscure browser, but the fix should be trivial for Google engineers?
- Moldoteck 3y ago
  maybe not that easy if they use monorepos
- chx 3y ago
  Well, as these things usually go... There's a feature important to Middle Manager 32456. You can't just revert it for a not-Google browser. That's just a no-go. So a fix needs to be developed, QA'd, rolled. I presume it's going to be an out-of-schedule roll so it'll probably involve some bureaucracy. (One of my clients is a few thousand people public company and even there a hotfix requires QA lead approval.) Nothing ever is simple.
- egeozcan 3y ago
  Perhaps it's been a while since it's broken? url-bar search works so this seem to only happen if you navigate to google.com directly. How many people navigate to google to search, while using firefox on android? Just a hunch though. OTOH, it's amazing that apparently they don't have UI tests for FF mobile.
  - shadowgovt 3y ago
    > OTOH, it's amazing that apparently they don't have UI tests for FF mobile. Why? They don't have UI tests for Opera on the Nintendo Wii either, and at this point I bet the install base for Wii-Opera is still larger than the install base for FF mobile. TBH, when I was there it surprised me that Google didn't have a dedicated hardware test-bench room with rows upon rows of browser deployments that every UI change needed to be burn-in tested on, but... They don't. They never did. In general, their strategy is to be nimble and deploy rapidly (with the expectation they can roll back rapidly). In that context, it actually makes sense why they don't have that warehouse of test-bench installations... They'd slow-down rollbacks as well as rollouts. A handful of projects have dedicated testing targets. They're driven mostly by the ideology of individual Googlers (some people really like Firefox) and a handful of high-value users that have specific installs Google isn't interested in pissing off. Since they do very little (relatively speaking) B2B business, that's a very short list of names.
    - mook 3y ago
      > Next update will be within 12 hours. (From the status page linked above.) That sounds pretty far from being nimble and deploying rapidly. Not really a knock on the people doing the work — doing stuff in high stress sucks. But it's clear that they're intentionally deprioritizing a competitor.
      - kevindamm 3y ago
        Only way to really test that is to break search for chrome and see how long that takes to rollback.
      - KirinDave 3y ago
        For one, that "deprioritize a competitor" is not clear at all. Why would that be so? Isn't it far more likely, given the rarity of these events, that a test regression occurred or some other subtle issue rather than assumed malfeasance? For two, that "next update in 12 hours" is user comms. For me, at least, google.com works fine both on curl and my browser. That's a fairly normal cadence for big companies. On the larger point about "nimble and deploying rapidly", the people who generally brag about "being nimble and deploying rapidly" almost never serve an even 1/100th the audience the size of Google.com does, and it's really questionable if, at that scale, you actually want to risk global regressions even on trivial bugs. So I don't know what that user is talking about, and I agree with you that they are obviously not that. That approach may be antithetical to the modern startup engineer frantic to prove their stock's hypothetical worth to their investors, unconcerned about trivial revenue loss from frontpage issues because of whatever latest node.js drama nuked their continuously deployed website. But the fact that "the landing search page is broken for 1% of users in a rare but public use case" is news at all is because Google's approach for search sets our expectations that this won't happen.
        chias 3y ago
        I think the parent comment's point is: if this were affecting a version of Google-branded Chrome with similar market share, do you think we would still be getting "next update in 12 hours"? I remember an internal mantra in Google along the lines of "if you break something in production, roll back first, ask questions about how it happened later, even if you think it's a simple fix". It feels telling that this is not what is happening in this circumstance.
- kekub 3y ago
  It appears that the problem is related to the version string transmitted by Firefox Mobile. If they were to send an older user agent string, the issue would likely be resolved. However, any rollback would need to be implemented by Firefox, and it seems they are not at fault here. From Google's standpoint, this issue may be considered a "new" bug, which means they need to conduct an investigation and address it. Consequently, a rollback is not a viable solution for them.
  - dTP90pN 3y ago
    Why would Firefox have to "rollback" their UA string back to version 64, released 6 years ago (2018)? That seems utterly ridiculous for a server side UA sniffing bug rolled out by the Google Search Team.

robin_reala 3y ago

What I don’t understand is why this isn’t just a rollback, or at worst a revert commit and redeploy. I can forgive an issue with a slightly obscure browser, but the fix should be trivial for Google engineers?

Moldoteck 3y ago
maybe not that easy if they use monorepos
chx 3y ago
Well, as these things usually go... There's a feature important to Middle Manager 32456. You can't just revert it for a not-Google browser. That's just a no-go. So a fix needs to be developed, QA'd, rolled. I presume it's going to be an out-of-schedule roll so it'll probably involve some bureaucracy. (One of my clients is a few thousand people public company and even there a hotfix requires QA lead approval.) Nothing ever is simple.
egeozcan 3y ago
Perhaps it's been a while since it's broken? url-bar search works so this seem to only happen if you navigate to google.com directly. How many people navigate to google to search, while using firefox on android? Just a hunch though. OTOH, it's amazing that apparently they don't have UI tests for FF mobile.
- shadowgovt 3y ago
  > OTOH, it's amazing that apparently they don't have UI tests for FF mobile. Why? They don't have UI tests for Opera on the Nintendo Wii either, and at this point I bet the install base for Wii-Opera is still larger than the install base for FF mobile. TBH, when I was there it surprised me that Google didn't have a dedicated hardware test-bench room with rows upon rows of browser deployments that every UI change needed to be burn-in tested on, but... They don't. They never did. In general, their strategy is to be nimble and deploy rapidly (with the expectation they can roll back rapidly). In that context, it actually makes sense why they don't have that warehouse of test-bench installations... They'd slow-down rollbacks as well as rollouts. A handful of projects have dedicated testing targets. They're driven mostly by the ideology of individual Googlers (some people really like Firefox) and a handful of high-value users that have specific installs Google isn't interested in pissing off. Since they do very little (relatively speaking) B2B business, that's a very short list of names.
  - mook 3y ago
    > Next update will be within 12 hours. (From the status page linked above.) That sounds pretty far from being nimble and deploying rapidly. Not really a knock on the people doing the work — doing stuff in high stress sucks. But it's clear that they're intentionally deprioritizing a competitor.
    - kevindamm 3y ago
      Only way to really test that is to break search for chrome and see how long that takes to rollback.
    - KirinDave 3y ago
      For one, that "deprioritize a competitor" is not clear at all. Why would that be so? Isn't it far more likely, given the rarity of these events, that a test regression occurred or some other subtle issue rather than assumed malfeasance? For two, that "next update in 12 hours" is user comms. For me, at least, google.com works fine both on curl and my browser. That's a fairly normal cadence for big companies. On the larger point about "nimble and deploying rapidly", the people who generally brag about "being nimble and deploying rapidly" almost never serve an even 1/100th the audience the size of Google.com does, and it's really questionable if, at that scale, you actually want to risk global regressions even on trivial bugs. So I don't know what that user is talking about, and I agree with you that they are obviously not that. That approach may be antithetical to the modern startup engineer frantic to prove their stock's hypothetical worth to their investors, unconcerned about trivial revenue loss from frontpage issues because of whatever latest node.js drama nuked their continuously deployed website. But the fact that "the landing search page is broken for 1% of users in a rare but public use case" is news at all is because Google's approach for search sets our expectations that this won't happen.
      - chias 3y ago
        I think the parent comment's point is: if this were affecting a version of Google-branded Chrome with similar market share, do you think we would still be getting "next update in 12 hours"? I remember an internal mantra in Google along the lines of "if you break something in production, roll back first, ask questions about how it happened later, even if you think it's a simple fix". It feels telling that this is not what is happening in this circumstance.
kekub 3y ago
It appears that the problem is related to the version string transmitted by Firefox Mobile. If they were to send an older user agent string, the issue would likely be resolved. However, any rollback would need to be implemented by Firefox, and it seems they are not at fault here. From Google's standpoint, this issue may be considered a "new" bug, which means they need to conduct an investigation and address it. Consequently, a rollback is not a viable solution for them.
- dTP90pN 3y ago
  Why would Firefox have to "rollback" their UA string back to version 64, released 6 years ago (2018)? That seems utterly ridiculous for a server side UA sniffing bug rolled out by the Google Search Team.