10 ms·

There were outages around the same time last year. Somebody in the HN thread commented back then that the employees evaluation and promotion window ends around

There were outages around the same time last year. Somebody in the HN thread commented back then that the employees evaluation and promotion window ends around december/eoy, thus more releases are made. https://en.m.wikipedia.org/wiki/Google_services_outages https://en.m.wikipedia.org/wiki/Google_services_outages

alpb 5y ago
> evaluation and promotion window ends around december Googler here, that is not when those promotion/perf evaluation windows happen.
oblio 5y ago
He he. It's kind of hilarious. Maybe they should stagger reviews to ensure high availability? :-D
- kroolik 5y ago
  Nothing wrong with it. People make errors, distributed systems aren't easy. More frequent changes - more likely a bug was introduced. My post is just a speculation, let's wait for the actual technical details doc from Google.
  - oblio 5y ago
    Well, there are various whitepapers that reformulate this exact truth you've highlighted: issues tend to happen more when changes are made. Regarding the technical details doc, Google will never state that outright in individual postmortems. And they will definitely not draw this to the logical conclusion regarding the spiky yearly activity.
    - yunohn 5y ago
      > logical conclusion regarding the spiky yearly activity Why is it logical? There’s tons of changes being deployed at all times, at all large companies. Across products, verticals, everything - hundreds of changes at any given point. Some of these changes can introduce hard-to-predict bugs in globally distributed systems. Most of the time, external users don’t even notice before they’re fixed. Like another commenter said, performance reviews do not coincide with with the year-end at Google and other companies.
      - oblio 5y ago
        > Like another commenter said, performance reviews do not coincide with with the year-end at Google and other companies. When are they? My bet is that they're roughly around the same time period, even if they don't start or close on December 31st.
      - michaelt 5y ago
        Hypothetically speaking? 11 months a year there's no incentive to cut corners and only do 2 weeks of testing on something that really needs 3. If one month a year rushing things out with a bit less testing is rewarded, I can believe some people would respond to that. Of course, I wouldn't go as far as to call this a "logical conclusion" as the evidence I've seen is very slim.
        fragmede 5y ago
        Except that there are two a year at Google., so it's 10 months of the year, 5 and 5, and Covid's made everything weird.
    - UncleMeat 5y ago
      It can’t be logical because it is based on bad facts. We wrote evals in August. Anybody racing to get things launched before perf did so months ago. The timing described in the top post is just false. We write the next one in February. We are almost exactly half way between perf cycles. Your hunch that this lines up with the end of the year is false.
moffkalast 5y ago
Then they get fired instead, well done ha.
- kroolik 5y ago
  Why would you fire an engineer you have just spent millions to train?
  - goldcd 5y ago
    I'd guess because Europe-wide outages are costing more than millions
    - kreeben 5y ago
      But now you have someone in your team who will never, ever make that same mistake again and should be your new go-to guy for all X related changes (X being DNS or what-have-you). Firing someone with that type of experience does not lead to success. 100% of all devs make huge mistakes, at least once.
      - KronisLV 5y ago
        > But now you have someone in your team who will never, ever make that same mistake again and that should be your new go-to guy for all DNS related changes. I'm not entirely sure that's always true. For example, i've seen people introduce N+1 issues into a codebase, spend evenings fixing them and refactoring code to fix production issues... just to later introduce those very same types of issues. Sure, you can learn from mistakes, have post-mortems and so on (provided that your org even does those and that anyone listens and cares about the conclusions from those), but to me it feels like the most foolproof way is to ensure that no-one can make these mistakes again, be it with a checklist (which tend to be ignored, honestly), or better yet, an automated CI step or a new test suite. In my eyes, it's basically the same as with unit tests - everyone agrees that you need them, but people rarely write enough of them. So if you introduce something to prevent them from not doing what they should, e.g. a quality gate within a CI step which will disallow a merge once the coverage falls below a set margin, suddenly things are a lot better in the long run.
delroth 5y ago
Perf has been over for almost a month now, and the evaluation period was over more than two months ago.
- kroolik 5y ago
  Then it might, indeed, be unrelated. Thanks for clarification!
- samhw 5y ago
  Could it not be knock-on effects of earlier changes? Or indeed post-review changes from scared/precarious engineers?
  - mucle6 5y ago
    I think we are exactly mid way between the end year 2021 perf and mid year 2022 perf, so its not clear to me that this could be to blame
    - skinkestek 5y ago
      That explains: The people who did the stuff have either gotten their promotions and moved on or didn't get a promotion and have given up. :-)
- dspillett 5y ago
  > and the evaluation period was over more than two months ago Excellent time to slack off a bit, make a few mistakes, then come next eval you can point to a marked improvement over the intervening six-to-nine months!
  - roenxi 5y ago
    The line of logic this thread has followed so far suggests there will be a reason for a Google outage every month of the year.
    - 5y ago
inglor 5y ago
I would be very interested in any research about code quality with relation to promotion packets. Ideally acamdemic. I am not sure where to look for it
- rajin444 5y ago
  Is there an objective way to measure code quality? We barely understand the systems we’re writing code for - I don’t see how you could objectively judge the code to manage those systems without fully understanding the systems first. You could have wave some metrics like number of bugs or test coverage, but I can’t think of any (aforementioned included) that aren’t subject to tons of confounding variables.
de6u99er 5y ago
Some people are definitely not gatting a raise.
- jankeymeulen 5y ago
  More likely someone actually will. A blameless postmortem will be written, and the people that will fix the bug or systems issue will have something to work on with high visibility and high impact, which tends to translate in good performance ratings. (Googler, opinions are my own)
  - radishingr 5y ago
    "Impact: fixed processes that led to 8 hr outage" seems like an easy case to make.
trhway 5y ago
November 11 is a public holiday in US. And it seems it has in recent years became a company holiday in many companies too. May be that break in established over many years cadence of workday and holidays plays a role here, like say affecting smoothness of support transition between US-EMEA-APJ.
- mikelward 5y ago
  Veterans Day is not a day off for US Googlers.
deleted 5y ago
[deleted]
kixiQu 5y ago
Far more likely IMO to coincide with trying to get Exciting New Features out before release freezes for some conference or other.
deanCommie 5y ago
And this is why nobody serious should build on GCP. Google's more interested in placating their primadonna engineers than solving customer problems.
bamboozled 5y ago
There is also a lot of pressure on engineers around black Friday and Cyber Monday to get things done before any change embargoes come into place. This is coming from someone who just worked 16 hours straight to scale things before a change freeze. I don't know how or if this would impact Google, but I'm sure someone there has at least thought about those dates.

alpb 5y ago

> evaluation and promotion window ends around december Googler here, that is not when those promotion/perf evaluation windows happen.

oblio 5y ago

He he. It's kind of hilarious. Maybe they should stagger reviews to ensure high availability? :-D

kroolik 5y ago
Nothing wrong with it. People make errors, distributed systems aren't easy. More frequent changes - more likely a bug was introduced. My post is just a speculation, let's wait for the actual technical details doc from Google.
- oblio 5y ago
  Well, there are various whitepapers that reformulate this exact truth you've highlighted: issues tend to happen more when changes are made. Regarding the technical details doc, Google will never state that outright in individual postmortems. And they will definitely not draw this to the logical conclusion regarding the spiky yearly activity.
  - yunohn 5y ago
    > logical conclusion regarding the spiky yearly activity Why is it logical? There’s tons of changes being deployed at all times, at all large companies. Across products, verticals, everything - hundreds of changes at any given point. Some of these changes can introduce hard-to-predict bugs in globally distributed systems. Most of the time, external users don’t even notice before they’re fixed. Like another commenter said, performance reviews do not coincide with with the year-end at Google and other companies.
    - oblio 5y ago
      > Like another commenter said, performance reviews do not coincide with with the year-end at Google and other companies. When are they? My bet is that they're roughly around the same time period, even if they don't start or close on December 31st.
    - michaelt 5y ago
      Hypothetically speaking? 11 months a year there's no incentive to cut corners and only do 2 weeks of testing on something that really needs 3. If one month a year rushing things out with a bit less testing is rewarded, I can believe some people would respond to that. Of course, I wouldn't go as far as to call this a "logical conclusion" as the evidence I've seen is very slim.
      - fragmede 5y ago
        Except that there are two a year at Google., so it's 10 months of the year, 5 and 5, and Covid's made everything weird.
  - UncleMeat 5y ago
    It can’t be logical because it is based on bad facts. We wrote evals in August. Anybody racing to get things launched before perf did so months ago. The timing described in the top post is just false. We write the next one in February. We are almost exactly half way between perf cycles. Your hunch that this lines up with the end of the year is false.

moffkalast 5y ago

Then they get fired instead, well done ha.

kroolik 5y ago
Why would you fire an engineer you have just spent millions to train?
- goldcd 5y ago
  I'd guess because Europe-wide outages are costing more than millions
  - kreeben 5y ago
    But now you have someone in your team who will never, ever make that same mistake again and should be your new go-to guy for all X related changes (X being DNS or what-have-you). Firing someone with that type of experience does not lead to success. 100% of all devs make huge mistakes, at least once.
    - KronisLV 5y ago
      > But now you have someone in your team who will never, ever make that same mistake again and that should be your new go-to guy for all DNS related changes. I'm not entirely sure that's always true. For example, i've seen people introduce N+1 issues into a codebase, spend evenings fixing them and refactoring code to fix production issues... just to later introduce those very same types of issues. Sure, you can learn from mistakes, have post-mortems and so on (provided that your org even does those and that anyone listens and cares about the conclusions from those), but to me it feels like the most foolproof way is to ensure that no-one can make these mistakes again, be it with a checklist (which tend to be ignored, honestly), or better yet, an automated CI step or a new test suite. In my eyes, it's basically the same as with unit tests - everyone agrees that you need them, but people rarely write enough of them. So if you introduce something to prevent them from not doing what they should, e.g. a quality gate within a CI step which will disallow a merge once the coverage falls below a set margin, suddenly things are a lot better in the long run.

delroth 5y ago

Perf has been over for almost a month now, and the evaluation period was over more than two months ago.

kroolik 5y ago
Then it might, indeed, be unrelated. Thanks for clarification!
samhw 5y ago
Could it not be knock-on effects of earlier changes? Or indeed post-review changes from scared/precarious engineers?
- mucle6 5y ago
  I think we are exactly mid way between the end year 2021 perf and mid year 2022 perf, so its not clear to me that this could be to blame
  - skinkestek 5y ago
    That explains: The people who did the stuff have either gotten their promotions and moved on or didn't get a promotion and have given up. :-)
dspillett 5y ago
> and the evaluation period was over more than two months ago Excellent time to slack off a bit, make a few mistakes, then come next eval you can point to a marked improvement over the intervening six-to-nine months!
- roenxi 5y ago
  The line of logic this thread has followed so far suggests there will be a reason for a Google outage every month of the year.
  - 5y ago

inglor 5y ago

I would be very interested in any research about code quality with relation to promotion packets. Ideally acamdemic. I am not sure where to look for it

rajin444 5y ago
Is there an objective way to measure code quality? We barely understand the systems we’re writing code for - I don’t see how you could objectively judge the code to manage those systems without fully understanding the systems first. You could have wave some metrics like number of bugs or test coverage, but I can’t think of any (aforementioned included) that aren’t subject to tons of confounding variables.

de6u99er 5y ago

Some people are definitely not gatting a raise.

jankeymeulen 5y ago
More likely someone actually will. A blameless postmortem will be written, and the people that will fix the bug or systems issue will have something to work on with high visibility and high impact, which tends to translate in good performance ratings. (Googler, opinions are my own)
- radishingr 5y ago
  "Impact: fixed processes that led to 8 hr outage" seems like an easy case to make.

trhway 5y ago

November 11 is a public holiday in US. And it seems it has in recent years became a company holiday in many companies too. May be that break in established over many years cadence of workday and holidays plays a role here, like say affecting smoothness of support transition between US-EMEA-APJ.

mikelward 5y ago
Veterans Day is not a day off for US Googlers.

deleted 5y ago

[deleted]

kixiQu 5y ago

Far more likely IMO to coincide with trying to get Exciting New Features out before release freezes for some conference or other.

deanCommie 5y ago

And this is why nobody serious should build on GCP. Google's more interested in placating their primadonna engineers than solving customer problems.

bamboozled 5y ago

There is also a lot of pressure on engineers around black Friday and Cyber Monday to get things done before any change embargoes come into place. This is coming from someone who just worked 16 hours straight to scale things before a change freeze. I don't know how or if this would impact Google, but I'm sure someone there has at least thought about those dates.