5 ms·

Where does "these are really good points, but why don't we have tooling which sets everything up automatically?" fit on the scale?

raldi 5y ago
Or: These are really good points for a visibly-user-facing post-alpha service, but isn't it a bit overengineered for an experimental internal service whose clients can tolerate the risk of occasional downtime?
- nostrademons 5y ago
  L5 Xoogler who left for a startup.
nostrademons 5y ago
L9+
- packetslave 5y ago
  T7-T9 Vision
  - rahimiali 5y ago
    Is there a page that documents this anecdote? I’d like to link to it next time i use this phrase. Ironically, googling for it doesn’t turn up anything relevant.
    - tommiegannert 5y ago
      Google has officially apologized. The person in question had to take their blog offline due to bad behavior from readers (unrelated to Google AFAIK). Overall, this is a dark chapter. It's also been scrubbed internally. Not obliterated, but you won't accidentally bump into it as a Noogler. As much as I think people should take responsibility for their own actions, it's probably for the better to let this one rest now. Who caused it is irrelevant at this point, though. We (Xooglers, and Googlers) can take responsibility for our actions, and not continue perpetuating it.
      - rahimiali 5y ago
        I use the phrase to describe how leveling is not about what junior people think it’s about. I’m not clear about the responsibility you’re talking about.
SilasX 5y ago
Yeah, that was my reaction. I get the need for all this reliability/failover, but it's horrible failure of abstraction/separation of concerns. There's no reason the serving team should have to learn how to do all of those things on the checklist, since it can be done by anyone who's already learned the infra. You're expecting them to learn all kinds of stuff outside of their specialty, when they should be able to kick the app over the wall and let infra ensure that the app is deployed in two separate PCR zones with the failover plan etc, which should itself be mostly automated.
- q3k 5y ago
  > when they should be able to kick the app over the wall and let infra ensure that the app is deployed in two separate PCR zones with the failover plan etc, which should itself be mostly automated Not entirely - the developers should actively participate in designing the actual failover scenario and making sure the application can handle that (anything from being okay with some downtime due to the failover happening to designing an actual multi-region multi-master application). Making assumptions like 'infra will handle it' is a great way to not only get unexpected outages (because the developers assumed there would be no downtime because failover is magic, or that writes will never be lost) but to also introduce tensions between teams (because you now have an outside team having to wrangle an application into reliability when the original authors don't give a crap about it). I get and agree with your point, the tooling and processes should definitely be simplified/automated when possible, and developers deserve a working platform that just works. The whole point of a platform team is to abstract away the mundane to let people do their job. But reliability is everyone's job, not just the infra's team, and developers must understand the tradeoffs and technology involved in order to not design broken systems.
  - SilasX 5y ago
    If that's the point: A) It's doing a horrible job conveying it. A dev does need to be concerned on how to handle failover, but only at a certain abstraction level. They should be required to specify something in the form "given server A fails and has to pass to B, what do you do?" That does not require you to know the terminology about PCRs and how to make decisions about which cells (or whatever) to pick on deployment, or avoiding the "gotcha" about making sure the two servers are in different PCR zones. At that point, it's just following a checklist that needs no knowledge of the specifics of the app, and, to the extent that it's accurately representing how Google was, is indicative of bad processes. B) Many things should be infra's job, as they're cleanly orthogonal to what dev's are doing. For example, how to apply a security patch to a DB. That's unrelated to the operation of the app. I do get your point though, and I wouldn't say something like this about e.g. testing (which was the short, "reasonable" part of the video!) -- the devs have intimate knowledge of what counts as passing and failing and should be writing tests, and not 100% passing it over to QA. But that's precisely because such concerns are deeply tied in to the thing they are concerned with. "SQL 3.4.1 vs 3.4.2" is not.
nuerow 5y ago
> Where does "these are really good points, but why don't we have tooling which sets everything up automatically?" fit on the scale? My guess it fits nowhere because the L5s don't have the ability to automate it, and the L6s think it's trivial and as it's done sparingly then it doesn't justify the work to do things differently. And this is why we can't have nice things.
- ikiris 5y ago
  More like you aren't going to get promoted for automating someone else's toil. Also, now who's going to support it, better deprecate it since the library changed / got deprecated / it's tuesday.
  - Jensson 5y ago
    > More like you aren't going to get promoted for automating someone else's toil. Lots of people were promoted for automating these things. They built easy to use services, got extra headcount since they became important and climbed the ranks. So not sure why you'd think that. It may be different at other companies, but at Google building stuff that many other engineers depends on is a major way to get promoted. Of course if you automate something and nobody uses your automation tooling then you wont get promoted, but if your work gets used by basically every new engineer you'll climb the ranks quickly.
- azornathogron 5y ago
  And yet it's been a decade since this video and practically everything it mentions is a non-problem now. No one is spinning up new borgmon instances. Spanner is replicated by default. Only very low level services need to care about PCRs. If you use one of the approved frameworks it will set up practically all the production configuration for you. Basic alerting for your service is automated, just turn it on, picking cells to run in is automated, scaling your service is automated, etc. Actually getting quota remains a problem... :-p Anyway I would argue we can and do have nice things, and that has happened precisely through the efforts of a huge number of people at all levels. Edit to add: of course, there are always new problems to complain about! It's the march of progress after all.
omreaderhn 5y ago
That would be 'Xoogler' because Google's engineering and broader corporate culture does not reward work like that and so when you realize that, you leave. In general, Googlers have very little idea how far behind the rest of the industry they are when it comes to tooling. I am a Xoogler.
- mwcampbell 5y ago
  I got the impression, based on a blog post by Eric Lawrence [1], that Google's developer tooling was top-notch (except for devs working on open-source projects like Chromium). Did it get worse since 2017, or are you talking about a different kind of tooling? [1]; https://textslashplain.com/2017/02/01/google-chrome-one-year-in/ https://textslashplain.com/2017/02/01/google-chrome-one-year...
  - throwawayfgg 5y ago
    Google's developer tooling is top-notch and amazing and constantly improving.
GeneralMayhem 5y ago
I think that was more or less the intended response. And ten years on, most of these things are automated. This video was a kick in the pants internally.
cmrdporcupine 5y ago
imho the Google interview process selects for people who thrive on organizational challenges.

raldi 5y ago

Or: These are really good points for a visibly-user-facing post-alpha service, but isn't it a bit overengineered for an experimental internal service whose clients can tolerate the risk of occasional downtime?

nostrademons 5y ago
L5 Xoogler who left for a startup.

nostrademons 5y ago

L9+

packetslave 5y ago
T7-T9 Vision
- rahimiali 5y ago
  Is there a page that documents this anecdote? I’d like to link to it next time i use this phrase. Ironically, googling for it doesn’t turn up anything relevant.
  - tommiegannert 5y ago
    Google has officially apologized. The person in question had to take their blog offline due to bad behavior from readers (unrelated to Google AFAIK). Overall, this is a dark chapter. It's also been scrubbed internally. Not obliterated, but you won't accidentally bump into it as a Noogler. As much as I think people should take responsibility for their own actions, it's probably for the better to let this one rest now. Who caused it is irrelevant at this point, though. We (Xooglers, and Googlers) can take responsibility for our actions, and not continue perpetuating it.
    - rahimiali 5y ago
      I use the phrase to describe how leveling is not about what junior people think it’s about. I’m not clear about the responsibility you’re talking about.

SilasX 5y ago

Yeah, that was my reaction. I get the need for all this reliability/failover, but it's horrible failure of abstraction/separation of concerns. There's no reason the serving team should have to learn how to do all of those things on the checklist, since it can be done by anyone who's already learned the infra. You're expecting them to learn all kinds of stuff outside of their specialty, when they should be able to kick the app over the wall and let infra ensure that the app is deployed in two separate PCR zones with the failover plan etc, which should itself be mostly automated.

q3k 5y ago
> when they should be able to kick the app over the wall and let infra ensure that the app is deployed in two separate PCR zones with the failover plan etc, which should itself be mostly automated Not entirely - the developers should actively participate in designing the actual failover scenario and making sure the application can handle that (anything from being okay with some downtime due to the failover happening to designing an actual multi-region multi-master application). Making assumptions like 'infra will handle it' is a great way to not only get unexpected outages (because the developers assumed there would be no downtime because failover is magic, or that writes will never be lost) but to also introduce tensions between teams (because you now have an outside team having to wrangle an application into reliability when the original authors don't give a crap about it). I get and agree with your point, the tooling and processes should definitely be simplified/automated when possible, and developers deserve a working platform that just works. The whole point of a platform team is to abstract away the mundane to let people do their job. But reliability is everyone's job, not just the infra's team, and developers must understand the tradeoffs and technology involved in order to not design broken systems.
- SilasX 5y ago
  If that's the point: A) It's doing a horrible job conveying it. A dev does need to be concerned on how to handle failover, but only at a certain abstraction level. They should be required to specify something in the form "given server A fails and has to pass to B, what do you do?" That does not require you to know the terminology about PCRs and how to make decisions about which cells (or whatever) to pick on deployment, or avoiding the "gotcha" about making sure the two servers are in different PCR zones. At that point, it's just following a checklist that needs no knowledge of the specifics of the app, and, to the extent that it's accurately representing how Google was, is indicative of bad processes. B) Many things should be infra's job, as they're cleanly orthogonal to what dev's are doing. For example, how to apply a security patch to a DB. That's unrelated to the operation of the app. I do get your point though, and I wouldn't say something like this about e.g. testing (which was the short, "reasonable" part of the video!) -- the devs have intimate knowledge of what counts as passing and failing and should be writing tests, and not 100% passing it over to QA. But that's precisely because such concerns are deeply tied in to the thing they are concerned with. "SQL 3.4.1 vs 3.4.2" is not.

nuerow 5y ago

> Where does "these are really good points, but why don't we have tooling which sets everything up automatically?" fit on the scale? My guess it fits nowhere because the L5s don't have the ability to automate it, and the L6s think it's trivial and as it's done sparingly then it doesn't justify the work to do things differently. And this is why we can't have nice things.

ikiris 5y ago
More like you aren't going to get promoted for automating someone else's toil. Also, now who's going to support it, better deprecate it since the library changed / got deprecated / it's tuesday.
- Jensson 5y ago
  > More like you aren't going to get promoted for automating someone else's toil. Lots of people were promoted for automating these things. They built easy to use services, got extra headcount since they became important and climbed the ranks. So not sure why you'd think that. It may be different at other companies, but at Google building stuff that many other engineers depends on is a major way to get promoted. Of course if you automate something and nobody uses your automation tooling then you wont get promoted, but if your work gets used by basically every new engineer you'll climb the ranks quickly.
azornathogron 5y ago
And yet it's been a decade since this video and practically everything it mentions is a non-problem now. No one is spinning up new borgmon instances. Spanner is replicated by default. Only very low level services need to care about PCRs. If you use one of the approved frameworks it will set up practically all the production configuration for you. Basic alerting for your service is automated, just turn it on, picking cells to run in is automated, scaling your service is automated, etc. Actually getting quota remains a problem... :-p Anyway I would argue we can and do have nice things, and that has happened precisely through the efforts of a huge number of people at all levels. Edit to add: of course, there are always new problems to complain about! It's the march of progress after all.

omreaderhn 5y ago

That would be 'Xoogler' because Google's engineering and broader corporate culture does not reward work like that and so when you realize that, you leave. In general, Googlers have very little idea how far behind the rest of the industry they are when it comes to tooling. I am a Xoogler.

mwcampbell 5y ago
I got the impression, based on a blog post by Eric Lawrence [1], that Google's developer tooling was top-notch (except for devs working on open-source projects like Chromium). Did it get worse since 2017, or are you talking about a different kind of tooling? [1]; https://textslashplain.com/2017/02/01/google-chrome-one-year-in/ https://textslashplain.com/2017/02/01/google-chrome-one-year...
- throwawayfgg 5y ago
  Google's developer tooling is top-notch and amazing and constantly improving.

GeneralMayhem 5y ago

I think that was more or less the intended response. And ten years on, most of these things are automated. This video was a kick in the pants internally.

cmrdporcupine 5y ago

imho the Google interview process selects for people who thrive on organizational challenges.