10 ms·

Background: https://rachelbythebay.com/w/2021/10/30/5tb/ https://rachelbythebay.com/w/2021/10/30/5tb/ This video was hugely influential on changing the way Goo

by raldi 5y ago

Background: https://rachelbythebay.com/w/2021/10/30/5tb/ https://rachelbythebay.com/w/2021/10/30/5tb/ This video was hugely influential on changing the way Google does internal tools and operations.

justicezyx 5y ago
I worked at TI, Planet and later Borg, I did not feel much influence of this video other a chuckle. Or I might be too low level to perceive.
- jeffbee 5y ago
  I think it was a very common perception among application-level SWEs and SREs that TI, platforms, and Borg did not themselves use the stack enough to perceive its flaws.
quelltext 5y ago
How did things change?
- compiler-guy 5y ago
  The most obvious change that came from this video is that Google abandoned the Borgmon readability requirement. At Google, every change needs approval from someone who has passed a detailed style-guide review process in the given language. Now over multiple changes, it used to require one fairly big one. It's still a pain in the languages that require it--which is all the main ones, but very few of the niche ones. Many other things changed as well. Much of what the video complains about got automated and better documented. But the company has grown so much, and the product lines have diversified so dramatically, that there are still plenty of places to complain about the overhead.
  - dekhn 5y ago
    Context on borgmon: https://sre.google/sre-book/practical-alerting/ https://sre.google/sre-book/practical-alerting/ borgmon was a truly weird system.
    - mikelward 5y ago
      It still is, but it used to be, too.
      - essnine 5y ago
        an unexpected mitch hedberg line appeared from the tall grass!
  - m0zg 5y ago
    I've proudly managed to avoid Borgmon in favor of Monarch. Which was new at the time, but worked all right even back then. I have a lot fewer gray hairs because of that. They should have kept and rigidly enforced the Borgmon readability requirement to force people to migrate off that convoluted, idiosyncratic piece of shit.
  - StillBored 5y ago
    I've never understood places with rigid style guides policed by people. Its idiotic, because we have computers and in places like google presumably a fair number of them know basic parsing/lex sufficiently that if they can't make a tool like clang-format that automatically reformats on save/commit/whatever then they can use a tool like clang-tidy to toss warnings during a development/CI/whatever phase. Putting people in charge of formatting/style is just an excuse for wasting time bikeshedding, either the code is wrong and a tool can tell you, or its not wrong.
    - kccqzy 5y ago
      That's not what readability is. There are plenty of automated tools that will give you results from running lint, ClangTidy and other tools. Readability is mostly about structuring your code well to be easily read. It's about architecting your code within a single file. It's about telling a junior SWE who reinvented the wheel use a library function he/she didn't know about instead.
      - StillBored 5y ago
        So the rules can be codified sufficiently to test people on, but they can't be codified for a computer? The only one that sounds more difficult to codify is telling people of the existence of duplicate functions. But as someone who contributes to the linux kernel, I can tell you right now that the only way that works reliably is to have a very large pool of reviewers. Very experienced engineers frequently miss what people are doing in other parts of the source base, the name might not be what they expect, etc, etc, etc. In the case of linux there are a fair number of duplicates, or similar functions, and people write coccinelle patches to replace them on a fairly regular basis after they have been in the kernel for years. So, I doubt giving someone a formal gatekeeper flag, really helps vs just having wider change review.
        kccqzy 5y ago
        > Very experienced engineers frequently miss what people are doing in other parts of the source base, the name might not be what they expect, etc, etc Very true. Readability can't help with that, nor is it designed to. It's mostly there to help novices and new hires. Experienced engineers already have readability themselves so they don't need this extra review.
- mathteddybear 5y ago
  Broadly speaking, there are tools to automate this or that, some technologies are getting deprecated and replaced by new ones Also probably the privacy review could be a bigger bottleneck these days ;-)
- scottlamb 5y ago
  I think you can read about some of these changes in Google's SRE and SWE books (even if they don't mention this video in particular), at least the ones most likely to be interesting to someone outside Google. But dropping Borgmon readability was the most immediate and obvious. It was basically true that no one had Borgmon readability. The policy was a catch-22: you couldn't get readability for the simple/formulaic Borgmon macro invocations that were encouraged and often sufficient. You could only get it for doing something "clever". I got it by writing fancy borgmon rules to paper over a problem that (in hindsight) I should have solved elsewhere. Another was easing quota management. IMHO the most unbelievable thing in the video was that after Broccoli Man told Panda Woman to get quota in two cells, she just said "done". Besides the hassle in transcribing what you needed into the request system [1], various types of quota were chronically unavailable where you needed them, even in tiny amounts. In 2010, I kept a critical infrastructure service running by regularly IMing major clients' on-calls asking them to donate 0.1 cpu(!) of their quota in some cell or another when I didn't have quite enough to grow. There was a "gray market" mailing list where people would trade resources they couldn't get through the primary system. But eventually, they built a system that for small services would make the quota just happen for you. Overall, it was a kick in the pants for the most basic infrastructure teams that made them see how unnecessarily hard this is for their internal customers, prompting them to make small things just happen while keeping large things possible. In any large organization, it's healthy to get this kind of feedback regularly. The actual specific changes and technologies are pretty specific to Google in 2010... [1] Many people managed this very very tediously with spreadsheets. I eventually wrote a tool to generate the requests based on comparing your intended production config with your current quota.
  - jeffbee 5y ago
    Production priority quota horse trading in the days before it was easy was a real skill. But non-production quota was free and virtually infinite, even in those days.
    - the-rc 5y ago
      Or converting between priority levels or users that were under your product. If you ran a large enough service, there was also begging your users for spare quota or warning them that their recent donation couldn't be fully deployed because a necessary change in some secondary borgcfg template had eaten into the service's resources, so it would help everybody if they knew anyone with a few cores laying around... By the way, do you happen to be the jwb that filed a... creative BT quota request?
- vechagup 5y ago
  There's been a big investment in server platforms that strive to enable SWEs to build a new service that follows Best Practices with as little knowledge and handholding as possible. These consist of conformance tests that yell at you while you're coding if you are trying something generally thought to be bad, and semi-automated workflows that help you bring your code to production. When everything works as intended, the production workflows set up a decent set of alerts, acquire resources, configure CI/CD pipelines, and launch your jobs with just a few button presses on your part. (In practice, one of the steps will probably require debugging, but eh, it seems way better than the broccoli man video.)
jrockway 5y ago
I feel like people forgot after about five years. I remember wasting a week filling out various pieces of paperwork and submitting byzantine configuration CLs so that some contractor would have permission to view a certain webpage through the corporate proxy. (I think what made me most mad is that regular employees could view the website with no additional configuration. I can understand if I was filling out tickets to get approval, or a security review, but the actual configuration of the proxy had to change to allow this, in addition to getting all of those approvals!) My team didn't make the website, and the contractor didn't work on my team, so I'm honestly not sure why I was involved. I just remember being annoyed about it. I'm sure there are some memes about it in the archive.
- ikiris 5y ago
  Contractor access is its own hellscape.
  - q3k 5y ago
    ... socially, too. That part sucked.
    - abustamam 5y ago
      Separation of the classes and all
      - ak217 5y ago
        That's the scary, and perhaps most immoral, part about Google. More so than other megacorps, it relies on a shadow workforce of contractors to do core parts of its work. It's quite the caste system, and most Google FTEs are oblivious or choose to ignore it. I know a Google contractor who was denied access to a lactation room at her office (she's the type of contractor who works full-time alongside regular employees). When she tried to fix that, she got sent into an infinite loop of support tickets being bounced around between different departments claiming it was the other one's responsibility. Literally no one was able to fix that for her. Her manager was on a different continent, and wasn't able to help either.
inoffensivename 5y ago
It was hugely influential in identifying the frustration of getting things done at Google. In my experience it's even more true now than it was back then, the number of things you have to deal with has just grown. I've been at Google since 2006 and I feel like I'm losing my mind with all the complexity.
- jez 5y ago
  Out of genuine curiosity, what keeps you at Google for 15 years despite perceived increase in complexity to getting things done? I'm wondering whether the answer is like "there's a lot of complexity, but I like the work I do more" or "I like the people more" or some other reason.
  - Dangeranger 5y ago
    I think they call them “golden handcuffs”.
    - marcyb5st 5y ago
      Googler here. I think the technical term is Golden cage :)
    - oblio 5y ago
      Heh, that's <<if>> they want to leave. If they don't, they call it "comfy job with awesome paycheck and not a lot of pressure" :-p
      - dTal 5y ago
        (this comment now obsolete)
        oblio 5y ago
        There, I fixed it! https://cheezburger.com/5821507840/his-and-hers-shower-head https://cheezburger.com/5821507840/his-and-hers-shower-head
sicromoft 5y ago
See also the recent discussion here of "I Don’t Know How To Count That Low": https://news.ycombinator.com/item?id=28988281 https://news.ycombinator.com/item?id=28988281
throwawayamz27 5y ago
This is not true, Google is no longer capable of changing anything internally, it's like a government in that respect, it can make a new process/program (that might on the surface make some things look better) but underneath the old system is still there.
- kevin_thibedeau 5y ago
  Well they can launch new semi-independent initiatives under the Alphabet umbrella. They just need to use that to create a web search engine. Something sorely lacking in their current portfolio.

by raldi 5y ago

Background: https://rachelbythebay.com/w/2021/10/30/5tb/ https://rachelbythebay.com/w/2021/10/30/5tb/ This video was hugely influential on changing the way Google does internal tools and operations.

justicezyx 5y ago

I worked at TI, Planet and later Borg, I did not feel much influence of this video other a chuckle. Or I might be too low level to perceive.

jeffbee 5y ago
I think it was a very common perception among application-level SWEs and SREs that TI, platforms, and Borg did not themselves use the stack enough to perceive its flaws.

quelltext 5y ago

How did things change?

compiler-guy 5y ago
The most obvious change that came from this video is that Google abandoned the Borgmon readability requirement. At Google, every change needs approval from someone who has passed a detailed style-guide review process in the given language. Now over multiple changes, it used to require one fairly big one. It's still a pain in the languages that require it--which is all the main ones, but very few of the niche ones. Many other things changed as well. Much of what the video complains about got automated and better documented. But the company has grown so much, and the product lines have diversified so dramatically, that there are still plenty of places to complain about the overhead.
- dekhn 5y ago
  Context on borgmon: https://sre.google/sre-book/practical-alerting/ https://sre.google/sre-book/practical-alerting/ borgmon was a truly weird system.
  - mikelward 5y ago
    It still is, but it used to be, too.
    - essnine 5y ago
      an unexpected mitch hedberg line appeared from the tall grass!
- m0zg 5y ago
  I've proudly managed to avoid Borgmon in favor of Monarch. Which was new at the time, but worked all right even back then. I have a lot fewer gray hairs because of that. They should have kept and rigidly enforced the Borgmon readability requirement to force people to migrate off that convoluted, idiosyncratic piece of shit.
- StillBored 5y ago
  I've never understood places with rigid style guides policed by people. Its idiotic, because we have computers and in places like google presumably a fair number of them know basic parsing/lex sufficiently that if they can't make a tool like clang-format that automatically reformats on save/commit/whatever then they can use a tool like clang-tidy to toss warnings during a development/CI/whatever phase. Putting people in charge of formatting/style is just an excuse for wasting time bikeshedding, either the code is wrong and a tool can tell you, or its not wrong.
  - kccqzy 5y ago
    That's not what readability is. There are plenty of automated tools that will give you results from running lint, ClangTidy and other tools. Readability is mostly about structuring your code well to be easily read. It's about architecting your code within a single file. It's about telling a junior SWE who reinvented the wheel use a library function he/she didn't know about instead.
    - StillBored 5y ago
      So the rules can be codified sufficiently to test people on, but they can't be codified for a computer? The only one that sounds more difficult to codify is telling people of the existence of duplicate functions. But as someone who contributes to the linux kernel, I can tell you right now that the only way that works reliably is to have a very large pool of reviewers. Very experienced engineers frequently miss what people are doing in other parts of the source base, the name might not be what they expect, etc, etc, etc. In the case of linux there are a fair number of duplicates, or similar functions, and people write coccinelle patches to replace them on a fairly regular basis after they have been in the kernel for years. So, I doubt giving someone a formal gatekeeper flag, really helps vs just having wider change review.
      - kccqzy 5y ago
        > Very experienced engineers frequently miss what people are doing in other parts of the source base, the name might not be what they expect, etc, etc Very true. Readability can't help with that, nor is it designed to. It's mostly there to help novices and new hires. Experienced engineers already have readability themselves so they don't need this extra review.
mathteddybear 5y ago
Broadly speaking, there are tools to automate this or that, some technologies are getting deprecated and replaced by new ones Also probably the privacy review could be a bigger bottleneck these days ;-)
scottlamb 5y ago
I think you can read about some of these changes in Google's SRE and SWE books (even if they don't mention this video in particular), at least the ones most likely to be interesting to someone outside Google. But dropping Borgmon readability was the most immediate and obvious. It was basically true that no one had Borgmon readability. The policy was a catch-22: you couldn't get readability for the simple/formulaic Borgmon macro invocations that were encouraged and often sufficient. You could only get it for doing something "clever". I got it by writing fancy borgmon rules to paper over a problem that (in hindsight) I should have solved elsewhere. Another was easing quota management. IMHO the most unbelievable thing in the video was that after Broccoli Man told Panda Woman to get quota in two cells, she just said "done". Besides the hassle in transcribing what you needed into the request system [1], various types of quota were chronically unavailable where you needed them, even in tiny amounts. In 2010, I kept a critical infrastructure service running by regularly IMing major clients' on-calls asking them to donate 0.1 cpu(!) of their quota in some cell or another when I didn't have quite enough to grow. There was a "gray market" mailing list where people would trade resources they couldn't get through the primary system. But eventually, they built a system that for small services would make the quota just happen for you. Overall, it was a kick in the pants for the most basic infrastructure teams that made them see how unnecessarily hard this is for their internal customers, prompting them to make small things just happen while keeping large things possible. In any large organization, it's healthy to get this kind of feedback regularly. The actual specific changes and technologies are pretty specific to Google in 2010... [1] Many people managed this very very tediously with spreadsheets. I eventually wrote a tool to generate the requests based on comparing your intended production config with your current quota.
- jeffbee 5y ago
  Production priority quota horse trading in the days before it was easy was a real skill. But non-production quota was free and virtually infinite, even in those days.
  - the-rc 5y ago
    Or converting between priority levels or users that were under your product. If you ran a large enough service, there was also begging your users for spare quota or warning them that their recent donation couldn't be fully deployed because a necessary change in some secondary borgcfg template had eaten into the service's resources, so it would help everybody if they knew anyone with a few cores laying around... By the way, do you happen to be the jwb that filed a... creative BT quota request?
vechagup 5y ago
There's been a big investment in server platforms that strive to enable SWEs to build a new service that follows Best Practices with as little knowledge and handholding as possible. These consist of conformance tests that yell at you while you're coding if you are trying something generally thought to be bad, and semi-automated workflows that help you bring your code to production. When everything works as intended, the production workflows set up a decent set of alerts, acquire resources, configure CI/CD pipelines, and launch your jobs with just a few button presses on your part. (In practice, one of the steps will probably require debugging, but eh, it seems way better than the broccoli man video.)

jrockway 5y ago

I feel like people forgot after about five years. I remember wasting a week filling out various pieces of paperwork and submitting byzantine configuration CLs so that some contractor would have permission to view a certain webpage through the corporate proxy. (I think what made me most mad is that regular employees could view the website with no additional configuration. I can understand if I was filling out tickets to get approval, or a security review, but the actual configuration of the proxy had to change to allow this, in addition to getting all of those approvals!) My team didn't make the website, and the contractor didn't work on my team, so I'm honestly not sure why I was involved. I just remember being annoyed about it. I'm sure there are some memes about it in the archive.

ikiris 5y ago
Contractor access is its own hellscape.
- q3k 5y ago
  ... socially, too. That part sucked.
  - abustamam 5y ago
    Separation of the classes and all
    - ak217 5y ago
      That's the scary, and perhaps most immoral, part about Google. More so than other megacorps, it relies on a shadow workforce of contractors to do core parts of its work. It's quite the caste system, and most Google FTEs are oblivious or choose to ignore it. I know a Google contractor who was denied access to a lactation room at her office (she's the type of contractor who works full-time alongside regular employees). When she tried to fix that, she got sent into an infinite loop of support tickets being bounced around between different departments claiming it was the other one's responsibility. Literally no one was able to fix that for her. Her manager was on a different continent, and wasn't able to help either.

inoffensivename 5y ago

It was hugely influential in identifying the frustration of getting things done at Google. In my experience it's even more true now than it was back then, the number of things you have to deal with has just grown. I've been at Google since 2006 and I feel like I'm losing my mind with all the complexity.

jez 5y ago
Out of genuine curiosity, what keeps you at Google for 15 years despite perceived increase in complexity to getting things done? I'm wondering whether the answer is like "there's a lot of complexity, but I like the work I do more" or "I like the people more" or some other reason.
- Dangeranger 5y ago
  I think they call them “golden handcuffs”.
  - marcyb5st 5y ago
    Googler here. I think the technical term is Golden cage :)
  - oblio 5y ago
    Heh, that's <<if>> they want to leave. If they don't, they call it "comfy job with awesome paycheck and not a lot of pressure" :-p
    - dTal 5y ago
      (this comment now obsolete)
      - oblio 5y ago
        There, I fixed it! https://cheezburger.com/5821507840/his-and-hers-shower-head https://cheezburger.com/5821507840/his-and-hers-shower-head

sicromoft 5y ago

See also the recent discussion here of "I Don’t Know How To Count That Low": https://news.ycombinator.com/item?id=28988281 https://news.ycombinator.com/item?id=28988281

throwawayamz27 5y ago

This is not true, Google is no longer capable of changing anything internally, it's like a government in that respect, it can make a new process/program (that might on the surface make some things look better) but underneath the old system is still there.

kevin_thibedeau 5y ago
Well they can launch new semi-independent initiatives under the Alphabet umbrella. They just need to use that to create a web search engine. Something sorely lacking in their current portfolio.