4 ms·

I’m always surprised to see the totality of support for this workflow. My biggest gripes were: - Owners - Readability review (or really the 18month queue to g

I’m always surprised to see the totality of support for this workflow. My biggest gripes were: - Owners - Readability review (or really the 18month queue to get Java/python readability) Although seemingly innocuous, this made maintaining internal libraries very challenging. There was no way to update every call site of your library in an efficient way. Tools like Rosie were added on top so that you could shard your PRs and shepherd many PRs through a Byzantine approval process. Compared to Facebook, I found it much harder to reach the same levels of productivity given Owners and readability review. I don’t think libraries like React could get developed at Google given how hard it would be to evolve the API surface.

jeffbee 3y ago
Google3 is a huge codebase of high quality that moves exceptionally quickly, so it just seems like your priors are getting in the way. Complaining that a large-scale change among tens of thousands of software developers requires a tool (Rosie) is sort of the same thing as all those people who complain that it's too hard to cope with having millions of machines in prod, i.e. the type of people who wash out of the company in a few months and then go on to a career of complaining about it online.
- marcinzm 3y ago
  The same applies to Facebook which is their comparison and not some random 10 person startup.
  - jeffbee 3y ago
    Except Facebook's codebase is an order of magnitude smaller.
    - tallowen 3y ago
      What makes you say that Facebook's codebase is an order of magnitude smaller?
      - dilyevsky 3y ago
        Fb had 100 million loc in 2019[0] while google already had 2 billion[1] [0] -https://www.wired.com/story/facebook-zoncolan-static-analysis-tool/ https://www.wired.com/story/facebook-zoncolan-static-analysi... [1] - https://www.wired.com/2015/09/google-2-billion-lines-codeand-one-place/ https://www.wired.com/2015/09/google-2-billion-lines-codeand...
        ot 3y ago
        That's an apples-and-oranges comparisons, 100M is just the size of the frontend/middleware code (Hack), while 2B includes everything, including configuration and generated code.
ASinclair 3y ago
> I don’t think libraries like React could get developed at Google given how hard it would be to evolve the API surface. That's more of a feature and not a bug to me. Users of a library would appreciate it if the API surface doesn't change dramatically over time.
- compiler-guy 3y ago
  That's the strange thing about the GP comment: Teams can and do change their API surface dramatically over time. They use the LSC process that the GP doesn't like to do it, and it is 99% invisible to the API users. It works, if not exactly flawlessly every time, very consistently well, and there are dozens of such changes in flight all the time that API users barely even realize exist. Don't know what the alternative would be.
- tallowen 3y ago
  I don't think the burden of the changing API surface was place on the callers of that API. With core library developers updating their callsites, it meant that people calling into early versions of React, never had to deal with upgrading the react version in their package. As a caller of these libraries, it was much nicer to have the React team take care of upgrades where possible rather than having to deal with it myself. Furthermore, there were tons of changes that could happen across the board at facebook that didn't happen at google due to owners files. Lots of people were empowered to make broad changes to the entire codebase which in aggregate made the codebase better. Coming from Facebook, all the systems around owners and readability represented a huge amount of time to be able to make changes that should have been easy. It also left the codebase with many warts that would have been relatively quick to fix locally but a pain to land. Facebook went in the other direction - there were better and better tools for making broad updates
- CalChris 3y ago
  Dramatically, no. But then a couple of releases of deprecated and then hasta la vista is fine with me if it means progress.
tantalor 3y ago
Owners and readability are constraints of the version control system, not the code review system. Now, finding somebody with authority to approve is a constraint of code review. But that would be true of any code review tool. So it's not really fair to complain about.
cmrdporcupine 3y ago
OWNERS always made sense to me, but the readability stuff at Google I found just... pointless productivity destroyer. I can't understand why they've kept it. In fact they made it far far worse with the incremental process. With the crazy interview process, the copious static analysis and formatting tools and style guides + style enforcement tools, and with OWNERS and team members reviewing things... what is even the point? What are they actually checking for? Back when I started there, I got my C++ readability quite easily with just a month wait or so. Then I briefly worked with a bit of Go, and encountered my first taste of the incremental process, and it.. sucked. When they rolled out incremental readability, I never bothered with any other languages. Luckily, later, I ended up working in the Chromium tree which required no such readability status.
joatmon-snoo 3y ago
As someone who left in 2021: - OWNERS is absolutely a necessary and important thing, and yes it sucked when it made finding an approver hard, but the point of OWNERS was to optimize for _local_ ownership. (For non-Google folks: think CODEOWNERS files, but hierarchical/recursive, so approvers in /OWNERS, a/OWNERS, a/b/OWNERS, and a/b/c/OWNERS can approve changes anywhere in a/b/c/...) - I joined in 2017 and it never took 18 months to get readability. If it took you 18 months, it was because you spent 2 weeks writing code in one of those languages and then didn't keep it up. I worked through the stats on this too- by the time I left in 2021, there was virtually no delay to enter the readability process for Java and it was maybe a few weeks for Python. > There was no way to update every call site of your library in an efficient way. Does FB have something better here? G had sufficiently many tools for this, IMO: csearch+grep+sed for the easy changes, Refaster/clang-tidy for more complex stuff (admittedly C++ AST matchers are black magic that few people on even the C++ teams understood). Although I do wish I had known about Comby before I'd left. Rosie - the tool for executing large-scale changes, i.e. if you made a change to 1K+ files (often 10K+ or 100K+), you needed a way to break the change up into multiple PRs - was also an absolutely critical part of this. I never found the approval process Byzantine - undocumented, perhaps, but remarkably streamlined considering that LSCs meant making simultaneous changes to code in Ads, Android, Cloud, Geo, Search, Technical Infrastructure, YouTube, or however Google breaks down the engineering these days.
- zeroonetwothree 3y ago
  At Facebook there is no OWNERS so if you make an API change you can just update all the code yourself at once. It makes it much easier to rapidly iterate and improve shared libraries.
  - colonCapitalDee 3y ago
    Yes, and it also makes it much easier for random people to break your code. It's a trade-off
    - bluGill 3y ago
summerlight 3y ago
The owner system is kind of a necessary evil to enforce reviews from the domain experts. I've seen lots of junior engineers who just want to submit their code and move on and I appreciate their productivity, but there have been so many cases where a seemingly innocuous change eventually led to a million dollars incident and only those domain experts could've detected it. And I've still gotten lots of pagers exactly due to this reason even with this enforcement thanks to someone able to find a relatively lenient reviewer on the codebase with broad ownership. > There was no way to update every call site of your library in an efficient way. There are global approvers. It's not super easy to get their approval (you need to go through the large scale change process), but getting each owner's approval will be exempted if you can get the one.
joshuamorton 3y ago
> Although seemingly innocuous, this made maintaining internal libraries very challenging. There was no way to update every call site of your library in an efficient way. Tools like Rosie were added on top so that you could shard your PRs and shepherd many PRs through a Byzantine approval process. Ownership is the easiest issue to solve here. Beyond a few dozen clients, you're going to run into the inability to run tests fast enough, and beyond a few hundred clients, you won't be able to sync fast enough, even if you skip all presubmits (there will, in expectation, always be a merge conflict in some file). 3-stage migrations/rosie are necessary at that point anyway. Owners is only the problem at like, 5-10 clients, and when I made changes at that scale, messaging owners was usually fast and effective.

jeffbee 3y ago

Google3 is a huge codebase of high quality that moves exceptionally quickly, so it just seems like your priors are getting in the way. Complaining that a large-scale change among tens of thousands of software developers requires a tool (Rosie) is sort of the same thing as all those people who complain that it's too hard to cope with having millions of machines in prod, i.e. the type of people who wash out of the company in a few months and then go on to a career of complaining about it online.

marcinzm 3y ago
The same applies to Facebook which is their comparison and not some random 10 person startup.
- jeffbee 3y ago
  Except Facebook's codebase is an order of magnitude smaller.
  - tallowen 3y ago
    What makes you say that Facebook's codebase is an order of magnitude smaller?
    - dilyevsky 3y ago
      Fb had 100 million loc in 2019[0] while google already had 2 billion[1] [0] -https://www.wired.com/story/facebook-zoncolan-static-analysis-tool/ https://www.wired.com/story/facebook-zoncolan-static-analysi... [1] - https://www.wired.com/2015/09/google-2-billion-lines-codeand-one-place/ https://www.wired.com/2015/09/google-2-billion-lines-codeand...
      - ot 3y ago
        That's an apples-and-oranges comparisons, 100M is just the size of the frontend/middleware code (Hack), while 2B includes everything, including configuration and generated code.

ASinclair 3y ago

> I don’t think libraries like React could get developed at Google given how hard it would be to evolve the API surface. That's more of a feature and not a bug to me. Users of a library would appreciate it if the API surface doesn't change dramatically over time.

compiler-guy 3y ago
That's the strange thing about the GP comment: Teams can and do change their API surface dramatically over time. They use the LSC process that the GP doesn't like to do it, and it is 99% invisible to the API users. It works, if not exactly flawlessly every time, very consistently well, and there are dozens of such changes in flight all the time that API users barely even realize exist. Don't know what the alternative would be.
tallowen 3y ago
I don't think the burden of the changing API surface was place on the callers of that API. With core library developers updating their callsites, it meant that people calling into early versions of React, never had to deal with upgrading the react version in their package. As a caller of these libraries, it was much nicer to have the React team take care of upgrades where possible rather than having to deal with it myself. Furthermore, there were tons of changes that could happen across the board at facebook that didn't happen at google due to owners files. Lots of people were empowered to make broad changes to the entire codebase which in aggregate made the codebase better. Coming from Facebook, all the systems around owners and readability represented a huge amount of time to be able to make changes that should have been easy. It also left the codebase with many warts that would have been relatively quick to fix locally but a pain to land. Facebook went in the other direction - there were better and better tools for making broad updates
CalChris 3y ago
Dramatically, no. But then a couple of releases of deprecated and then hasta la vista is fine with me if it means progress.

tantalor 3y ago

Owners and readability are constraints of the version control system, not the code review system. Now, finding somebody with authority to approve is a constraint of code review. But that would be true of any code review tool. So it's not really fair to complain about.

cmrdporcupine 3y ago

OWNERS always made sense to me, but the readability stuff at Google I found just... pointless productivity destroyer. I can't understand why they've kept it. In fact they made it far far worse with the incremental process. With the crazy interview process, the copious static analysis and formatting tools and style guides + style enforcement tools, and with OWNERS and team members reviewing things... what is even the point? What are they actually checking for? Back when I started there, I got my C++ readability quite easily with just a month wait or so. Then I briefly worked with a bit of Go, and encountered my first taste of the incremental process, and it.. sucked. When they rolled out incremental readability, I never bothered with any other languages. Luckily, later, I ended up working in the Chromium tree which required no such readability status.

joatmon-snoo 3y ago

As someone who left in 2021: - OWNERS is absolutely a necessary and important thing, and yes it sucked when it made finding an approver hard, but the point of OWNERS was to optimize for _local_ ownership. (For non-Google folks: think CODEOWNERS files, but hierarchical/recursive, so approvers in /OWNERS, a/OWNERS, a/b/OWNERS, and a/b/c/OWNERS can approve changes anywhere in a/b/c/...) - I joined in 2017 and it never took 18 months to get readability. If it took you 18 months, it was because you spent 2 weeks writing code in one of those languages and then didn't keep it up. I worked through the stats on this too- by the time I left in 2021, there was virtually no delay to enter the readability process for Java and it was maybe a few weeks for Python. > There was no way to update every call site of your library in an efficient way. Does FB have something better here? G had sufficiently many tools for this, IMO: csearch+grep+sed for the easy changes, Refaster/clang-tidy for more complex stuff (admittedly C++ AST matchers are black magic that few people on even the C++ teams understood). Although I do wish I had known about Comby before I'd left. Rosie - the tool for executing large-scale changes, i.e. if you made a change to 1K+ files (often 10K+ or 100K+), you needed a way to break the change up into multiple PRs - was also an absolutely critical part of this. I never found the approval process Byzantine - undocumented, perhaps, but remarkably streamlined considering that LSCs meant making simultaneous changes to code in Ads, Android, Cloud, Geo, Search, Technical Infrastructure, YouTube, or however Google breaks down the engineering these days.

zeroonetwothree 3y ago
At Facebook there is no OWNERS so if you make an API change you can just update all the code yourself at once. It makes it much easier to rapidly iterate and improve shared libraries.
- colonCapitalDee 3y ago
  Yes, and it also makes it much easier for random people to break your code. It's a trade-off
  - bluGill 3y ago

summerlight 3y ago

The owner system is kind of a necessary evil to enforce reviews from the domain experts. I've seen lots of junior engineers who just want to submit their code and move on and I appreciate their productivity, but there have been so many cases where a seemingly innocuous change eventually led to a million dollars incident and only those domain experts could've detected it. And I've still gotten lots of pagers exactly due to this reason even with this enforcement thanks to someone able to find a relatively lenient reviewer on the codebase with broad ownership. > There was no way to update every call site of your library in an efficient way. There are global approvers. It's not super easy to get their approval (you need to go through the large scale change process), but getting each owner's approval will be exempted if you can get the one.

joshuamorton 3y ago

> Although seemingly innocuous, this made maintaining internal libraries very challenging. There was no way to update every call site of your library in an efficient way. Tools like Rosie were added on top so that you could shard your PRs and shepherd many PRs through a Byzantine approval process. Ownership is the easiest issue to solve here. Beyond a few dozen clients, you're going to run into the inability to run tests fast enough, and beyond a few hundred clients, you won't be able to sync fast enough, even if you skip all presubmits (there will, in expectation, always be a merge conflict in some file). 3-stage migrations/rosie are necessary at that point anyway. Owners is only the problem at like, 5-10 clients, and when I made changes at that scale, messaging owners was usually fast and effective.