4 ms·

I've maintained some pretty big libraries inside Google at one time or another in the last 16 years. I believe our system is bad for both sides (library maintai

I've maintained some pretty big libraries inside Google at one time or another in the last 16 years. I believe our system is bad for both sides (library maintainers and users). Here are some of my frustrations: * the One Version Rule means that library authors can't change a library "upstream": your changes must work now or they can't get in. This sets the bar for landing changes very high, so development moves at a snail's pace. The version control and build system we have makes it difficult to work inside a branch, so collaboration on experimental/new things between engineers is difficult. * users can create any test they wish that exercises their integration with your library. They can depend on things you never promised (Hyrum's law, addressed in the book). Each of these tests becomes a promise you, the library maintainer, make to that user: we will never break that test, no matter how weird it is. This is another huge burden on library maintainers. * the One Version Rule means that, as a user, I can't just pin my dependence on some software to version X. For example, if I depend on Bigtable Client, just give me version 1.4. I don't need the meager performance improvements in version 1.5, because I don't want to risk breaking my project. This means every roll-up release you make, every sync to HEAD you do, risks bringing in some bug in the bleeding-edge version of literally everything you depend on.

mucle6 4y ago
>The version control and build system we have makes it difficult to work inside a branch nit: use fig
- danpalmer 4y ago
  Not sure how much we should discuss in a public forum, but while you may be technically correct, I don't think this is the right solution here. Long-running shared Git branches are a useful tool for large scale changes and integrations. They're not ideal in small teams, but unavoidable at some level and useful if done well. Fig isn't doing that.
- inoffensivename 4y ago
  I use fig, it doesn't solve the problem I'm talking about.
yowlingcat 4y ago
I'd agree with you that this would be bad for most companies I've worked at barring arguably one (Amazon, which is also gigantic) -- at any startup/medium sized company, these practices would not be right-sized to the work being done. Even during my tenure at AWS inside Amazon, I have difficulty believing this would be useful; for example, at AWS, we'd separate services by tiers based on how close to the center they were inside the internal service graph. Running EC2/S3 or another tier 1 service? Yes, you're probably going to index on moving a bit slower to reduce operational risk than not. Running a new AWS service that is more of a "leaf node" than a center node? You can go ahead and move fairly quickly so long as you're obeying AWS best practices, which while somewhat hefty by startup standards are quite relaxed by other corporate standards. What I wonder is whether this kind of heterogeneity would have been a better path for Google than what you describe. Or, is it the case that the sheer monolithic scale of search/ads at Google is such that it just wouldn't make sense, and that continuing to pile incremental resources into the monstrosity (and I mean this gently/positively) of search is what the company must do and so what the engineering culture must enable. But, as you might be alluding to, perhaps the current approach doesn't even suit the needs of the company and is purely bad even for Google's specific problems -- in that case, is it simply there due to cultural cruft/legacy? I haven't worked at Google before, so it's hard for me to say something based on my experience with it.
Rarebox 4y ago
I thought google did this versioning thing for libraries before, but it was stopped for reasonable reasons (g3 components). Basically if you could pin lib versions everyone would be stuck on old versions for a long time, causing difficult debugging work for each user of the library. You'd then also have all sorts of diamond problems: what if you want newest absl but older bigtable client? It's a difficult problem no matter which way you go.
miiiiiike 4y ago
Thank you for saying this. I bought the physical copy of the book a few years ago, before I was a heavy user of Google OSS projects. I cracked it after I was shocked by how bad many of the practices seemed to be.. And it's all working as intended? The bit on docs is confusing. Google docs of some of the most poorly organized, least easily referenced docs I've ever seen. [0] Skip forward a few years and my projects are full of workarounds for years old Google bugs. It feels like fixing basic functionality just isn't a priority. Most of them are literally labeled "NOT A PRIORITY". [0]: You can read Scrapy's docs, and the docs for most major Python libraries, from beginning to end and just "know" how to use it (https://docs.scrapy.org/_/downloads/en/latest/pdf/ https://docs.scrapy.org/_/downloads/en/latest/pdf/). With Google docs you have to piece together fragments of information connected by an complex web of "[barely] Related Topics".
pclmulqdq 4y ago
I worked on one of those infrastructure services (like Bigtable). It was a huge boon to my productivity that the one version rule existed. If we had to maintain 6 past months of releases, we would never get anything done. Since breakages in our client library were our problem, getting a months-old bug means that we can just tell you to recompile rather than figuring out how to backport a fix for you. Hyrum's Law considerations got really weird, especially when people take them too far. I think this is based on the kernel idea of "don't break userspace," but its practical implications are nuts. Hyrum's law has killed infrastructure projects that could have made things a lot better, and has resulted in crazy test-only behavior being the norm. One person on an adjacent team loved taking behaviors as promises (and he also had a reputation as one of the most prolific coders at G). We had to clean up his messes every time he relied on unspecified behavior. I pushed back on his nonsense a few times, particularly when he used an explicitly banned combination of configuration options that we forgot to check-fail on, but always lost. 1.5 SWEs on our team were full-time cleaning up after him.
- inoffensivename 4y ago
  You are a rare engineer who actually pushes back on nonsense. For that, I say "thank you". I feel that there are too few people at Google who will say "stop, that is going to create a problem in the future". Some people are very enthusiastic and well meaning and "productive" (high change count), but create a burden for their colleagues in their fervor to change things.
- phaedrus 4y ago
  To the point in your last sentence, a few years ago I worked at a software company where there was a prolific code writer who similarly tied up about ~2 SWEs cleaning up his messes. He was a Perl programmer who was set loose on C# code. The broken ways the company tracked productivity meant the prolific writer rated as doing great while the three SWEs who had to spend 2/3rds of their time fixing the messes he left behind were rated as unproductive.
  - winterplace 4y ago
    How did the company track productivity? Why did the one get rated as doing great?
kodah 4y ago
Re: the one version rule On the other hand, my users can pin versions, and we maintain a longer LTS window for those features. To this day, that LTS window has never been exercised because we end up having to build backwards compatibility into everything we do. The backwards compatibility promise also means our testing is extremely verbose.
titzer 4y ago
Google overcompensates for a number of practices that don't scale by throwing ungodly computational power at them. I don't know how much CPU forge/TAP consumes these days, but I remember when it was at least 90K cores in a single cluster. It's insane to me that hundreds of thousands of giga-brains are pinned 100% 24/7 to dynamically check literally trillions of things because the combinatorial space was too hard to factor. This is not to disparage the people who built those systems, but there is only so much concrete you can put in a rocket ship.
alpb 4y ago
This is inherently contradictory with the trunk-based development model. I get the feeling you need to pin to v1.4 but ideally by being on the trunk head at all times, you force everyone (especially the library owners, and yourself, by writing tests around your wrapper) to do things properly such as having enough tests in place. Otherwise, you find yourself digging a grave for yourself when the time comes to migrate from v1.4 to v1.7, and it becomes grunt work that nobody wants to take on.

mucle6 4y ago

>The version control and build system we have makes it difficult to work inside a branch nit: use fig

danpalmer 4y ago
Not sure how much we should discuss in a public forum, but while you may be technically correct, I don't think this is the right solution here. Long-running shared Git branches are a useful tool for large scale changes and integrations. They're not ideal in small teams, but unavoidable at some level and useful if done well. Fig isn't doing that.
inoffensivename 4y ago
I use fig, it doesn't solve the problem I'm talking about.

yowlingcat 4y ago

I'd agree with you that this would be bad for most companies I've worked at barring arguably one (Amazon, which is also gigantic) -- at any startup/medium sized company, these practices would not be right-sized to the work being done. Even during my tenure at AWS inside Amazon, I have difficulty believing this would be useful; for example, at AWS, we'd separate services by tiers based on how close to the center they were inside the internal service graph. Running EC2/S3 or another tier 1 service? Yes, you're probably going to index on moving a bit slower to reduce operational risk than not. Running a new AWS service that is more of a "leaf node" than a center node? You can go ahead and move fairly quickly so long as you're obeying AWS best practices, which while somewhat hefty by startup standards are quite relaxed by other corporate standards. What I wonder is whether this kind of heterogeneity would have been a better path for Google than what you describe. Or, is it the case that the sheer monolithic scale of search/ads at Google is such that it just wouldn't make sense, and that continuing to pile incremental resources into the monstrosity (and I mean this gently/positively) of search is what the company must do and so what the engineering culture must enable. But, as you might be alluding to, perhaps the current approach doesn't even suit the needs of the company and is purely bad even for Google's specific problems -- in that case, is it simply there due to cultural cruft/legacy? I haven't worked at Google before, so it's hard for me to say something based on my experience with it.

Rarebox 4y ago

I thought google did this versioning thing for libraries before, but it was stopped for reasonable reasons (g3 components). Basically if you could pin lib versions everyone would be stuck on old versions for a long time, causing difficult debugging work for each user of the library. You'd then also have all sorts of diamond problems: what if you want newest absl but older bigtable client? It's a difficult problem no matter which way you go.

miiiiiike 4y ago

Thank you for saying this. I bought the physical copy of the book a few years ago, before I was a heavy user of Google OSS projects. I cracked it after I was shocked by how bad many of the practices seemed to be.. And it's all working as intended? The bit on docs is confusing. Google docs of some of the most poorly organized, least easily referenced docs I've ever seen. [0] Skip forward a few years and my projects are full of workarounds for years old Google bugs. It feels like fixing basic functionality just isn't a priority. Most of them are literally labeled "NOT A PRIORITY". [0]: You can read Scrapy's docs, and the docs for most major Python libraries, from beginning to end and just "know" how to use it (https://docs.scrapy.org/_/downloads/en/latest/pdf/ https://docs.scrapy.org/_/downloads/en/latest/pdf/). With Google docs you have to piece together fragments of information connected by an complex web of "[barely] Related Topics".

pclmulqdq 4y ago

I worked on one of those infrastructure services (like Bigtable). It was a huge boon to my productivity that the one version rule existed. If we had to maintain 6 past months of releases, we would never get anything done. Since breakages in our client library were our problem, getting a months-old bug means that we can just tell you to recompile rather than figuring out how to backport a fix for you. Hyrum's Law considerations got really weird, especially when people take them too far. I think this is based on the kernel idea of "don't break userspace," but its practical implications are nuts. Hyrum's law has killed infrastructure projects that could have made things a lot better, and has resulted in crazy test-only behavior being the norm. One person on an adjacent team loved taking behaviors as promises (and he also had a reputation as one of the most prolific coders at G). We had to clean up his messes every time he relied on unspecified behavior. I pushed back on his nonsense a few times, particularly when he used an explicitly banned combination of configuration options that we forgot to check-fail on, but always lost. 1.5 SWEs on our team were full-time cleaning up after him.

inoffensivename 4y ago
You are a rare engineer who actually pushes back on nonsense. For that, I say "thank you". I feel that there are too few people at Google who will say "stop, that is going to create a problem in the future". Some people are very enthusiastic and well meaning and "productive" (high change count), but create a burden for their colleagues in their fervor to change things.
phaedrus 4y ago
To the point in your last sentence, a few years ago I worked at a software company where there was a prolific code writer who similarly tied up about ~2 SWEs cleaning up his messes. He was a Perl programmer who was set loose on C# code. The broken ways the company tracked productivity meant the prolific writer rated as doing great while the three SWEs who had to spend 2/3rds of their time fixing the messes he left behind were rated as unproductive.
- winterplace 4y ago
  How did the company track productivity? Why did the one get rated as doing great?

kodah 4y ago

Re: the one version rule On the other hand, my users can pin versions, and we maintain a longer LTS window for those features. To this day, that LTS window has never been exercised because we end up having to build backwards compatibility into everything we do. The backwards compatibility promise also means our testing is extremely verbose.

titzer 4y ago

Google overcompensates for a number of practices that don't scale by throwing ungodly computational power at them. I don't know how much CPU forge/TAP consumes these days, but I remember when it was at least 90K cores in a single cluster. It's insane to me that hundreds of thousands of giga-brains are pinned 100% 24/7 to dynamically check literally trillions of things because the combinatorial space was too hard to factor. This is not to disparage the people who built those systems, but there is only so much concrete you can put in a rocket ship.

alpb 4y ago

This is inherently contradictory with the trunk-based development model. I get the feeling you need to pin to v1.4 but ideally by being on the trunk head at all times, you force everyone (especially the library owners, and yourself, by writing tests around your wrapper) to do things properly such as having enough tests in place. Otherwise, you find yourself digging a grave for yourself when the time comes to migrate from v1.4 to v1.7, and it becomes grunt work that nobody wants to take on.