5 ms·
I was the maintainer of a third-party library used by thousands of dependent applications at Google. I have to admit, I still have not seen on the outside a sy
by dekhn 8y ago
I was the maintainer of a third-party library used by thousands of dependent applications at Google. I have to admit, I still have not seen on the outside a system that allows me to change the version of numpy, and know that thousands of dependent applications either work or break, within an hour of making my change.
Being able to write and use a mapreduce with a high level of confidence that my code would continue to work years later was another nice benefit. MRs I wrote in the first year at Google still compiled and ran with minimal changes almost 10 years later(!) which is amazing given the amount of environmental change that occurred.
That said, somebody could still halt development across the company by changing and checking in a core file (like proto defs for websearch) without testing.
Whatever social system led to google3/borg and the amazing productivity associated with it, it was a special moment that hasn't been replicated many times.
- shereadsthenews 8y agoPeople who like multirepos are always saying how easy it is to pin dependencies but like you I haven’t seen anyone doing it right since I left Google. The monorepo third-party system works well in practice. Ps thanks for getting scipy into third_party all those years ago.
- draw_down 8y agoWhat’s really strange is how many people in that recent monorepo discussion were advocating for it in the context of a small org. I’ve only encountered monorepo in a large org, where I’m pretty convinced it’s more efficient than the alternative.
- pjc50 8y ago> The monorepo third-party system works well in practice. It's worth noting that this is only viable at Google because they don't use git. Git's insistence on every client having a full copy of all history of every file in the repository makes monorepo much more expensive. I see conflicting reports over whether google use Perforce or something proprietary called "piper"?
- detaro 8y agoAFAIK they used perforce in the past and then built their own later.
- alexhutcheson 8y agoThis paper has details and history regarding Perforce and Piper: https://ai.google/research/pubs/pub45424 https://ai.google/research/pubs/pub45424
- anonymouzz 8y agoAfaik it's a custom system with download-and-cache-on-demand. Note that there's nothing forbidding you from writing a virtual git filesystem that fetches objects from some centralized repo as files are open()ed. Git on cloud steroids.
- chokolad 8y agoMicrosoft actually had done it and pushed the whole Windows repo into git. https://github.com/Microsoft/VFSForGit https://github.com/Microsoft/VFSForGit
- pacala 8y agoFWIW, there is [used to be?] a git interface for the centralized monorepo. Really handy for managing multiple dev braches at the same time.
- monocasa 8y agoVfsforgit let's you do that
- sangnoir 8y ago> Git's insistence on every client having a full copy of all history of every file in the repository makes monorepo much more expensive. You might be happy to know Microsoft created a Virtual File System for Git[1] [2] so you do not have to have every file checked out in your working directory. Microsoft uses Git in a monorepo (for Windows, and it's 2.5 million files/300GB[2]) 1. https://github.com/Microsoft/VFSForGit https://github.com/Microsoft/VFSForGit 2. https://vfsforgit.org/ https://vfsforgit.org/
- pacala 8y agoPinning dependencies is easy, and gets one stuck on Java 1.3 and IE 6. Since the library maintainer doesn't know it broke dependent code, the dependent code will be unlikely to have a smooth upgrade path. It's just a matter of having the luxury of picking up a time to pay the cost of dependency upgrades. Which cost may be high in a world where your dependencies also pin dependencies, likely at different versions. The underlying assumption it that the project will fold in 2 years anyways, so one may get away with never doing dependency upgrade.
- alexis_fr 8y agoThat’s a good point. Do we have any solution from companies who use git/multirepos for their libraries? Is there anyway to compile refactored libraries against existing code?
- detaro 8y agoFor various public package managers, there's tools that automatically send e.g. a GitHub PR for an updated dependency file when a dependency updates, which then is tested by CI. I guess something like that could be made for private repos and test versions of packages too, reporting back to the authors of the dependency.
- willnorris 8y agoAnd for anyone that wants to learn more about how Google manages third-party code, all of our docs are public at https://opensource.google.com/docs/thirdparty/ https://opensource.google.com/docs/thirdparty/
- mlthoughts2018 8y agoI think the problem is more just a reversal of perspective. It’s up to each different app to detect and respond to a breaking change from a dependency. The idea of wanting to make a change to a third party library and then see all the downstream consumers who would be broken by that change if they updated to start consuming that change is an incredibly stupid thing to want, and it’s no measure of success whatsoever to build something that gives you that information. It’s like the most giant case of coupling you can imagine (letting the statuses of thousands of consumer apps act as any type of constraint on the developer choices of the third party library, as opposed to all those consumer apps opting in to changes on their own terms by updating their dependencies). Imagine if I have shared a bunch of copies of my resume with a bunch of recruiters. They are out there selling me as a candidate or whatever. Now I decide I want to change my resume, but I don’t know if it’s going to upset the approach some recruiter is taking. If I can’t update my resume unless I first consult a big oracle that tells me which recruiters will be negatively impacted, that’s a problem, and not at all some type of live-with-able “customer service” positive thing. It’s just plain old bad coupling. Creating such a system that could automatically diff the old resume’s usage constraints against my proposed changes would be a gigantic waste of time. The exact opposite of something to celebrate. I say this as someone who routinely writes in-house software libraries used by dozens or hundreds of other apps, various teams, and even a few that are open source. The primary thing gauging the health of our development is that we are decoupled from any consumers. We are free to make whatever changes we want, and whether downstream teams would like to receive those changes is wholly an opt-in process with versioned dependencies and easy rollbacks controlled by those consumers.
- idontpost 8y agoI agree in principle, but in practice its important to remember that your package only exists to be consumed by its dependencies. That is its sole purpose. If your changes aren't serving those consumers, then they're the wrong changes. It's also useful information in the edge case where consumers are relying on undocumented or unintentional behavior in your package. Yes, you don't want a hard-constraint of no-breaking changes ever, but knowing immediately when a change is breaking change (especially if you didn't intend it to be) is useful.
- Teknoman117 8y agoAre you allowed to share if the internal repository a custom version control system or is it one of the open source ones?
- hknd 8y agoIt is, like most of the tools in Google/Facebook/..., a custom solution. These companies need their custom solution because of their scale.
- alexis_fr 8y agoIt begs the question : are big companies like Atlassian, GitHub or say VolksWagen required, by their scale, to have a slick solution in order to perform on the international market?
- dekhn 8y agoit's already been published. Google used Perforce for at least a decade and then cloned the perforce wire protocol when it was clear perforce wouldn't keep scaling (there's a great presentation online about google using RAMSANs to store the perforce index), but backed the repo in something like bigtable or spanner. I'll give google credit for one thing: it can change backends a lot without too much user visible pain.
- joshuamorton 8y ago>I'll give google credit for one thing: it can change backends a lot without too much user visible pain. Amusingly, this comment may explain how someone in this thread can be incredulous about the idea that things get rewritten every 2-3 years (which yes is an exaggeration). Google is very good at making sweeping infrastructural changes (generally improvements, I might add) without significant user pain.