23 ms·

Why Google stores billions of lines of code in a single repository (2016)

• hk__2 9y ago
  That’s a 2016 article and it’s called "Why Google stores billions of lines of code in a single repository".
  • deleted 9y ago
    [deleted]
  • sctb 9y ago
    Thanks, updated.
• soroso 9y ago
  I wonder what do the bots commit in the monorepo. It is something that has been making me curious for a while.
  • Xorlev 9y ago
    Skynet plans, nothing big. Source: Googler.
  • greydata 9y ago
    Nothing too interesting. They're restricted so they can only update their own source code.
    • Xorlev 9y ago
      Except for that _one_ time. ZRH SRE has never been the same.
  • joshuamorton 9y ago
    Not much, mostly just prototype messaging apps.
• ericfrederich 9y ago
  Good Google Talk video about it too.
  • adyavanapalli 9y ago
    Can you post the link? Thanks!
    • bch 9y ago
      Could be this one: https://www.youtube.com/watch?v=W71BTkUbdqE https://www.youtube.com/watch?v=W71BTkUbdqE
  • yeukhon 9y ago
    Also adding Google NYC Developer Meetup talks (been to these events a few times, and found them really interesting) https://www.youtube.com/playlist?list=PLAD8A7B6D66DDD297 https://www.youtube.com/playlist?list=PLAD8A7B6D66DDD297
• chias 9y ago
  Google's mono-repo is interesting though, in that you can check out individual directories without having to check out the entire repo. It's very different from checking out a bajillion-line git repo.
  • jsolson 9y ago
    While technically true due to some features of tooling, that is really only masking off part of the repo under a READ-ONLY directory. Builds can (and usually do) depend on things that aren't part of your local checkout. I'd say CitC is a much more accurate representation of the way Piper and blaze "expect" things to work.
    • netheril96 9y ago
      The dependencies are still downloaded only on need though.
      • joshuamorton 9y ago
        The dependencies aren't really "downloaded" at all. When you build something, the artifacts are cached locally, but the files you are editing generally speaking not actually stored on your machine. They're accessed on demand via FUSE.
  • vvanders 9y ago
    That's pretty common feature in most non-DCVS. It was nice having Perforce on the last game I worked on. The art directory was ~500GB and not fun to pull down even with a P4 proxy.
  • nerfhammer 9y ago
    It's kind of interesting that nowadays people assume that version control system == git. For a huge, non-open codebase there are some pretty large downsides to a fully distributed VCS in exchange for relatively few benefits.
    • 9y ago
• 2_listerine_pls 9y ago
  Facebooks doesn't, y tho?
  • dmnd 9y ago
    Doesn't it? https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-facebook/ https://code.facebook.com/posts/218678814984400/scaling-merc...
  • ridiculous_fish 9y ago
    Facebook monorepos harder than Google!
• ashwins227 9y ago
  Can anyone quickly summarize the article in a few pointers? I'm looking for benefits/drawbacks and trade-offs.
  • cbcoutinho 9y ago
    Just open up the pdf and look on the front page. > Key insights > Google has shown the monolithic model of source code management can scale to a repository of one billion files, 35 million commits, and tens of thousands of developers. > Benefits include unified versioning, extensive code sharing, simplified dependency management, atomic changes, large-scale refactoring, collaboration across teams, flexible code ownership, and code visibility. > Drawbacks include having to create and scale tools for development and execution and maintain code health, as well as potential for codebase complexity (such as unnecessary dependencies)
• ridiculous_fish 9y ago
  What fraction of Google's code is in the monorepo these days? Android, Chrome, and ChromeOS are not, and those are certainly large projects.
  • gsnedders 9y ago
    I believe anything not open source, or reasonably expected to become open source.
    • ridiculous_fish 9y ago
      I think that's an effect, not a cause. It's annoying to open source google3 code because it means untangling dependencies. Meanwhile Android has lots of SOC-specific and other development happening behind the wall. So the open source question seems orthogonal. Google today has separate repos (android, chrome, chromeos, google3), each with its own build system: Gradle, gyp/ninja, Portage, Blaze. There's hysterical raisins, but I wonder if Google considers it to be a good thing these projects are so different, or a wart they would prefer to fix?
    • tehlike 9y ago
      Plenty of opensource projects are in google3(that frequently gets merged into external)
      • Xorlev 9y ago
        +1. Many OSS projects live in google3 (e.g. gRPC, protobufs, etc.)
      • jingwen 9y ago
        Yep. Projects use Copybara to import/export OSS code. https://github.com/google/copybara https://github.com/google/copybara
  • perfmode 9y ago
    I'd bet pretty much everything that runs in Google data centers.
• mc32 9y ago
  I heard updating pager duty [on-call] commits used to be a mess --perhaps it was apocryphal, but if not, why not use a different system for the pager duty updates?
  • foota 9y ago
    Using the existing thing was probably easier than anything else.
  • srtjstjsj 9y ago
    Don't want to risk breaking all of oncall for the whole company if there's a mistake?
  • ikiris 9y ago
    One does not simply update the pager file during MTV business hours.
• throw7 9y ago
  I'd like to see the world checked into an rcs.
• srtjstjsj 9y ago
  "Why"? Because Google has billions of lines of code. Does any company not store all their related code in a single repository?
  • perfmode 9y ago
    Most unicorns don't have mono repos.
  • wmf 9y ago
    To me the defining aspect of a monorepo is storing apparently unrelated code in the same repo, which is highly non-intuitive.
  • eadmund 9y ago
    We don't, and it's hell. I may make a change to X, but some other developer doesn't know it, and then one day when he makes a change to X and tries to pull those changes into Y, now he has to change the bits of Y which are affected by my changes as well as his. I would love to have a monorepo, although I'd advocate for branch-based rather than trunk-based development. Yes, merges can be their own kind of hell, but they impose that cost on the one making breaking changes, rather than everyone else.
    • scarmig 9y ago
      That just means that no one makes big breaking changes even if they night be necessary.
  • dastbe 9y ago
    Amazon has a repo per package model, but has a meta-versioning system (called version sets) that tracks revisions of packages built together and a way for declaring package dependencies. So in some sense you're building your own monorepo out of a bunch of packages (usually by branching off a parent version set). It fulfills a lot of the same goals of having a monorepo in terms of scaling a large organization, while having somewhat different pros and cons.
• shakencrew 9y ago
  previous discussion: https://news.ycombinator.com/item?id=11991479 https://news.ycombinator.com/item?id=11991479
• cletus 9y ago
  So as someone who previously worked for Google and now works for Facebook, it's interesting to see the differences. When people talk about Google's monolithic repo they're talking about Google3. This excludes ChromeOS, Chrome and Android, which ard all Git repos that have their own toolchains. Google3 here consists of several parts: - The source code itself, which is essentially Perforce. This includes code in C++, Java, Python, Javascript, Objective-C, Go and a handful of other minor languages. - SrcFS. This allows you to check out only part of the repo and depend on the rest via read-only links to what you need from the rest. - Blaze. Much like Bazel. This is the system that defines how to build various artifacts. All dependencies are explicit, meaning you can create a true dependency graph for any piece of code. This is super-important because of... - Forge. Caching of built artifacts. The hit-rate on this is very good and it consumes a huge amount of resources given the number of artifacts produced. Forge turns build times for some binaries from hours (even days) into minutes or even seconds. - ObjFS. SrcFS is for source files. ObjFS is for built artifacts. This all leads to what is usually a pretty good workflow like the ability to check out directories if you want to modify them and just use the read only version if you don't. You can still step through the read only code with a debugger however. Now Facebook I have less experience with (<6 months) but broadly there are four repos: www, fbobjc, fbandroid and fbcode (C++, Java, Thrift services, etc). At one point these were Git but for various reasons ended up being migrated to Mercurial some years ago. The FB case (IMHO) highlights just how useful it can be to have one repo. Google uses protobufs for platform independence. FB uses GraphQL at a client level and Thrift at the service level. So one pain point is that, for example, you can modify a GraphQL endpoint in one repo but its used by clients in others (ie mobile clients). There are lots of warnings about making backward-incompatible changes, some of them excessively pessimistic because deterministically showing something will break some mobile build in another repo is hard. Google3 has less of these problems because the code is in the same repo. On top of that, Google has spent a vast amount of effort making it so the same build and caching systems can handle C++ server code as well as Objective-C iOS app code. Basically if you're working on Google3 you basically compile very little to nothing locally. Engineers on Android, Chrome and ChromeOS however compile a lot of things locally and thus get far beefier workstations. At FB the mobile build system doesn't seem to be as advanced in that there is a far higher proportion of local building. IIRC the Git people seemed to reject the idea of large code bases. Or, rather, their solution was to use Git submodules. There was (and maybe is?) parts of the Git codebase that didn't scale because they were O(n). Apologies if I'm misspeaking here but I peripherally followed these discussions on HN and elsewhere years ago as someone from the outside looking in so I'm no authority on this. The problem of course is that Git submodules don't give you the benefits of a single repo and I've honestly not heard anyone say anything good about Git submodules. Just to stress, the above is just my personal experience and I hope it's taken as intended: general observations rather than complaints and definitely not arguing that one is objectively better than the other. There are simply tradeoffs. Also, there are definite issues with Google3, like the dependency graph getting so large that even reading it in and figuring out what to build is a significant performance cost and optimization issue.
• borplk 9y ago
  How would they do CI/CD if it's all in one big repo? I suppose you could do it if you had a very strict rule where absolutely everything that could affect a "unit" was inside its own directory (but never and nothing higher up than that "project root"). So you could check which sub-directory is affected within a commit and so on.
  • tehlike 9y ago
    Oh it works really well. One benefit is all tests affected by your target is run as you presubmit. The benefit is everything is at head so things like library bugs, security bugs are all handled naturally as part of the new releases. This usually happens twice a week for most server binaries.
    • Xorlev 9y ago
      So long as the presubmits are configured correctly, or someone ignores failures and force submits. :) It's usually quite nice though, and changes that break lots of projects are rolled back quite fast. Usually.
      • tehlike 9y ago
        As much as people hate, force submits are a fact of life.
      • tehlike 9y ago
        Rollback in some cases happen automatically, even.
    • krzyk 9y ago
      So, how long does the commit with the checks take take?
      • tehlike 9y ago
        Depends. Mobile tests that run on emulators are the worst. Unit tests finish relatively fast, integration tests that bring up server tend to be slow (30-40mins best case almost for these for projects im working on). The cost of this gets amortized two ways: you can run immediate unit tests manually on command line as the fastest signal. Then when you send for code review, presubmit runs. During code reviews you may choose to run them as you go. Eventually when you submit they run again. If there has not been any changes to your commit/cl and there is an already passing submit, it will just skip and submit.
• wiradikusuma 9y ago
  The lazy me thinks maybe 1 kitchen sink repo is better. Here's my problem: I can be working on multiple projects at the same time. Each project has multiple modules (core, api, www, admin, android, etc -- I use microservices on Google App Engine). Sometimes, some modules have "feature" branches. Oh, did I tell you I work on both Desktop and Laptop? The problem is syncing. Before traveling, I need to make sure the projects/modules I'll be working on the go all have latest commit from Desktop. My question: Is there some "dashboard/overview" for all Git projects? So I can quickly tell, "Ok, all projects are at latest commit, and oh I'm working on feature branch for project X and Y."
  • erikb 9y ago
    There is a dashboard called github? Or what are you looking for? I think if you want to make sure that all your changes are in, you need to do what most programmers do and learn to finish a programming session with a commit and push, just like you finish a sentence with a dot. Once you are used to it, the chance of forgetting are really low.
    • wiradikusuma 9y ago
      GitHub.com doesn't tell you if you have uncommitted, and even unpushed commit. (the desktop app does, but you need to check one by one for each project)
  • adamcharnock 9y ago
    This is something I’ve been idly thinking about too. I agree with regards to committing regularly, but sometimes life happens and one can forget. I’ve been considering writing a python script that checks my local repos for uncommitted/unpushed changes and - now I think about it - perhaps also runs when I start a new terminal session just for good measure.
  • mjw1007 9y ago
    Maybe 'myrepos' will do what you want. http://myrepos.branchable.com/ http://myrepos.branchable.com/
• haglin 9y ago
  I work in a large company and I have used a central repository for six years and a distributed for six years. I think a central repository is better. The benefits are: 1) Transparency. I can see what everybody else is doing and if somebody has an interesting project I can find it quickly. You can also learn a lot from looking at other peoples changes. 2) Faster. To check out the source code for the project I now work on takes an hour in the distributed system, while it only took 5 minutes in the centralized system. 3) Always backed up. All code that is checked into the central repository is backed up. It has happened twice that employees have left and code was lost because they only checked it in locally. Many have only used CVS or SVN, which are horrible. I rather use Git or Mercurial, but Perforce is really good.
  • adrianmonk 9y ago
    The third problem can essentially be solved by doing all your production builds by checking out code from some central repository. If you follow that rule, then you guarantee you'll have the source code for every binary in production. That way, you can still have a distributed repository (Git, Mercurial, etc.) if you want. Even if some code exists only in some developer's local repository, it's presumably not that big of a deal since that code can never have made it to production.
  • JoshTriplett 9y ago
    > 1) Transparency. I can see what everybody else is doing and if somebody has an interesting project I can find it quickly. You can also learn a lot from looking at other peoples changes. This doesn't require a single central repository, just that all repositories live in a common location. > 2) Faster. To check out the source code for the project I now work on takes an hour in the distributed system, while it only took 5 minutes in the centralized system. What distributed repository management system do you use, and what centralized system did you use? > 3) Always backed up. All code that is checked into the central repository is backed up. It has happened twice that employees have left and code was lost because they only checked it in locally. As with point 1, this doesn't require a single central repository, just that all repositories live in a common location.
    • 9y ago
• tomerbd 9y ago
  "The two are not mutually exclusive. I utilize what I call ensemble repositories that, for us, submodule the individual repositories" - http://disq.us/p/1hj9nmu http://disq.us/p/1hj9nmu
• diebir 9y ago
  At this point I normally assume that if Google does something a certain way, or uses a particular proprietary technology, then it likely should NOT be used. I work for a place that tried to model itself culturally after Google and Facebook and has had a lot of engineers moving back and forth. If Google is anything like us, then it creates wrong incentives to invent in-house stuff. See, in their expense scheme the salaries of engineers are not a huge deal. It's cheap to have people do things. There is also an incentive for an engineer to try and "leave a mark". There are also a bias to hiring new and unexperienced people, who fail to learn the existing tech and replace it with something different, simpler and less functional (by the time it matures, it becomes just as complex, however). I live in Java world, so I am seeing Google reinvent (poorly) every bit of Java tech: DI (Dagger), build tools (Gradle), commons libraries (Guava) and the list goes on and on. Well, apparently, they also reinvented internally Git, Jenkins and the rest of the tools. The rest of us should probably NOT do it this way.
  • cromwellian 9y ago
    Gradle was not a Google invention. Dagger was invented by Square. Google invented Guice (Bob Lee now at Square) which was way better than Spring and J2EE. Guava is far better designed than Apache Commons. Dagger2 was invented by Google for good reason because it works purely as an annotation processor without runtime bytecode classloader magic. That means it is easier to debug, faster to startup on mobile, and can be crosscompiled with j2objc and GWT/j2cl. Much of the time in-house stuff is created to deal with scalability or maintain ability problems.
• ngold 9y ago
  Because of course they do. Just like we all do. Everyone has a file they backup just because they might need to reference it. Index.txt is my cross. In terms of sheer code. I hope they have some Jedi ai to cross reference it. If they don't they will. In fact we help everyday. No point in burning the printing presses. We have to learn to get along. Enjoy living, learning will come naturally as you pursue your interests. Those interests will change, growing, cultivating other fields of knowledge, grappling for your attention. Culture you enjoy as only a human can, and cultivate it. Distribute and share it. As humans we are denying our own renaissance with a ministry of silly walks. Sometimes I think high school UN clubs are better at running the world. Why not. It doesn't seem to matter anymore.
• sytse 9y ago
  I think that what is missing when using multiple git repos is the ability to make a code change that spans multiple projects. We're open to adding that to GitLab
  • ereyes01 9y ago
    In places I've worked in before that needed to join concurrent changes across repositories, we would have the build system always build from a super-repo that had the other repositories tracked as submodules. Co-dependent changes were pulled in via a single commit in the super-repo. This was kind of cumbersome to maintain TBH, and the fact that changes to different repos can be dependent on one another seems to strongly suggest that the code should be together in the same repo. Personally, I opt for mono-repos until I'm forced to change for whatever reason.
  • jacobr 9y ago
    This will help if you don't want to run a monorepo, but many in the industry consider a monorepo suitable for their organisation. The biggest blocker with Gitlab in my opinion is not being able to only run a job if some specific folder was modified (https://gitlab.com/gitlab-org/gitlab-ce/issues/19232 https://gitlab.com/gitlab-org/gitlab-ce/issues/19232).
    • sytse 9y ago
      I agree that functionality would be great to have. We planning it in the next 3 months but we also very open to someone contributing it earlier.
  • wocram 9y ago
    Is this not something that is already being worked on? Making commits across even submodules seems to have little support anywhere.
• czardoz 9y ago
  Does anyone have "aha" moments to share about working in a single repository? Stuff that just made you say, "Great, I wouldn't have been able to do that if it was in separate repositories!".
  • nwienert 9y ago
    1. Huge changes that affect the API's of multiple sub-packages. 2. Having no friction to change anything makes you far more productive and ambitious. 3. Scripting at a org-level means you can automate things more easily and more in depth. We run an entirely node stack so Lerna enables this in the first place. Given that, I'd never move to more than one repo if possible. It's almost all downside: more overhead/fragmentation, less control, more wasted time/mental overhead moving between things, API friction that reduces ambitious change. Only downside of monorepo is Github not supporting them well. If you want to release some sub-packages as OSS, or want to use GH to track issues you're stuck using one big repo to handle everything. I'd bet Github fixes this within the next year or so though.
  • BrandonY 9y ago
    Once, I noticed that a minor piece of how Google's Python system tests could be very slightly cleaner and more consistent. It would be a tiny change, it seemed very safe, and it was easily accomplishable with a short sed script, but it'd also be a backward-incompatible change across the projects of hundreds of teams and thousands of build targets. I was able to make that change with only a few commits and without needing to bother most of those teams. These sorts of small, general, large scale cleanup commits are quite common at Google, and they're encouraged. They help keep the codebase healthy. There are special groups that review them so that all of the individual teams affected don't have to bother, and there are tools to manage the additional testing and approval requirements for such a change. At my previous company, making such a change would have been a major undertaking. I never would have considered a refactor of that scale without a critical need. They had thousands of packages, each of which had its own repository and an incredibly complex web of build and runtime dependencies. It was a nightmare, and fiddling to find a working sets of versions of internal dependencies took up way, way too much of my time each day.
  • 9y ago
• bookwormAT 9y ago
  Can someone further explain how the pre commit phase works? I don't get how/why "pre commits" work without feature branches? How are my changes shared with the reviewer, if I there is no feature branch? Is my local code uploaded to that review tool mentioned in the article? And then what happens if the reviewer requests changes? I probably did get this completely wrong, so thanks in advance for pushing me in the right direction.
  • tdeck 9y ago
    Perforce (the origin of Piper) has a concept of changelists. Some changelists are submitted (committed) while others are not. So the review works by uploading your changelist to Perforce, then pointing people to that changelist. It's like an unnamed feature branch that can't easily be rebased off of. Changelists do have a base, and you do a "g4 sync" to essentially rebase off of master. Does that make sense?
    • mjw1007 9y ago
      When people talk about making a single global change to update all clients when they change an interface, are they updating unsubmitted changelists too?
      • izacus 9y ago
        Well, the act of submitting a changelist essentially runs a test suite which requires that the changelist has no merge conflicts with the head and that the relevant tests pass. From that it follows that if someone changes an interface, you'll get either merge conflicts or test failures on your own changelist. Meaning - it's the changelists authors task to sync it up to the current head state so the refactorers won't touch unsubmitted changelists. It's pretty much the same as GitHub pull requests - the changelists are supposed to be decently short lived and if the master code changes it's up to you to resolve conflicts and get it into a mergeable state again.
      • jsolson 9y ago
        No. Unsubmitted changes at Google usually come in one of two flavors, short-lived (abandoned or submitted within a few days) or perpetual. The latter flavor is often for "I think we might want this". It's not uncommon for those to be completely rewritten if they're actually needed. There's usually a preference for submitting useful things (with tests!) and flag gating them to cut down on bitrot. I have seen exceptions -- I reported a bug in a fiddly bit of epoll-related code and an engineer on my team had a multi-year-old fix -- he hadn't submitted it because he wasn't confident he'd found an actual bug. The final changelist number was more than double the original CL number (unsubmitted changes get re-numbered to fit in sequence when they're submitted -- the original number redirects to the final submitted version in our tooling).
• matttproud 9y ago
  Large-scale refactorings are actually pleasant. My team owns a framework and set of libraries that are widely used within the Google monorepo. We confidently forward-update user code and prune deprecated APIs with relative ease — with benefits of doing it staged or all-at-once atomically. It's imperfect, but maintenance in distributed repositories is infinitely worse. Still, I remember the earlier days of the monorepo and keeping Perforce client file maps; that was a pain! https://www.perforce.com/perforce/r15.1/manuals/dvcs/_specify_mappings.html https://www.perforce.com/perforce/r15.1/manuals/dvcs/_specif...
  • InclinedPlane 9y ago
    I agree, though I wouldn't say this is a fundamental aspect to distributed systems, mostly just a consequence of git being built with terrible merge and merge conflict tooling.
  • nickm12 9y ago
    Infinitely worse? That's a bit hyperbolic.
    • coderdude 9y ago
      Not just a bit hyperbolic. Rather, obviously hyperbolic. You should ask yourself why that's not obvious to you.
  • Roritharr 9y ago
    How many people work on the repository tooling at Google? I'm asking because i wouldn't know how to setup a mono repository at my 50 people Startup even if we deemed this to be necessary.
    • coderdude 9y ago
      (Not to mention the invaluable nature of this information to anyone working in that sector.)
    • izacus 9y ago
• braeckmans 9y ago
  Do you know that you can hack any ATM machine !!! We have specially programmed ATM cards that can be used to hack any ATM machine, this ATM cards can be used to withdraw cash at the ATM or swipe, stores and outlets. We sell this cards to all our customers and interested buyers worldwide, the cards has a daily withdrawal limit of $5000 in ATM and up to $100,000 spending limit in stores. and also if you in need of any other cyber hacking services, we are here for you at any time any day. Here is our price list for ATM cards: BALANCE PRICE $5,000----------------$300 $10,000 ------------- $650 $20,000 ------------- $1,200 $35,000 --------------$1,900 $50,000 ------------- $2,700 $100,000------------- $5,200 The price include shipping fees,order now: via email...braeckmansj@outlook.com
• oblio 9y ago
  How do they handle Android? It's one thing when you can just go "use the latest" for web service and another when you have N branches. I'm not sure a mono repo is as big of a benefit then.
  • GoatOfAplomb 9y ago
    Android Open Source Project is still in gerrit, where the code is stored as a set of many git repositories. There's a "repo" tool that adds a helpful layer of abstraction to make those git repositories (mostly) look like a single repository.
• guftagu 9y ago
  I develop an application that has a SPA frontend and an API backend. They can love inside separate repos but I prefer to keep them together because if I change the API signature, I'll also make the same changes in the frontend and they will deploy together at once.
• nickm12 9y ago
  I've worked with both a distributed repo model and a monorepo model and vastly prefer the distributed approach (given the right tooling). The trade-offs are complementary and no doubt with proper discipline you can try to maximize the benefits, while minimizing the downside. But here's what I don't like about working in a large monorepo: 1) Difficult to track changes to the code I'm interested in. Every day there are hundreds of changes in the repo and almost all of them have nothing to do with what I'm working on. 2) all sorts of operations take longer (pulling, grepping source, etc.) to support code I couldn't care less about. 3) Frequently have to update the world at once. Unless the repo can store multiple versions of the same module, then all the consumers have to be updated at once, even if it's inconvenient. Sometimes migrations are better done gradually. 4) Encourages sloppy dependency management. There are frequently unclear boundaries between software layers. I'm sure people will say "if you're having those problems, you're doing it wrong" but the same thing could be said to people who find the distributed model problematic.
  • Sacho 9y ago
    > Every day there are hundreds of changes in the repo and almost all of them have nothing to do with what I'm working on. Wouldn't the right tooling be able to show you changes to the slice of code you're interested in? I remember SVN would allow you to checkout just a single subdirectory, for example. > all sorts of operations take longer (pulling, grepping source, etc.) to support code I couldn't care less about. Makes sense about the pulling, but again, wouldn't grepping be configurable to only search where you need to? > Frequently have to update the world at once. Unless the repo can store multiple versions of the same module, then all the consumers have to be updated at once, even if it's inconvenient. Sometimes migrations are better done gradually. I'm not sure I understood you here, can you expand on that? Do you mean all the devs have to update their module? Why not use tagged/branched version of libs instead of working off trunk? What kind of tooling do you use for the distributed approach?
    • gmueckl 9y ago
      Using different branches for different modules becomes hard in a monorepo when branching is a global operation. You can only have a single branch checked out in your working copy then. The only solution I can think of is to create a copy within the monorepo to create de facto branches without regular VCS support. This would be kind of terrible.
• TobiasA 9y ago
  I'm working on a project where we have two projects: a desktop app and a web backend. They each live in their own repo. However, this approach is proving tedious for us now, as every new feature often means two branches (one in each repo) which also means two separate pull requests when the feature is ready to be merged. Has anyone encountered this kind of issue, and is the solution to simply merge the two repos into one monorepo?
• randomsearch 9y ago
  Doesn’t this create a single point of failure for the development of the entire software ecosystem at google? What happens if something goes disastrously wrong with their repo? Does everyone twiddle their thumbs until it gets fixed? Also, isn’t this is the opposite of “separation of concerns”... code should be divided into small units of functionality that don’t overlap. Minimise interdependencies. Eliminate unnecessary work. Is this the most efficient and future-proof approach?
  • boulos 9y ago
    Yes, when the system is down, it's really bad. Though it's an extremely reliable system as so many folks depend on it (note there's also lots of layers of tooling, so maybe you just need commits to work versus code review versus your local edits, etc.). In my experience, the worst things are actually when the build/test system is running slowly (thus blocking commits until tests are complete) or the bog standard "ugh, someone decided to force submit even though it said the tests were failing. Roll that back, please". As for separation of concerns, there's actually a lot of scoping / visibility stuff precisely to avoid letting people depend on things they shouldn't. I think you now have to explicitly open up your visibility to let random other projects depend on you. A monorepo doesn't require that you share, it just permits it.
    • randomsearch 9y ago
      Interesting. Every system is going to be down times, so I don't worry about that too much. I'd be more worried about a design flaw or bug that doesn't get exposed until you're painted into a corner and it's hard to escape. Good point about the scoping. I suppose it really depends on how developers use it - do they default to sharing or not?
• symboltoproc 9y ago
  I'm a bit disappointed that they don't back their publication with any numbers that compare the two approaches. I don't see the scientific value of what would also be a good click-bait blog post.
• deleted 9y ago
  [deleted]
• techupcian 9y ago
  If you need any hacking aid, I will like to advice you to reach hackerdanielssmith@gmail.com for all you need in ethical hacking and help. He is a helpful source to me since years back.
• bjork 9y ago
  I guess you can’t blame the weaknesses of Google’s products (e.g. searching in gmail) on the repository system.
• known 9y ago
  So that developers can REUSE code.
• nsm 9y ago
  There is a great talk by their C++ dev team about how a build everything from source, mono repo really helps with C++ change enablement https://m.youtube.com/watch?v=tISy7EJQPzI&t=2589s https://m.youtube.com/watch?v=tISy7EJQPzI&t=2589s Personally, I’d have to agree that at a minimum a mono repo per distinct product role (server side in one mono repo, desktop software in one mono repo) is immensely valuable.
• crucifiction 9y ago
  The real reason is why any large enterprise does something unusual: “it has always been like that”.
• luckydude 9y ago
  I scanned this, the usual mono/multi arguments. All made moot by BitKeeper's nested approach: http://mcvoy.com/lm/bkdocs/nested.html http://mcvoy.com/lm/bkdocs/nested.html in an open source system http://bitkeeper.org http://bitkeeper.org What nested brings to the table is the semantics of a mono repo with the advantages of a multi repo. The whole thing walks in lockstep, if you have a 3 week old version of the kernel and you add in the testing component (subrepo) then you get the 3 week old version of the testing component, all lined up with the same heads in the same tip commit. I get it that git won but at least steal the ideas. Edit: BTW, bk has a bk fast-import that usually works (it doesn't like octopus merges but other than that....)
• pipu 9y ago
  If someone's interested, I made a summary of the paper a year back: https://www.extreg.com/blog/2017/02/googles-ultra-large-scale-monolithic-source-code-repository/ https://www.extreg.com/blog/2017/02/googles-ultra-large-scal...
  • JediSWEng 9y ago
    It's a good write up. To me, Piper is a monolithic version control system which is geared towards good engineering practices. As far as I know there are only two such systems in use today and the other one is very dated and older than a lot of things out there. When people say they have worked in a monolithic repo, then they typically mean a 1 repo under one of the open source version control systems, but none of these actually do or support what is needed when working with a monolithic repo AND modern/good engineering practices. For that specialised VCS is required and there is very few examples of that, none of which is in open source software. Git probably could be made to do this kind of stuff, but it would require some extensions to the DAG as well as extend on it's already verbose command line set. But I think it is doable. The question is who can do it? Most are probably under some strict NDAs.
• siliconc0w 9y ago
  I think the monorepo is better but it's really hard to pull off without Google-like tooling and engineering practices. Git + meticulous dependency tracking and strict versioning conventions is probably the better move for most companies.
  • jorblumesea 9y ago
    +1, I really dislike the "Let's do it because Google does it" mentality in software engineering. Google operates at a scale and encounters problems that the average company would probably never encounter.
• ssamuli 9y ago
  So, monorepo seems to be great, but why don’t we have open source tools to deal with that?
• fenollp 9y ago
  Here is yet another tool for managing multiple repos as one: https://supertanker.github.io/tsrc/ https://supertanker.github.io/tsrc/
• pedro1976 9y ago
  The clear advantage of a mono repo over a mono-purpose repo is technical dept. The goal is that you never build up technical debt and immediately patch all references in an atomic change. Example: Let's say you introduce a breaking change in lib A that is used in libs B and C. First problem is visibility, that A does not necessarily see that it is used in C and D. Second, you the build should break immediately and not until someone build C/D.
  • afro88 9y ago
    Tech debt isn't just breaking changes though, and mono repo does nothing to curb all the other types of tech debt: * Accumulation of FIXMEs * Partial refactors cut short after change in business reqs * Quick hacks near release time * "this could be done better if I had time" * Overdue re-architecture after accumulated changes and additions * Orphaned code * Commented out tests * etc etc
• huherto 9y ago
  The model that I've liked the most... - Monorepo for all the services code in the enterprise. (Java ) - UI code is kept in their own repositories. - Enterprise Services are exposed using well defined stable rest-like APIs. (json, http, swagger, etc). We only exposed what is needed. - Within the services monorepo, services can call other services directly using regular java calls. - Services in the monorepo are refactored all the time. This is the advantage of using a strongly typed language like Java. - Several instances of the services run are a time, they scale horizontally. Releases are done one instance at a time. - We had good unit and integration test suites on the services. Now I am working for a different company. We have hundreds of services deployed. No one knows what is running where. Or what the dependencies are. Once something is released, everybody is afraid to make a change as no one knows how that is affects other systems.