15 ms·

We Halved Go Monorepo CI Build Time

• pjmlp 4y ago
  Alternatively don't use monorepos, and take advantage of binary libraries. Naturally there is the caveat that using binary libraries isn't a thing in Go.
  • morelisp 4y ago
    Binary libraries are only relevant if a lot of time is spent compiling the library code. There's no data in the article that suggests this is the case.
  • taktoa 4y ago
    Monorepos are not mutually exclusive with binary dependencies.
  • masklinn 4y ago
    > Alternatively don't use monorepos And now you get into the fun situation of having to handle ecosystemic asynchronous updates between your multiple repositories.
    • viraptor 4y ago
      These are orthogonal concepts. If you don't have a monorepo, your CI can still build all downstream packages and redeploy changes on every update. If you have a monorepo, you can also ignore dependency changes and keep various services behind and in a broken-behaviour state if they'd get redeployed from main. Monorepo makes the sync updates slightly easier, but it's neither necessary nor forces it.
      • dymk 4y ago
        One commit in a monorepo now becomes N commits for every repo in every other repo
        • morelisp 4y ago
          Only if you need to update all N other projects every time, which is unusual. Our services aren't even particularly well-factored and we update median 2, mode 1, mean 3-4 when we change a core library. (Out of ~25.)
          • bobsomers 4y ago
            > Only if you need to update all N other projects every time, which is unusual. This isn't true in my experience at large companies. For this to be true, you would either need to have absolutely no shared core libraries, or core libraries that are so stable and unchanging that they never receive updates. Updates to those core libraries usually result in a cascade of changes across a ton of repos, tons of tests that need to be rerun, and a slew of now incompatible versions. All of these problems go away with a monorepo. Monorepos are absolutely the right solution for 98% of companies out there because they will never reach the engineering scale where monorepos start to struggle. For the 2% of companies that do reach that engineering scale, these kinds of blog posts get written. The mistake is reading blog posts like this and thinking is necessary or even relevant to any more than 2% of the companies out there. The vast majority of shops will never need anything like this.
            • morelisp 4y ago
              You seem to have ruled out the obvious choice of "downstream projects just don't update." We update the projects we need the new library behavior in. There's no reason to touch the other 20 until there's some new behavior they do need. I don't really see this as a mono vs. multi repo issue either. In a mono repo you can still choose which projects update and the more you choose the bigger the diff and more involved the review process. In a multi repo case you can build analogous tooling to generate downstream MRs automatically. If you need to touch every downstream project regularly just to keep things functional, you don't have a mono-repo, you have a monolith. (Which can also be fine, really! But it's not the same thing at all.) (Yes, I know with 25 projects we're nowhere near Uber scale. But based on my experience at larger companies, this seems to scale more or less linearly - in terms of downstream project count, library count, and developer time available to dedicate to such things. If anything, developer time available is what seems to scale super-linearly.)
    • spion 4y ago
      We know how to do that. Versioning, deprecations etc. We do that with all external dependencies.
      • masklinn 4y ago
        All of which adds more effort, more synchronisation, more delay, more processes, and more ceremony. > We do that with all external dependencies. Indeed. Why you’d want to also suffer that for internal dependencies when there is no reason to, I can’t fathom.
        • spion 4y ago
          I think this article clearly shows that the effort of handling a monorepo is way higher. If you have a good process for external dependencies you can apply that to internal. If you don't have a good process for external dependencies a monorepo won't help (unless you make all your dependencies internal, which is even more expensive)
          • rfoo 4y ago
            > If you have a good process for external dependencies you can apply that to internal. Yeah... but usually good process for external dependencies requires a lot of ceremony and takes maybe days to update one dependency. If the same happens to internal dependencies we will be just slower at making changes for no good reason. This is more of a cultural thing. In fact, we are creating complicated solutions to mitigate this problem, only to go full circle after a while: people implemented Service Mesh via proxy sidecars, it added more layers (than just embedding this into the RPC library), but even doing so is easier than persuading downstream project owners update their dependency in a timely manner, so it takes off. And now they are going to chase for "proxyless" due to "performance". Oh well.
            • spion 4y ago
              Why would a complex dependency update be faster in a monorepo? In my experience the biggest savings are for simple updates that happen acorss multiple repos - when the update itself is complex, the extra time to make multiple PRs is dwarfed by the actual update's complexity, and it helps if the update can be separated into several projects where the version can be bumped separately.
    • pjmlp 4y ago
      A much better scenario from my experience, given that not everyone is on the last version anyway.
    • elteto 4y ago
      I never understood the hand-waiving around multirepos. Parallel, distributed updates are extremely hard to get right in software, but, for some magical reason, not when checking in code across N separate repos with interdependencies.
  • jstimpfle 4y ago
    You can do both.
  • rfoo 4y ago
    I don't see how your suggestion helps in the "I changed standard library and there are 10000 downstream test to rerun" use case, which is what they blogged about. You either: * Just don't test such changes at change time, break down and push the testing effort to downstream projects in disguise of "dependency management". * Structure your engineering effort to avoid such changes, for example avoid having a base library shared between teams. Neither is ideal for a corporation environment, because both harms velocity. We accept these in the "open source world" because we don't have better options, but the same does not hold for corps.
  • sebastialonso 4y ago
    Quite common to hear comments like this, from people who have never worked at the scale in which the annoyances of monorepo are eclipsed by its advantages. I'm not saying anything particular to you, but it's quite hard to get to that scale in the first place, say global top 50 in big tech, so it's natural most folk can't understand why X company "still" runs monorepos. At that scale, anything reducing complexity is a bonus. At that scale, monorepos are great for that.
    • pjmlp 4y ago
      Well, in those 30 years, I am certain that some well known international medical research laboratories, mobile operators and nuclear physics research centers fit the bill.
• bogdanu 4y ago
  Phabricator is no longer maintained, from what I remember and is used most of the time with svn. I 'm curious why they opted for it in the first place, instead of a more modern platform.
  • boberoni 4y ago
    Not an Uber engineer, but I worked at another company with a lot of ex-facebook senior engineers in the founding team. They were already familiar with the Phabricator workflow, so they stuck to what they knew. My guess is that the early Uber team also included a lot of ex-facebook engineers. I was new to Phabricator, but I picked it up quickly. Even though it might not constantly pump out new features, Phabricator gets the job done.
    • bogdanu 4y ago
      I'm currently working with Phabricator too, but in our case it was chosen a long time ago, before other platforms were popular
  • pojzon 4y ago
    This sounds more and more like: „We solved an issue we created ourselves”. Which sometimes is ofcourse what you have to do.
  • charcircuit 4y ago
    The open source version of phabricator is no longer maintained, but that only happened 1 year ago. Facebook's internal phabricator is still maintained and there is phorge which is attempting to fork and maintain the open source version of phabricator. Uber may opt to just maintain phabricator themselves. >is used most of the time with svn It supports mercurial, git, and svn. >I 'm curious why they opted for it in the first place, instead of a more modern platform. It is a modern platform and works well.
    • jesboat 4y ago
      Facebook's "phabricator" is very different from OSS, to the extent that at this point probably most of what's left is terminology and architectural boundaries. The diff (PR) UI was completely rewritten; I think repo browsing too. Most of the storage got refactored to use common storage code.
• deleted 4y ago
  [deleted]
• Galanwe 4y ago
  TLDR: How company wide monorepo/monodeployment multiplied by 10 the CI times, but we managed to halve that.
  • deleted 4y ago
    [deleted]
  • deleted 4y ago
    [deleted]
  • ZeKK14 4y ago
    Well, it's not that easy. Let's say you have this layout : - applications: front, backend, background-worker - libs: database-orm In a multi-repo layout, if you want to make a change in database-orm, you'll make your PR in its repo, test, and make a release with your changes. Nice and easy right ? Well, you're not done. Now you have to make a PR to update the dependency on each repo using this library. If you're lucky, nothing breaks and it's quite quick. But it's not always so easy : you notice that you actually broke something down the line in the backend. You have to fix it (in your library), and do it all over again. You can also have libraries depending on other libraries, multiplying the effort when you messed up something. The monorepo handles that, you update one library and you can see what you broke down the line, and fix all of that quicker. Also, changes are easier to follow since one modification impacting several applications or libraries can be made into only one commit. You can tell me that libraries should have a nice definition and the applications should be independent from the actual internals, but that's rarely the case. It's a tradeoff and lots of companies are going this route.
    • morelisp 4y ago
      > But it's not always so easy : you notice that you actually broke something down the line in the backend. You have to fix it (in your library), and do it all over again. You can also have libraries depending on other libraries, multiplying the effort when you messed up something. Since this is Go, it's trivial to point in-development branches of downstream projects to an in-development branch of a dependency before merging it for those projects' testing processes. (This still doesn't cover 100% of cases of course, but nothing does. The better answer is not to over-factor in the first place.)
• progbits 4y ago
  > CTC creates a Merkle-style tree... You call it Merkle-style tree despite the fact it is obviously a DAG? -- Edit: > Large changes are also more prone to outages. If we land them outside the working hours, there would be limited resources to mitigate potential outages. To prevent this, we ended up asking engineers to get up early to deploy these changes Huh, landing changes to HEAD and releasing to production are not decoupled?
  • fnord123 4y ago
    > You call it Merkle-style tree despite the fact it is obviously a DAG? All trees are DAGs so that's not really adding much.
    • progbits 4y ago
      But not all DAGs are trees and the one in the picture (and generally all graphs of dependencies) isn't.
      • nl 4y ago
        It's a multirooted tree. And describing it as a Merkle-style tree conveys additional implementation about the implementation - presumably the tree stores the hash ("computed from all of its source files and inputs") which just saying "it's a DAG" doesn't tell you.
        • dymk 4y ago
          Calling it a Merkle style DAG would have solved the issue. Or just avoid the fancy names and call it what it is: they walk the dependency graph to identify build targets that have a changed dependency.
  • ZephyrBlu 4y ago
    DAG doesn't seem like a useful identifier here since seems like a DAG encompasses many different types of trees, and in this case the specific data structure is closer to a Merkle tree than a generic DAG.
• Shadonototra 4y ago
  Build time has always been essential, even more so important since the wide adoption of TDD Picking a language that has fast build speed, allows you to be more flexible, and able to test and deploy quicker They had Go, wich builds very fast, but they failed to educate their developers to maintain a healthy build pipeline to avoid things getting too slow This kind of developer will cost your company millions, make sure to educate them properly ;) I'd never work for a company with slow build speed, part of the reason why i refuse to touch languages like Rust and C++, due to their insanely slow build speed, i refuse to live in a world like that
  • sethammons 4y ago
    Recently switched from a Go shop to Elixir. Ooof, the compilation feels like molasses.
    • sodapopcan 4y ago
      Have run xref at all? I also work at an Elixir shop with excruciating compile times. There are many files where if you save them, 80+ others are recompiled. I was able to fix a ton of them but they just crawled back up again. There’s an underlying issue that we’re still working on but I can with high confidence that the culprit is cyclical deps.
  • globalreset 4y ago
• deltaci 4y ago
  It's amazing to see big company can throw so much engineering effort into it, while for majority of the CI users, just getting a 2x faster CI machine can achieve the same outcome with much less cost. [0] https://buildjet.com/for-github-actions/blog/a-performance-review-of-github-actions-the-cost-of-slow-hardware https://buildjet.com/for-github-actions/blog/a-performance-r... edited: wrong link
  • draw_down 4y ago
  • moreira 4y ago
    I've been thinking about this for a while, and it seems to apply to a lot of things; serving a HTTP request on cheap EC2 instances won't come close to doing it on a dedicated server with great single-thread performance. So even though you can more easily horizontally scale and handle infinite requests, the latency of each request will be much poorer than if you were just running on better hardware.
  • shawabawa3 4y ago
    Speaking from experience working on CI at a large company, I'm sure they've "just got a 2x CI machine" about 6 times. At some point you can't just burn more money and you need to optimise
    • tootie 4y ago
      Computers are usually way cheaper than people. The difference would just be Cap Ex vs Op Ex. That being said, they must be burning zillions of cycles rebuilding code that hasn't changed by using a monorepo.
      • crummy 4y ago
        Maybe I misread the article, but I thought they used tools to ensure they only rebuilt parts of the dependency graph, not wasting zillions of cycles?
• morelisp 4y ago
  This is a poor article with little applicability outside of Uber. There's a reason it's gotten no comments the previous three times it was posted.
  • tankenmate 4y ago
    I would disagree; they showed the problem they had, how they analysed the data to understand the problem, how they used that data to identify possible solutions, how they selected the solution, and how they implemented an tested the solution. There are other organisations apart from Uber that have similar issues (although they are probably small in number). But there is a lot to learn from understanding their approach. Having said that it is much harder to learn from others' mistakes than your own. Learning from your own success is hard too. Learning from others' success is the hardest by far. At least we have a glimpse (albeit indirectly) in this article of the mistakes they made.
    • tankenmate 4y ago
      I guess I should also say, if you're looking for a practical how to document, this is not that (nor does it need to be).
• raverbashing 4y ago
  Is it just me or most of those "monorepo" articles are about how the company switches to monorepo then just has to fight all the problems that come up with it? Use multiple repos. It doesn't hurt you know. And you can have proper granularity The costs of monorepo seem to be much higher than just dealing with inter-dependencies. And you can always group projects if it makes sense
  • viraptor 4y ago
    Obviously the articles will be about solving the problems. Nobody wants to write a "yep, the thing that worked still works well for like 99% of cases (we solved the 1% in other ways)". This of course produces bias for people reading the posts, because it's not trivial to tell if the solved problem was about to bring the company down or was just an annoyance that they solved in a cool way they want to share.
  • detaro 4y ago
    Not like there aren't plenty problems with (and articles about) working with multiple repos. Neither is a silver bullet, you just get to pick which set of problems you prefer to have to solve (or which ones you think you can get solved due to human factors)
  • CJefferson 4y ago
    I've never seen an article from a large company saying "we use multiple repos and it's great". Setting as all large companies seem to end up monorepo, I'm guessing in practice it is better. I'd be happy to see a counter-example, but I've not seen one yet.
    • raverbashing 4y ago
      Nobody is going to write a blogpost "we use multiple repos and it's ok for us". And that is the majority of companies. Heck, even most individual developers use multiple repos. Monorepos are usually ok (and even talking about team/project granularity) until you outgrow it. Buildsystems like one-repo-per-project much better than monorepos. (Shipping a fix doesn't require rebuilding "the whole world" for example)
• alexhornby 4y ago
  Why the described CTC algo instead of querying bazel graph to deduce targets to build given changed files? e.g. https://github.com/bazel-contrib/target-determinator https://github.com/bazel-contrib/target-determinator
  • IshKebab 4y ago
    Yeah I was wondering that. Isn't half the point of Bazel that is can know things like this without any extra effort?
    • esprehn 4y ago
      That is what they do, click through the link in the article: https://github.com/bazelbuild/bazel/issues/7962#issuecomment-553042506 https://github.com/bazelbuild/bazel/issues/7962#issuecomment...
• hiyer 4y ago
  It seems to me that an easier way to avoid this problem is to not have monorepos in the first place. Uber's architecture is, AFAIK, heavily based on microservices, so I wonder what's the advantage they get in using a monorepo.
  • de_keyboard 4y ago
    Monorepos have the big advantage that cross-project changes can be made together in one PR.
    • bottled_poe 4y ago
      This is the correct answer. Putting all integration changes in a single PR improves CI test coverage and is easier to verify in a code review.
      • XorNot 4y ago
        This really begs the question why Github aren't working on cross-repo PR support - or anyone. Everytime mono-repos come up, cross-repo PRs are the reason given as to why they're needed - why can't we get this supported well?
        • mattnewton 4y ago
          That’s not just a GitHub issue, that would require serious UX work on the authoring side to make such “commits” that are actually commits in multiple repos at the same time. Effectively this is recreating a monorepo (maybe with something like for submodules?)
          • treis 4y ago
            I don't think it's recreating a monorepo. What you're looking for is something like a DB transaction where all the merges in different repos succeed or everything gets rolled back.
• viraptor 4y ago
  Some alternative solutions to similar issues: Zuul (https://zuul-ci.org/ https://zuul-ci.org/) was created for openstack to solve the issue of optimistic merges / PR queue testing. When you use buildkite with own containers on AWS ecs you can use efs to do a git clone with reference. (https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---reference-if-ableltrepositorygt https://git-scm.com/docs/git-clone#Documentation/git-clone.t...) Essentially what they do with a packed base repo, but you only end up sending what you need, not more. The binary cache is available in other flavours too. If you don't use go, then sccache (https://github.com/mozilla/sccache https://github.com/mozilla/sccache) may be useful.
• likortera 4y ago
  If you use monorepos and at the same time you use microservices, what you are doing is called a distributed monolith.
  • slively 4y ago
    This is a common mis-conception that the application design and/or deployment is tied to the repo or repos. They are completely orthogonal. Micro services can be done in mono or multi repo. A distributed monolith can be done in mono or multi repo. Either can also be done without version control.
    • likortera 4y ago
      I'm saying using BOTH monorepo AND microservices is a clear signal you're building a distributed monolith. What you say is also true
      • slively 4y ago
        Ah I see, I could definitely see that being a useful heuristic. Also it seems like the origins of micro services comes from places like Google that use a mono repo. The proof is in the pudding in the end I suppose.
    • preseinger 4y ago
      Microservices are meant to decouple teams at both the deployment (runtime) level _and_ the source (repo) level. Microservices + monorepo is at least at some level self-subversive.
      • sebastialonso 4y ago
        Agree with the spirit of this observation, but at the scale we work with at Uber, at comes a point in which "too much" decoupling starts biting you in the ass. Sounds kind of bizarre actually, but this is a particular characteristic of a microservice architecture after it's reached a certain amount of elements in the network and certain amount of complexity. I can't imagine deploying a functionality that needs to interact with 20 other services, if I can't have some zero-cost assurances like Go's typing and having all IDL available at developing time. This is of course is even better if at compile time I can check all contracts are still honoured. Monorepo is a really "cheap" way of getting all this for, close to, free.
• icambron 4y ago
  Putting aside questions of monorepos, I am always astounded by how much big companies invest in their build/change management/review systems. Shipping becomes a whole product in itself. This appears to be a serious piece of engineering, but it’s directed at Uber’s tooling, not Uber’s business problems or even Uber’s product. I’m sure it’s fascinating for people who are into that, and I’m not saying there isn’t a substantive problem to solve here, but I’d personally never go anywhere near a problem like this. It’s too many steps removed from the value. I’d keep thinking “we are down a very deep rabbit hole. A side quest off a side quest. Should we be doing this?” But I’ve always worked at smaller places, where things like toolchains and sales ops and marketing tech don’t take on huge lives of their own.
  • caust1c 4y ago
    Personally, Uber doesn't come off to me as an engineering culture that values simplicity. And in fact, in this very article, the last paragraph actually states that they are aiming to increase the complexity of the change validation process?!? > Additionally, it helped our team focus on increasing the complexity and features of our change validation process. Personally, I haven't been super impressed with any of the engineering projects I've heard come out of Uber, but what they're doing obviously works to some degree so I can't be one to judge.
    • speedgoose 4y ago
      Deck.gl is a fine map visualisation library.
  • stingraycharles 4y ago
    I’d say that this is just internal platform / infrastructure support, aimed at making the whole organisation be more effective. It’s absolutely adding a lot of value to the organisation, it’s just not visible from the outside. But saying that “it’s too many steps removed from the value” appears to be a misrepresentation of the value it adds.
  • MPSimmons 4y ago
    Per DORA and the information in Accelerate, shipping code frequently and reliably contributes directly to good outcomes. It seems like a good investment.
• setheron 4y ago
  Doesn't bazel already calculate a minimum build set based on changed files?
• umvi 4y ago
  How do go monorepos work? Does each library/binary have its own directory with its own go.mod file? And then you need a build system more powerful than go build to descend into each directory and build (if necessary)? Does this play nice with gopls (go language server) allowing you to jump around to definitions?
  • dilyevsky 4y ago
    Afaik Uber uses Bazel to build everything so it’s not standard go toolchain but it’s using parts of it. And yes as of recently it decently integrates with gopls https://github.com/bazelbuild/rules_go/wiki/Editor-setup https://github.com/bazelbuild/rules_go/wiki/Editor-setup
  • doktorhladnjak 4y ago
    Monorepos have one go.mod file. Bazel or similar build systems optimized for monorepos usually are used.
• esprehn 4y ago
  Most of what this article seems to be saying is that Uber wasn't using all the features of bazel (and their setup also predates some of them), and as they've started to use them things are getting a lot faster. Prior to doing so they were hacking around it like controlling when folks could land large changes. The rest of the article is about their transition from their legacy infra like Jenkins to more modern and flexible things like buildkite (which is of course relevant for anyone in a similar transition). If your company/project starts with bazel + remote cache + RBE you're pretty close to the end state they're trying to get to. See also for OSS and paid solutions: https://bazel.build/community/remote-execution-services https://bazel.build/community/remote-execution-services
• SergeAx 4y ago
  If you read a history of Uber getting to monorepos [1], [2], it is obviously that they did it as an attempt to make their shared spaghetti code a bit more structured. Now they are writing articles about fighting a consequences of that move. For me the moral is: never go monorepo. It is at least counterproductive, but more like destructive even for large capable teams. [1] https://eng.uber.com/ios-monorepo/ https://eng.uber.com/ios-monorepo/ [2] https://eng.uber.com/android-engineering-code-monorepo/ https://eng.uber.com/android-engineering-code-monorepo/