10 ms·

It seems to me that an easier way to avoid this problem is to not have monorepos in the first place. Uber's architecture is, AFAIK, heavily based on microservic

by hiyer 4y ago

It seems to me that an easier way to avoid this problem is to not have monorepos in the first place. Uber's architecture is, AFAIK, heavily based on microservices, so I wonder what's the advantage they get in using a monorepo.

de_keyboard 4y ago
Monorepos have the big advantage that cross-project changes can be made together in one PR.
- bottled_poe 4y ago
  This is the correct answer. Putting all integration changes in a single PR improves CI test coverage and is easier to verify in a code review.
  - XorNot 4y ago
    This really begs the question why Github aren't working on cross-repo PR support - or anyone. Everytime mono-repos come up, cross-repo PRs are the reason given as to why they're needed - why can't we get this supported well?
    - mattnewton 4y ago
      That’s not just a GitHub issue, that would require serious UX work on the authoring side to make such “commits” that are actually commits in multiple repos at the same time. Effectively this is recreating a monorepo (maybe with something like for submodules?)
      - treis 4y ago
        I don't think it's recreating a monorepo. What you're looking for is something like a DB transaction where all the merges in different repos succeed or everything gets rolled back.
        mattnewton 4y ago
        Maybe I am missing something: to do that you need some kind of “meta-repository” linking commits between all the different sub repos in some order, no? If that happens for a significant portion of the commits in a repo, isn’t that basically equivalent to a mono repo with git submodules or something?
        treis 4y ago
        Well a meta repo isn't a mono repo. In that situation you can still update individual repos. Analogous to running a DB operatioj without a transaction. But that said, no I don't think you need a meta repo. You need something. Call it a Change set. MVP would be say merge commits in 4 repos. If any fail then revert. If they all succeed then deploy in a specified order.
    - dalyons 4y ago
      Sourcegraph does a decent job of this with their “batch changes” feature
- BenoitEssiambre 4y ago
  It also helps keep people in sync with module and library versions. Without a monorepo, oftentimes what happens is that everybody ends up on different versions of libraries and modules. For every security update to a library, you'll have to campaign for everyone to pull that fix instead of everyone getting all fixes on every pull. It also forces dealing with incompatibilities between libraries early on when they happen instead of later when they are more entrenched and difficult to fix.
  - spion 4y ago
    That can be handled with alerts and deprecations just like any other library dependencies
    - idontknowifican 4y ago
      no it can’t, an alert won’t make a PM allocate time to “tech debt” and these alerts are rarely time free upgrades
      - spion 4y ago
        If you have an issue handling tech debt due to PMs, you'll have that problem with other tech debt too; typically solved by having a better process to handle it
  - deepsun 4y ago
    What I've seen in practice in smaller companies, is actually the other way around: everyone is afraid updating any library version because who knows what other team's work it might break. Theoretically, everyone must have extensive tests to cover everything, but in practice it's hard. But when your team owns a repo, then at least the damage is contained within your team. Google solves that problem with heavy NIH syndrome (its hard to get promoted by utilizing an external third-party lib, better develop your own), and writing tests, yes. And for those few third-party libraries that google still depends on, updating them is a big PITA.
    - 4y ago
- spion 4y ago
  A well factored system should not have cross-project changes. Individual projects should be able to upgrade dependencies separately.
  - idontknowifican 4y ago
    can you provide more info here? i’m finding it hard to think of a system at a large company that would have no internal libraries used by multiple projects, that wouldn’t require ever making a cross project change.
    - spion 4y ago
      Changes will be handled just like external library updates.
      - idontknowifican 4y ago
        i suspect your internal model of a monorepo is fairly disjoint with an actual implementation. you’re arguing for non-monorepo based version concepts, without giving space to why having everyone locked to “latest” has large cost savings long term for maintenance and security. without those constraints, a monorepo is a bad choice.
        spion 4y ago
        I've worked with both models, and the monorepo model has been far more demanding in terms of time and resources to maintain as you push past the limits of the available tools one by one. Standard build tools go out of the window almost immediately (lots of work to wrap everything with Bazel), then standard collaboration tools (GH) and so on. A dedicated tooling team becomes a must. What helps maintain polyrepos sane 1. A healthy methodology for dependency management Evolve APIs with deprecations. Decide on a healthy amount of time for a deprecations to live. Set up alerts when deprecations reach certain amount of time. Set up dashboards (e.g. Grafana) to track dependencies. Help other teams update, and think about how to make updates less painful. 2. Define good boundaries and API contracts to adhere to. The most important thing for a good API contract is stability. 3. Don't prematurely split into microservices. Better ideas for stable API contracts emerge the longer you can wait.
- barrkel 4y ago
  This is almost the only advantage. A related "advantage" is forced upgrades when dependencies change - since you can't have code depending on different versions of in-repo libraries and services, client code needs to adapt or die. The more clients a library or service has, the more expensive this is, and it's an ongoing maintenance cost for every client. Since people changing the service don't feel the full pain of this maintenance cost, changes keep on happening, and eventually clients get culled because it's too expensive to keep on maintaining them all. From the outside, this looks like the company abandoning venerable but still working product, and makes people scratch their heads, wondering why.
  - ithkuil 4y ago
    You can have a monorepo without having all the modules share the dependencies. For example you can have multiple go.mod files in a single monorepo. People often conflate these two things because often teams actually want to have centralized dependencies (so that you're forced to update or die as you said). If that doesn't work for you you can choose to have modules (or groups of modules) keep their independent set of dependencies, all while keeping the code in the monorepo.
- prestoooooo 4y ago
  Pros * Breaking changes can be done at once. Very helpful for runtime deps. * No chance that a repo is out of date. * Upgrades are atomic (may be hard to test a system in a half state).
  - erik_seaberg 4y ago
    While you’re deploying, production is going to be in a half-upgraded state (or maybe half rolled-back!), so it’s pretty important to be able to test that.
    - iambvk 4y ago
      This depends on the deployment model. I am curious, how does your release and deployment looks like?
      - erik_seaberg 4y ago
        Release is a privileged tool pushing to Git. Deployment is incremental across a small but growing number of containers running a microservice, so both old and new code might be running concurrently for up to a couple of hours. Percentage experiments are pretty common, which means both old and new paths actually have to work in the same commit.
- likortera 4y ago
  Then why not call it just a repo and build a monolith? I've seen people going crazy with this bullshit of monorepos to the point every single directory was a "package" when it could be just a plain module import. If you want to do microservices, then putting everything back into a single repository and enforcing everyone to use the same version and every change to require upgrades and deploys across the board is totally backwards. Either do a monolith and have that consistency, or do microservices and allow teams to follow their own rules as long as they keep APIs stable. Nonorepos + microservices is just a demonstration of everything that's wrong in the technical aspects of our industry. Just applying absolutely everything you read about without even considering if it might be better or not for your specific use case.
  - oftenwrong 4y ago
    The way you store your source code doesn't have much to do with how your system is deployed. It's the same way you can have a closet in which you store clothing for many purposes; clothing for cold weather, clothing for hot weather, clothing for formal occasions, clothing for specific recreational activities, et cetera. You can store code for different purposes in the same source tree. You can deploy something that is built from just one part of the source tree.
    - skytreader 4y ago
      > You can store code for different purposes in the same source tree. You can deploy something that is built from just one part of the source tree. At the risk of sounding memetic, the question is not so much "Can you?" but "Should you?". Should you store a monolithic application across multiple repositories? Should you store a distributed application in a monolithic repository? I agree with @likortera that you should not. And that's because... > The way you store your source code doesn't have much to do with how your system is deployed. I disagree with this statement. Your architecture (monolithic vs distributed) imposes certain assumptions on other aspects of your distribution pipeline. Your workflow (in this discussion "how you store your source code") should support these assumptions, not hinder them. For example, one benefit of microservices is that cross-functional teams can develop independently of each other. And yet one cited advantage of monorepos is that everyone is on the same version of dependencies all the time. In short, your teams are not independent after all. Note that I'm keeping the example extremely generic to illustrate this inconsistency, a conflict of interest if you will, that I see people commit in this topic, because in my experience, these questions are not purely technical but involves product/business factors as well. Maybe for most of the people (operative emphasis on "MAYBE", because who am I to judge you), the discussion they need to have first is whether or not they are using the right architecture for their product in the first place. If you choose to have a microservices architecture, you have to live with the fact that your teams/services will operate at different cadences. If you feel the need to impose a One True Library Version All the Time, then go for a monolithic architecture, and store your code in the same way.
- kitd 4y ago
  The problem isn't per se monorepo v multi-repo, it's ensuring that your lines of communication between components, ie your APIs, are shared reliably and coherently. When teams take the stability & versioning of their APIs seriously, the need to use monorepos to share that info is greatly reduced. A multi-repo approach is perfectly feasible when all components are working to established APIs, which also alleviates the issues mentioned in the article.
- eikenberry 4y ago
  Large cross project changes mean you have a monolithic project. One project split over multiple repos, not multiple projects. If you're separate projects were really separate they'd maintain API compatibility and upgrade paths (eg. semantic versioning). SaaS and microservice architectures require this to maintain separation but most organizations lack the discipline and slowly revert back to a monolith as they are faster to develop (until they aren't).
chloerei 4y ago
MonolithFirst https://martinfowler.com/bliki/MonolithFirst.html https://martinfowler.com/bliki/MonolithFirst.html
- stitched2gethr 4y ago
  First yes, but it's not difficult to imagine Uber has outgrown their first architecture.
  - phyrex 4y ago
    Both google and Facebook have monorepos, and I think apple does too. I’m not sure about the others, but I wouldn’t be surprised if they did as well
    - bogwog 4y ago
      Google also has projects like this: https://github.com/google/breakpad https://github.com/google/breakpad It supports 5 platforms, but uses 4 completely different build systems, including 2 custom ones (3 if you count depot_tools). There is very little overlap between the platform versions, meaning it's effectively 5 different projects smashed together into a single folder, and pretty much no way to use them in a cross platform project without some serious work. There isn't even a basic abstraction over the similar callback APIs between the platforms, although that's not a huge deal because the effort to write a basic abstraction layer is nothing compared to the effort of getting to a point where you can actually use it in a cross-platform project. It's also funny that one of the build systems is GYP, which is basically a reinvention of CMake, except it's only used for the Windows build even though it can generate projects for the other platforms. Also, the VS project generator for GYP has been broken for a while (simple typo, trying to import OrderedDict from the wrong module. There's a PR to fix it, hasn't been merged for some reason), so it doesn't even work. Beyond that, it's also broken because GYP forces treating all warnings as errors, with a whitelist of warnings, yet the latest version (since yesterday at least) fails to build (tested on VS2019) because there's a warning that isn't in the whitelist. You could try to fork it and fix these issues, but depot_tools doesn't provide a way to change the clone URL for repos, meaning you need to dig through the source code and wrap it in your own script that interacts with the internal APIs to do a simple clone (hint: fetch.py has a 'run' method that you can call with a custom constructed 'spec' object, which is a dictionary where you can inject your own url; just look at the hard-coded spec object for breakpad as a starting point). If you don't use depot_tools, then you need to manually clone all of the dependencies in the project since they're not even set up as git submodules. There's also no versioning scheme whatsoever. Depot_tools seems to automatically checkout the latest version of everything (including itself). I spent the past week wrestling with this monstrosity. Ended up successfully writing a Conan package for it that builds for Windows and Linux (there's one on Conan center, but it only supports Linux). I have 3 more platforms to go, but I think it'll be a better idea to just scrap everything and refactor into something more reasonable using CMake. Instead of Breakpad, they also have a newer one called Crashpad, which is meant to improve reliability on Mac OS. Unfortunately, it depends on Chromium, so it won't work for my purposes. ...so all I'm saying is, maybe don't use Google as a role model for your project infrastructure. /end rant
dan-robertson 4y ago
They used to have many small repos according to https://eng.uber.com/go-monorepo-bazel/ https://eng.uber.com/go-monorepo-bazel/ so perhaps there are advantages that outweigh slow CI.
kmbfjr 4y ago
We’re going to have monorepos until everyone tires of solving the problems they didn’t have before switching to this decade’s shiny object for workflow.
masklinn 4y ago
> so I wonder what's the advantage they get in using a monorepo. Much easier to apply horizontal and vertical changes which extend outside individual services e.g. update the interface of a service, or fix a bad pattern.
thurn 4y ago
How does not having a monorepo solve this problem? They would still need to compile and run the same code in order to run these tests between 1000 small repositories vs 1 large one, presumably they see a lot of value in end-to-end tests. Is the suggestion just "an easier way to avoid this problem is to not have so many tests?"
- hiyer 4y ago
  If the change is specific to a single module, and the APIs exposed to other modules don't change (which can also be enforced with unit tests), you don't need to run the tests for every single repo for every change. Of course, one may argue that one could run module-specific tests in a monorepo as well. But that's not easy to do on a per-PR basis, so you just end up running the entire CI suite for every PR.
  - Psyonic 4y ago
    that’s not actually what happens at Uber. They use bazel for the go monorepo which can accurately determine what modules are affected by a given change and only runs those tests. They also use mostly homegrown CI tools alongside phabricator for code review, or at least they did while I was there.
  - thurn 4y ago
    So you'd stop running tests for all of the calling code? It sounds like you have a higher level of confidence than I do that non-API changes can never break any downstream repositories in subtle ways.
  - jayd16 4y ago
    I would argue it's impossible to know what you really changed without the integration tests. But also, why is it any different why are changes at a repo level all that much easier to track than, say, inspecting changed files and running tests according to what changed?
likortera 4y ago
More fancy words on somebody's CVs? I worked at a similarly large and almost as popular SF based company and I can assure you every innovation there was 100% motivated by somebody's desie to put.it in their CV. Nothing of what was made there made any sense for the business, it was just people playing with toys. And worst part was people introducing this stuff (such as the one pushing for Lerna and an "UI" monorepo would leave the company after 1 year leaving a mountain of tech debt behind. But hey, they had Lerna in their CV now.
- no_wizard 4y ago
  In some ways do you blame anyone? The industry has largely not optimized for retention by and large
Hendrikto 4y ago
The company I work for has recently transitioned from many small repos to a single monorepo. I can tell you that the development experience is MUCH better. 99% of issues required cross-repo coordination, which was a nightmare, and thing could easily get out of sync. With a monorepo, you don‘t need to think about which commits work with which other commits. One commit ID is a full description of every subcomponent.
- tootie 4y ago
  I don't really understand how that creates fewer bottlenecks. It may save communication overhead because every dev can spot integration issues on their own, but at the cost of monstrously complex and time-consuming build-and-test pipelines. The microservice approach just requires some kind of service contract and meaningful release versions. That's always been a very manageable cost in my experience.
  - yakkityyak 4y ago
    In my experience the Uber mono repo was a giant velocity killer. Go tooling just completely choked, and at the time they had tons of terrible hacks around go modules by having this weird system that required your mono repo to be in GOPATH, further breaking even more tooling. No clue if that’s fixed today but it soured the idea of monorepos for me, and that’s not incorporating how often submitqueue would go down.
  - ferdowsi 4y ago
    The sharing of service contracts and their shared dependencies is a difficult problem to solve by itself. There's companies trying to make money off of the problem (buf). Monorepos solve the problem out of the box.
  - sagarm 4y ago
    > just requires some kind of service contract and meaningful release versions. I've just switched to a company with many tiny repos and IMO it's a huge hassle. There's no automated integration testing, and manual integration testing is a huge pain to set up. I don't even know how I'd create integration tests that run as part of the CI. What version would I use? If you need to change both sides (very common), coordinating the commits and releases is very painful. The "monstrously complex" build-and-test pipelines are a significant cost, but the alternative is higher release failure rate and moving slower overall IMO.
maronato 4y ago
I worked at Uber. The main arguments that I kept hearing were: - easier to share and import common packages - proto and thrift files are kept close to services and clients are updated globally automatically - dependencies and go versions are managed globally and all services get the same security updates - standardized build processes make it easier to manage large deployments And honestly, after the initial repo download there were no visible downsides
liuliu 4y ago
The article clearly stated that they can commit a cross-cutting changes (a change that can affects thousands of binaries downstream) within 15 minutes. I haven't done any calculation on how long that kind of changes can be done with polyrepo setup. If they can do that, their story can only be better for single isolated changes. It seems from outside, Uber's monorepo development experience vastly better than many small startups with variety of development methodology (no matter monorepo, polyrepo, or continuous deployment, or waterfall development, or GitFlow (a type of waterfall)).
doktorhladnjak 4y ago
I worked at a company that went through this transition from thousands of microrepos to a monorepo. The main driver was application security. When everybody did their own thing in their own repo, rolling a new version of a library with a critical security fix was very expensive. It was impossible to complete in a reasonable amount of time, because you'd frequently find legacy services that were on some previous major version that required rewriting code or upgrading other libraries to implement the fix. A monorepo means every service is on the same set of versions of every dependency at a given point in time. It's much easier to reason about what is fixed, and move everything to a known good version.

by hiyer 4y ago

de_keyboard 4y ago

Monorepos have the big advantage that cross-project changes can be made together in one PR.

bottled_poe 4y ago
This is the correct answer. Putting all integration changes in a single PR improves CI test coverage and is easier to verify in a code review.
- XorNot 4y ago
  This really begs the question why Github aren't working on cross-repo PR support - or anyone. Everytime mono-repos come up, cross-repo PRs are the reason given as to why they're needed - why can't we get this supported well?
  - mattnewton 4y ago
    That’s not just a GitHub issue, that would require serious UX work on the authoring side to make such “commits” that are actually commits in multiple repos at the same time. Effectively this is recreating a monorepo (maybe with something like for submodules?)
    - treis 4y ago
      I don't think it's recreating a monorepo. What you're looking for is something like a DB transaction where all the merges in different repos succeed or everything gets rolled back.
      - mattnewton 4y ago
        Maybe I am missing something: to do that you need some kind of “meta-repository” linking commits between all the different sub repos in some order, no? If that happens for a significant portion of the commits in a repo, isn’t that basically equivalent to a mono repo with git submodules or something?
        treis 4y ago
        Well a meta repo isn't a mono repo. In that situation you can still update individual repos. Analogous to running a DB operatioj without a transaction. But that said, no I don't think you need a meta repo. You need something. Call it a Change set. MVP would be say merge commits in 4 repos. If any fail then revert. If they all succeed then deploy in a specified order.
  - dalyons 4y ago
    Sourcegraph does a decent job of this with their “batch changes” feature
BenoitEssiambre 4y ago
It also helps keep people in sync with module and library versions. Without a monorepo, oftentimes what happens is that everybody ends up on different versions of libraries and modules. For every security update to a library, you'll have to campaign for everyone to pull that fix instead of everyone getting all fixes on every pull. It also forces dealing with incompatibilities between libraries early on when they happen instead of later when they are more entrenched and difficult to fix.
- spion 4y ago
  That can be handled with alerts and deprecations just like any other library dependencies
  - idontknowifican 4y ago
    no it can’t, an alert won’t make a PM allocate time to “tech debt” and these alerts are rarely time free upgrades
    - spion 4y ago
      If you have an issue handling tech debt due to PMs, you'll have that problem with other tech debt too; typically solved by having a better process to handle it
- deepsun 4y ago
  What I've seen in practice in smaller companies, is actually the other way around: everyone is afraid updating any library version because who knows what other team's work it might break. Theoretically, everyone must have extensive tests to cover everything, but in practice it's hard. But when your team owns a repo, then at least the damage is contained within your team. Google solves that problem with heavy NIH syndrome (its hard to get promoted by utilizing an external third-party lib, better develop your own), and writing tests, yes. And for those few third-party libraries that google still depends on, updating them is a big PITA.
  - 4y ago
spion 4y ago
A well factored system should not have cross-project changes. Individual projects should be able to upgrade dependencies separately.
- idontknowifican 4y ago
  can you provide more info here? i’m finding it hard to think of a system at a large company that would have no internal libraries used by multiple projects, that wouldn’t require ever making a cross project change.
  - spion 4y ago
    Changes will be handled just like external library updates.
    - idontknowifican 4y ago
      i suspect your internal model of a monorepo is fairly disjoint with an actual implementation. you’re arguing for non-monorepo based version concepts, without giving space to why having everyone locked to “latest” has large cost savings long term for maintenance and security. without those constraints, a monorepo is a bad choice.
      - spion 4y ago
        I've worked with both models, and the monorepo model has been far more demanding in terms of time and resources to maintain as you push past the limits of the available tools one by one. Standard build tools go out of the window almost immediately (lots of work to wrap everything with Bazel), then standard collaboration tools (GH) and so on. A dedicated tooling team becomes a must. What helps maintain polyrepos sane 1. A healthy methodology for dependency management Evolve APIs with deprecations. Decide on a healthy amount of time for a deprecations to live. Set up alerts when deprecations reach certain amount of time. Set up dashboards (e.g. Grafana) to track dependencies. Help other teams update, and think about how to make updates less painful. 2. Define good boundaries and API contracts to adhere to. The most important thing for a good API contract is stability. 3. Don't prematurely split into microservices. Better ideas for stable API contracts emerge the longer you can wait.
barrkel 4y ago
This is almost the only advantage. A related "advantage" is forced upgrades when dependencies change - since you can't have code depending on different versions of in-repo libraries and services, client code needs to adapt or die. The more clients a library or service has, the more expensive this is, and it's an ongoing maintenance cost for every client. Since people changing the service don't feel the full pain of this maintenance cost, changes keep on happening, and eventually clients get culled because it's too expensive to keep on maintaining them all. From the outside, this looks like the company abandoning venerable but still working product, and makes people scratch their heads, wondering why.
- ithkuil 4y ago
  You can have a monorepo without having all the modules share the dependencies. For example you can have multiple go.mod files in a single monorepo. People often conflate these two things because often teams actually want to have centralized dependencies (so that you're forced to update or die as you said). If that doesn't work for you you can choose to have modules (or groups of modules) keep their independent set of dependencies, all while keeping the code in the monorepo.
prestoooooo 4y ago
Pros * Breaking changes can be done at once. Very helpful for runtime deps. * No chance that a repo is out of date. * Upgrades are atomic (may be hard to test a system in a half state).
- erik_seaberg 4y ago
  While you’re deploying, production is going to be in a half-upgraded state (or maybe half rolled-back!), so it’s pretty important to be able to test that.
  - iambvk 4y ago
    This depends on the deployment model. I am curious, how does your release and deployment looks like?
    - erik_seaberg 4y ago
      Release is a privileged tool pushing to Git. Deployment is incremental across a small but growing number of containers running a microservice, so both old and new code might be running concurrently for up to a couple of hours. Percentage experiments are pretty common, which means both old and new paths actually have to work in the same commit.
likortera 4y ago
Then why not call it just a repo and build a monolith? I've seen people going crazy with this bullshit of monorepos to the point every single directory was a "package" when it could be just a plain module import. If you want to do microservices, then putting everything back into a single repository and enforcing everyone to use the same version and every change to require upgrades and deploys across the board is totally backwards. Either do a monolith and have that consistency, or do microservices and allow teams to follow their own rules as long as they keep APIs stable. Nonorepos + microservices is just a demonstration of everything that's wrong in the technical aspects of our industry. Just applying absolutely everything you read about without even considering if it might be better or not for your specific use case.
- oftenwrong 4y ago
  The way you store your source code doesn't have much to do with how your system is deployed. It's the same way you can have a closet in which you store clothing for many purposes; clothing for cold weather, clothing for hot weather, clothing for formal occasions, clothing for specific recreational activities, et cetera. You can store code for different purposes in the same source tree. You can deploy something that is built from just one part of the source tree.
  - skytreader 4y ago
    > You can store code for different purposes in the same source tree. You can deploy something that is built from just one part of the source tree. At the risk of sounding memetic, the question is not so much "Can you?" but "Should you?". Should you store a monolithic application across multiple repositories? Should you store a distributed application in a monolithic repository? I agree with @likortera that you should not. And that's because... > The way you store your source code doesn't have much to do with how your system is deployed. I disagree with this statement. Your architecture (monolithic vs distributed) imposes certain assumptions on other aspects of your distribution pipeline. Your workflow (in this discussion "how you store your source code") should support these assumptions, not hinder them. For example, one benefit of microservices is that cross-functional teams can develop independently of each other. And yet one cited advantage of monorepos is that everyone is on the same version of dependencies all the time. In short, your teams are not independent after all. Note that I'm keeping the example extremely generic to illustrate this inconsistency, a conflict of interest if you will, that I see people commit in this topic, because in my experience, these questions are not purely technical but involves product/business factors as well. Maybe for most of the people (operative emphasis on "MAYBE", because who am I to judge you), the discussion they need to have first is whether or not they are using the right architecture for their product in the first place. If you choose to have a microservices architecture, you have to live with the fact that your teams/services will operate at different cadences. If you feel the need to impose a One True Library Version All the Time, then go for a monolithic architecture, and store your code in the same way.
kitd 4y ago
The problem isn't per se monorepo v multi-repo, it's ensuring that your lines of communication between components, ie your APIs, are shared reliably and coherently. When teams take the stability & versioning of their APIs seriously, the need to use monorepos to share that info is greatly reduced. A multi-repo approach is perfectly feasible when all components are working to established APIs, which also alleviates the issues mentioned in the article.
eikenberry 4y ago
Large cross project changes mean you have a monolithic project. One project split over multiple repos, not multiple projects. If you're separate projects were really separate they'd maintain API compatibility and upgrade paths (eg. semantic versioning). SaaS and microservice architectures require this to maintain separation but most organizations lack the discipline and slowly revert back to a monolith as they are faster to develop (until they aren't).

chloerei 4y ago

MonolithFirst https://martinfowler.com/bliki/MonolithFirst.html https://martinfowler.com/bliki/MonolithFirst.html

stitched2gethr 4y ago
First yes, but it's not difficult to imagine Uber has outgrown their first architecture.
- phyrex 4y ago
  Both google and Facebook have monorepos, and I think apple does too. I’m not sure about the others, but I wouldn’t be surprised if they did as well
  - bogwog 4y ago
    Google also has projects like this: https://github.com/google/breakpad https://github.com/google/breakpad It supports 5 platforms, but uses 4 completely different build systems, including 2 custom ones (3 if you count depot_tools). There is very little overlap between the platform versions, meaning it's effectively 5 different projects smashed together into a single folder, and pretty much no way to use them in a cross platform project without some serious work. There isn't even a basic abstraction over the similar callback APIs between the platforms, although that's not a huge deal because the effort to write a basic abstraction layer is nothing compared to the effort of getting to a point where you can actually use it in a cross-platform project. It's also funny that one of the build systems is GYP, which is basically a reinvention of CMake, except it's only used for the Windows build even though it can generate projects for the other platforms. Also, the VS project generator for GYP has been broken for a while (simple typo, trying to import OrderedDict from the wrong module. There's a PR to fix it, hasn't been merged for some reason), so it doesn't even work. Beyond that, it's also broken because GYP forces treating all warnings as errors, with a whitelist of warnings, yet the latest version (since yesterday at least) fails to build (tested on VS2019) because there's a warning that isn't in the whitelist. You could try to fork it and fix these issues, but depot_tools doesn't provide a way to change the clone URL for repos, meaning you need to dig through the source code and wrap it in your own script that interacts with the internal APIs to do a simple clone (hint: fetch.py has a 'run' method that you can call with a custom constructed 'spec' object, which is a dictionary where you can inject your own url; just look at the hard-coded spec object for breakpad as a starting point). If you don't use depot_tools, then you need to manually clone all of the dependencies in the project since they're not even set up as git submodules. There's also no versioning scheme whatsoever. Depot_tools seems to automatically checkout the latest version of everything (including itself). I spent the past week wrestling with this monstrosity. Ended up successfully writing a Conan package for it that builds for Windows and Linux (there's one on Conan center, but it only supports Linux). I have 3 more platforms to go, but I think it'll be a better idea to just scrap everything and refactor into something more reasonable using CMake. Instead of Breakpad, they also have a newer one called Crashpad, which is meant to improve reliability on Mac OS. Unfortunately, it depends on Chromium, so it won't work for my purposes. ...so all I'm saying is, maybe don't use Google as a role model for your project infrastructure. /end rant

dan-robertson 4y ago

They used to have many small repos according to https://eng.uber.com/go-monorepo-bazel/ https://eng.uber.com/go-monorepo-bazel/ so perhaps there are advantages that outweigh slow CI.

kmbfjr 4y ago

We’re going to have monorepos until everyone tires of solving the problems they didn’t have before switching to this decade’s shiny object for workflow.

masklinn 4y ago

> so I wonder what's the advantage they get in using a monorepo. Much easier to apply horizontal and vertical changes which extend outside individual services e.g. update the interface of a service, or fix a bad pattern.

thurn 4y ago

How does not having a monorepo solve this problem? They would still need to compile and run the same code in order to run these tests between 1000 small repositories vs 1 large one, presumably they see a lot of value in end-to-end tests. Is the suggestion just "an easier way to avoid this problem is to not have so many tests?"

hiyer 4y ago
If the change is specific to a single module, and the APIs exposed to other modules don't change (which can also be enforced with unit tests), you don't need to run the tests for every single repo for every change. Of course, one may argue that one could run module-specific tests in a monorepo as well. But that's not easy to do on a per-PR basis, so you just end up running the entire CI suite for every PR.
- Psyonic 4y ago
  that’s not actually what happens at Uber. They use bazel for the go monorepo which can accurately determine what modules are affected by a given change and only runs those tests. They also use mostly homegrown CI tools alongside phabricator for code review, or at least they did while I was there.
- thurn 4y ago
  So you'd stop running tests for all of the calling code? It sounds like you have a higher level of confidence than I do that non-API changes can never break any downstream repositories in subtle ways.
- jayd16 4y ago
  I would argue it's impossible to know what you really changed without the integration tests. But also, why is it any different why are changes at a repo level all that much easier to track than, say, inspecting changed files and running tests according to what changed?

likortera 4y ago

More fancy words on somebody's CVs? I worked at a similarly large and almost as popular SF based company and I can assure you every innovation there was 100% motivated by somebody's desie to put.it in their CV. Nothing of what was made there made any sense for the business, it was just people playing with toys. And worst part was people introducing this stuff (such as the one pushing for Lerna and an "UI" monorepo would leave the company after 1 year leaving a mountain of tech debt behind. But hey, they had Lerna in their CV now.

no_wizard 4y ago
In some ways do you blame anyone? The industry has largely not optimized for retention by and large

Hendrikto 4y ago

The company I work for has recently transitioned from many small repos to a single monorepo. I can tell you that the development experience is MUCH better. 99% of issues required cross-repo coordination, which was a nightmare, and thing could easily get out of sync. With a monorepo, you don‘t need to think about which commits work with which other commits. One commit ID is a full description of every subcomponent.

tootie 4y ago
I don't really understand how that creates fewer bottlenecks. It may save communication overhead because every dev can spot integration issues on their own, but at the cost of monstrously complex and time-consuming build-and-test pipelines. The microservice approach just requires some kind of service contract and meaningful release versions. That's always been a very manageable cost in my experience.
- yakkityyak 4y ago
  In my experience the Uber mono repo was a giant velocity killer. Go tooling just completely choked, and at the time they had tons of terrible hacks around go modules by having this weird system that required your mono repo to be in GOPATH, further breaking even more tooling. No clue if that’s fixed today but it soured the idea of monorepos for me, and that’s not incorporating how often submitqueue would go down.
- ferdowsi 4y ago
  The sharing of service contracts and their shared dependencies is a difficult problem to solve by itself. There's companies trying to make money off of the problem (buf). Monorepos solve the problem out of the box.
- sagarm 4y ago
  > just requires some kind of service contract and meaningful release versions. I've just switched to a company with many tiny repos and IMO it's a huge hassle. There's no automated integration testing, and manual integration testing is a huge pain to set up. I don't even know how I'd create integration tests that run as part of the CI. What version would I use? If you need to change both sides (very common), coordinating the commits and releases is very painful. The "monstrously complex" build-and-test pipelines are a significant cost, but the alternative is higher release failure rate and moving slower overall IMO.

maronato 4y ago

I worked at Uber. The main arguments that I kept hearing were: - easier to share and import common packages - proto and thrift files are kept close to services and clients are updated globally automatically - dependencies and go versions are managed globally and all services get the same security updates - standardized build processes make it easier to manage large deployments And honestly, after the initial repo download there were no visible downsides

liuliu 4y ago

The article clearly stated that they can commit a cross-cutting changes (a change that can affects thousands of binaries downstream) within 15 minutes. I haven't done any calculation on how long that kind of changes can be done with polyrepo setup. If they can do that, their story can only be better for single isolated changes. It seems from outside, Uber's monorepo development experience vastly better than many small startups with variety of development methodology (no matter monorepo, polyrepo, or continuous deployment, or waterfall development, or GitFlow (a type of waterfall)).

doktorhladnjak 4y ago

I worked at a company that went through this transition from thousands of microrepos to a monorepo. The main driver was application security. When everybody did their own thing in their own repo, rolling a new version of a library with a critical security fix was very expensive. It was impossible to complete in a reasonable amount of time, because you'd frequently find legacy services that were on some previous major version that required rewriting code or upgrading other libraries to implement the fix. A monorepo means every service is on the same set of versions of every dependency at a given point in time. It's much easier to reason about what is fixed, and move everything to a known good version.