4 ms·

I know some of the FAANGs do monorepo (Google being the biggest) but AWS does not. A monorepo is an organizational mess when trying to manage and transfer owne

I know some of the FAANGs do monorepo (Google being the biggest) but AWS does not. A monorepo is an organizational mess when trying to manage and transfer ownership across thousands of teams, contain the blast radius of changes, unless you invest a ton of resources into proprietary tooling that requires a bunch of maintenance, since all the open source solutions are terrible at this and the whole data model is built around splitting out individual project repositories. And then after all that effort, why wouldn’t you just use tooling the way it was intended, and the way it’s used in the open source model, so you can partition your CI/CD without a bunch of hacks, and don’t run into bizarre scaling issues with your VCS. It perplexes me people advocate for this strategy. All I can think is it’s another one of those cargo-cult ideas that everyone is doing because Google did it (So it must be good).

jvolkman 5y ago
Let's not pretend that Amazon hasn't spent significant effort over decades building and maintaining their own (non-monorepo) build systems and tooling.
radicality 5y ago
* Why is ownership management a mess in a monorepo? Can just decide “this team owns this folder hierarchy” etc. * ‘Contain blast radius of changes’ - is that actually difficult? Isn’t there tooling that figures out what changed and what dependencies need rebuilding? (eg Facebook buck) For context, I was for a very long time at FB so am definitely used to the monorepo way, and recently switched to place which uses github + many repos, and it feels so much worse. Honest question - how do you actually effectively share code between many repos? Example: How do I know that me changing my backend app’s API doesn’t break any other project in the company potentially calling it? It should be a compile/buildtime error for the other project, but how does that work if everything is in its own little repository?
- giaour 5y ago
  > Honest question - how do you actually effectively share code between many repos? Locally, you can use an Amazon-internal tool to check out multiple repos and make changes to all of them. The tooling calls this a "workspace," but it feels very much like working in a monorepo since building and testing can happen at the workspace level. > How do I know that me changing my backend app’s API doesn’t break any other project in the company potentially calling it? In terms of change management, Amazon dependency graphs are managed as "version sets." Changes have to be built into a given version set, and that build will also rebuild any package in the version set that consumes the repository whose changes are being built in. (Usually, repositories are configured to build into one of the owning team's version sets on each commit to the primary branch.)
- dragonwriter 5y ago
  > Honest question - how do you actually effectively share code between many repos? One way is: Each repo is a responsibility boundary and single source of truth, you use code from other repos the same as any other external dependency. > How do I know that me changing my backend app’s API doesn’t break any other project in the company potentially calling it? Changing an API breaks projects using it; you either do versioned APIs and/or coordinate changes with downstream consumers, the same as you would with an API with external customers. (Another way is “downstream projects checkout their dependencies and build against them as a routine part of their process.“)
  - sagarm 5y ago
    > Changing an API breaks projects using it; you either do versioned APIs and/or coordinate changes with downstream consumers, the same as you would with an API with external customers. This is a ton of work relative to building/running the tests for all your reverse dependencies and fixing the call sites for them (up to a certain level of scale, of course). > (Another way is “downstream projects checkout their dependencies and build against them as a routine part of their process.“) This happens automatically in a monorepo. Any breakages are revealed as soon as upstream makes the change.
    - dragonwriter 5y ago
      > This is a ton of work relative to building/running the tests for all your reverse dependencies and fixing the call sites for them If someone outside of the team responsible for the functionality at issue can fix all the fall sites for a breaking API change without adversely effecting the domain function of the component making the calls, it's a pretty good sign that there wasn't actually a need for a breaking API change in the first place, and a process change that makes unnecessary API churn more expensive is probably beneficial.
      - giaour 5y ago
        There are some cases where you have to make a breaking change, and in those cases, it is helpful to be able to assess the fallout without a ton of manual effort. E.g., sometimes a security hole can only be remedied by restricting certain inputs (like with Heartbleed), and that's technically a breaking change. Monorepos let you make that assessment within a certain scope, as does Amazon's internal build system. It's not a feature you want exercised regularly, but it sure is helpful when you need it.
ignoramous 5y ago
> It perplexes me people advocate for this strategy. All I can think is it's another one of those cargo-cult ideas that everyone is doing because Google did it (So it must be good). Not sure if it is a generic comment or a comment on TFA: i) If the latter, I'm compelled to point out that TFA doesn't nearly advocate for monorepos as much as it lists reasons why a few SV companies use it, how they use it, and what they get out of it. ii) If the former, then this blog post makes for a good read: https://tailscale.com/blog/modules-monoliths-and-microservices/ https://tailscale.com/blog/modules-monoliths-and-microservic...
giaour 5y ago
Amazon has Brazil versionsets and workspaces, which solve many of the same problems monorepos do. I really liked the way those extra resources let you organize "ad-hoc monorepos" from smaller repositories, but it's infrastructure that I haven't seen elsewhere. Since leaving Amazon, I've mostly worked with monorepos and wouldn't go back to multirepos without Brazil-style tooling.
- dastbe 5y ago
  very much this. there's also a cultural understanding that libraries/fat clients are equal or worse in terms of maintainability compared to services. equal in the sense that you have to treat them like a service, worse because it's easier to mess up and you don't get to control your rollout strategy. EDIT: Though while most teams have gone towards many smaller packages for their applications, I suspect that most would be better served by a team level monorepo. That gets you all of the benefits of monorepo locally and all of the benefits of manyrepo globally, and unless your project hits the ~200+ developer mark maintaining things will be stay tractable.
- ghoward 5y ago
  Hey, I'm hoping you can answer some questions for me. I'm building a build system and a VCS (separately). I want to do it right. Could you explain to me what Brazil is? Is it the build system? [1] Or is it the VCS that Amazon uses? If it is the build system, then it appears that versionsets are literally just a list of dependencies with their versions to use for a build. Is that correct? If not, or if you can give me more detail, what are versionsets, exactly? Also, what are workspaces? Does this quote from one of the comments on the link match? > A workspace consisted of a version set to track and any packages that were checked out. [1]: https://gist.github.com/terabyte/15a2d3d407285b8b5a0a7964dd6283b0 https://gist.github.com/terabyte/15a2d3d407285b8b5a0a7964dd6...
  - giaour 5y ago
    I am not an expert on Brazil, but the link you shared matches my recollection. It's important to remember that Brazil covers a lot of ground, and many internal tools at Amazon are external to Brazil but rely on it and the way it organizes resources. So I (or others on the internet) may incorrectly call something Brazil if it's part of the larger Brazil ecosystem. Let's start with a repository and build outwards: You have some code (let's say it's a Java library) that does something cool. To compile it, you add a Brazil file to the repository root. This file specifies what Brazil packages your code depends on, how it should be built, and what kind of artifact it will produce. Once that file is there, you can run "brazil-build" to produce a Brazil package (which is just a jar with some metadata). You want to use this library in a web service, so you check out both repositories in a single workspace. Every workspace has a source versionset where it fetches dependencies, but if the code repository for a package is checked out locally, "brazil-build" will build and use the local version instead. You make some changes to the library and web service, then test how they work together by running the web service from within the workspace folder. This ensures that it is using your local modifications to the library repository before those changes have been merged. Once you're satisfied with the change, you open a PR with a brazil-integrated tool that can show changes to multiple repositories as an atomic change (a "change set"). The CI system for this tool uses a Brazil workspace to make sure your update code packages build together and that all tests pass. If the PR is approved and you merge to main, there is probably a pipeline watching your repository for changes that proactively rebuilds one or more version sets based on the merged change set. Any package in the version set that depends on your library will be rebuilt using the changes from your change set. So while a versionset is implemented as a list of packages at specific versions, it's best to think of a versionset as more or less equivalent to a monorepo containing all packages on that list, since changes that cross package lines can be built into a versionset in an atomic unit. (This is very helpful if you need to push out a breaking change.) A package can exist in multiple versionsets, which is of course impossible with monorepos.
throwaway894345 5y ago
Not having to submit and coordinate PRs across a dozen repos is a pretty tangible benefit. > unless you invest a ton of resources into proprietary tooling that requires a bunch of maintenance, since all the open source solutions are terrible at this and the whole data model is built around splitting out individual project repositories. Agree that there's a bunch of tooling needed to operate a monorepo, but there's also a bunch of tooling to sanely manage dozens of "microrepos" as well (when an upstream library changes, update downstream libraries' dependency manifest files to take the new version, run tests, report errors back to upstream, etc). I don't know of any open source tools that manage this problem, but I'm guessing they aren't high-quality if only due to the complex nature of the problem space. > And then after all that effort, why wouldn’t you just use tooling the way it was intended, and the way it’s used in the open source model, so you can partition your CI/CD without a bunch of hacks, and don’t run into bizarre scaling issues with your VCS. Because the tooling sucks, as previously mentioned. Many changes require touching many repos, which means coordinating many pull requests and manually changing dependency manifest files and so on. Ultimately, the "repo" concept is limiting. We need tooling that is aware of dependencies one way or another, and sadly all such open source tooling sucks whether it assumes the relevant code lives in a single repo or across many repos.
- erik_seaberg 5y ago
  > when an upstream library changes, update downstream libraries' dependency manifest files This needs to happen periodically, when we have slack. Doing it continuously adds risks that aren’t really our job to take.
  - throwaway894345 5y ago
    In my experience, if it doesn't happen continuously, it simply doesn't happen at all until something breaks (and then there's a bunch of finger-pointing at upstream even though downstream didn't update). Your first line of defense is that downstream tests are run before anything that affects downstream is merged. The next line of defense is stuff like canary deployments which allow you to minimize blast radius and roll back quickly. Obviously this depends a great deal on your risk regime--if you're SaaS this is probably fine, but if you're embedded this is a non-starter.

jvolkman 5y ago

Let's not pretend that Amazon hasn't spent significant effort over decades building and maintaining their own (non-monorepo) build systems and tooling.

radicality 5y ago

* Why is ownership management a mess in a monorepo? Can just decide “this team owns this folder hierarchy” etc. * ‘Contain blast radius of changes’ - is that actually difficult? Isn’t there tooling that figures out what changed and what dependencies need rebuilding? (eg Facebook buck) For context, I was for a very long time at FB so am definitely used to the monorepo way, and recently switched to place which uses github + many repos, and it feels so much worse. Honest question - how do you actually effectively share code between many repos? Example: How do I know that me changing my backend app’s API doesn’t break any other project in the company potentially calling it? It should be a compile/buildtime error for the other project, but how does that work if everything is in its own little repository?

giaour 5y ago
> Honest question - how do you actually effectively share code between many repos? Locally, you can use an Amazon-internal tool to check out multiple repos and make changes to all of them. The tooling calls this a "workspace," but it feels very much like working in a monorepo since building and testing can happen at the workspace level. > How do I know that me changing my backend app’s API doesn’t break any other project in the company potentially calling it? In terms of change management, Amazon dependency graphs are managed as "version sets." Changes have to be built into a given version set, and that build will also rebuild any package in the version set that consumes the repository whose changes are being built in. (Usually, repositories are configured to build into one of the owning team's version sets on each commit to the primary branch.)
dragonwriter 5y ago
> Honest question - how do you actually effectively share code between many repos? One way is: Each repo is a responsibility boundary and single source of truth, you use code from other repos the same as any other external dependency. > How do I know that me changing my backend app’s API doesn’t break any other project in the company potentially calling it? Changing an API breaks projects using it; you either do versioned APIs and/or coordinate changes with downstream consumers, the same as you would with an API with external customers. (Another way is “downstream projects checkout their dependencies and build against them as a routine part of their process.“)
- sagarm 5y ago
  > Changing an API breaks projects using it; you either do versioned APIs and/or coordinate changes with downstream consumers, the same as you would with an API with external customers. This is a ton of work relative to building/running the tests for all your reverse dependencies and fixing the call sites for them (up to a certain level of scale, of course). > (Another way is “downstream projects checkout their dependencies and build against them as a routine part of their process.“) This happens automatically in a monorepo. Any breakages are revealed as soon as upstream makes the change.
  - dragonwriter 5y ago
    > This is a ton of work relative to building/running the tests for all your reverse dependencies and fixing the call sites for them If someone outside of the team responsible for the functionality at issue can fix all the fall sites for a breaking API change without adversely effecting the domain function of the component making the calls, it's a pretty good sign that there wasn't actually a need for a breaking API change in the first place, and a process change that makes unnecessary API churn more expensive is probably beneficial.
    - giaour 5y ago
      There are some cases where you have to make a breaking change, and in those cases, it is helpful to be able to assess the fallout without a ton of manual effort. E.g., sometimes a security hole can only be remedied by restricting certain inputs (like with Heartbleed), and that's technically a breaking change. Monorepos let you make that assessment within a certain scope, as does Amazon's internal build system. It's not a feature you want exercised regularly, but it sure is helpful when you need it.

ignoramous 5y ago

> It perplexes me people advocate for this strategy. All I can think is it's another one of those cargo-cult ideas that everyone is doing because Google did it (So it must be good). Not sure if it is a generic comment or a comment on TFA: i) If the latter, I'm compelled to point out that TFA doesn't nearly advocate for monorepos as much as it lists reasons why a few SV companies use it, how they use it, and what they get out of it. ii) If the former, then this blog post makes for a good read: https://tailscale.com/blog/modules-monoliths-and-microservices/ https://tailscale.com/blog/modules-monoliths-and-microservic...

giaour 5y ago

Amazon has Brazil versionsets and workspaces, which solve many of the same problems monorepos do. I really liked the way those extra resources let you organize "ad-hoc monorepos" from smaller repositories, but it's infrastructure that I haven't seen elsewhere. Since leaving Amazon, I've mostly worked with monorepos and wouldn't go back to multirepos without Brazil-style tooling.

dastbe 5y ago
very much this. there's also a cultural understanding that libraries/fat clients are equal or worse in terms of maintainability compared to services. equal in the sense that you have to treat them like a service, worse because it's easier to mess up and you don't get to control your rollout strategy. EDIT: Though while most teams have gone towards many smaller packages for their applications, I suspect that most would be better served by a team level monorepo. That gets you all of the benefits of monorepo locally and all of the benefits of manyrepo globally, and unless your project hits the ~200+ developer mark maintaining things will be stay tractable.
ghoward 5y ago
Hey, I'm hoping you can answer some questions for me. I'm building a build system and a VCS (separately). I want to do it right. Could you explain to me what Brazil is? Is it the build system? [1] Or is it the VCS that Amazon uses? If it is the build system, then it appears that versionsets are literally just a list of dependencies with their versions to use for a build. Is that correct? If not, or if you can give me more detail, what are versionsets, exactly? Also, what are workspaces? Does this quote from one of the comments on the link match? > A workspace consisted of a version set to track and any packages that were checked out. [1]: https://gist.github.com/terabyte/15a2d3d407285b8b5a0a7964dd6283b0 https://gist.github.com/terabyte/15a2d3d407285b8b5a0a7964dd6...
- giaour 5y ago
  I am not an expert on Brazil, but the link you shared matches my recollection. It's important to remember that Brazil covers a lot of ground, and many internal tools at Amazon are external to Brazil but rely on it and the way it organizes resources. So I (or others on the internet) may incorrectly call something Brazil if it's part of the larger Brazil ecosystem. Let's start with a repository and build outwards: You have some code (let's say it's a Java library) that does something cool. To compile it, you add a Brazil file to the repository root. This file specifies what Brazil packages your code depends on, how it should be built, and what kind of artifact it will produce. Once that file is there, you can run "brazil-build" to produce a Brazil package (which is just a jar with some metadata). You want to use this library in a web service, so you check out both repositories in a single workspace. Every workspace has a source versionset where it fetches dependencies, but if the code repository for a package is checked out locally, "brazil-build" will build and use the local version instead. You make some changes to the library and web service, then test how they work together by running the web service from within the workspace folder. This ensures that it is using your local modifications to the library repository before those changes have been merged. Once you're satisfied with the change, you open a PR with a brazil-integrated tool that can show changes to multiple repositories as an atomic change (a "change set"). The CI system for this tool uses a Brazil workspace to make sure your update code packages build together and that all tests pass. If the PR is approved and you merge to main, there is probably a pipeline watching your repository for changes that proactively rebuilds one or more version sets based on the merged change set. Any package in the version set that depends on your library will be rebuilt using the changes from your change set. So while a versionset is implemented as a list of packages at specific versions, it's best to think of a versionset as more or less equivalent to a monorepo containing all packages on that list, since changes that cross package lines can be built into a versionset in an atomic unit. (This is very helpful if you need to push out a breaking change.) A package can exist in multiple versionsets, which is of course impossible with monorepos.

throwaway894345 5y ago

Not having to submit and coordinate PRs across a dozen repos is a pretty tangible benefit. > unless you invest a ton of resources into proprietary tooling that requires a bunch of maintenance, since all the open source solutions are terrible at this and the whole data model is built around splitting out individual project repositories. Agree that there's a bunch of tooling needed to operate a monorepo, but there's also a bunch of tooling to sanely manage dozens of "microrepos" as well (when an upstream library changes, update downstream libraries' dependency manifest files to take the new version, run tests, report errors back to upstream, etc). I don't know of any open source tools that manage this problem, but I'm guessing they aren't high-quality if only due to the complex nature of the problem space. > And then after all that effort, why wouldn’t you just use tooling the way it was intended, and the way it’s used in the open source model, so you can partition your CI/CD without a bunch of hacks, and don’t run into bizarre scaling issues with your VCS. Because the tooling sucks, as previously mentioned. Many changes require touching many repos, which means coordinating many pull requests and manually changing dependency manifest files and so on. Ultimately, the "repo" concept is limiting. We need tooling that is aware of dependencies one way or another, and sadly all such open source tooling sucks whether it assumes the relevant code lives in a single repo or across many repos.

erik_seaberg 5y ago
> when an upstream library changes, update downstream libraries' dependency manifest files This needs to happen periodically, when we have slack. Doing it continuously adds risks that aren’t really our job to take.
- throwaway894345 5y ago
  In my experience, if it doesn't happen continuously, it simply doesn't happen at all until something breaks (and then there's a bunch of finger-pointing at upstream even though downstream didn't update). Your first line of defense is that downstream tests are run before anything that affects downstream is merged. The next line of defense is stuff like canary deployments which allow you to minimize blast radius and roll back quickly. Obviously this depends a great deal on your risk regime--if you're SaaS this is probably fine, but if you're embedded this is a non-starter.