7 ms·

My time at amazon is what firmly convinced me that the “monorepo is the only way to develop at scale” folks just hadn’t seen something like Brazil. It (and apo

My time at amazon is what firmly convinced me that the “monorepo is the only way to develop at scale” folks just hadn’t seen something like Brazil. It (and apollo) were truly a work of art and they were designed and built so long before everyone else was talking about reproducible builds. Apollo had us deploying exact copies to our dev environments before docker came along and made it easy. I have no idea how the designers had so much foresight.

solipsism 6y ago
How is it an improvement over the monorepo? The article claims it is but doesn't say how.
- nickm12 6y ago
  The main power of Brazil is that every buildable unit ("package" in Brazil) can have multiple branches and then a software stack can be defined in terms of the set of branches ("version set" in Brazil) across all the software in the stack. The "version set" defines the complete build, test, and runtime closure of the application. In that sense, a version set is very much like a branch in a monorepo. The difference is that version sets are more composable and a version set contains only a subset of the repo. Like branches, version sets can have an upstream version set, so a common pattern is to have three layers of version sets: the central one (called "live" at Amazon) which contains your build tools, open source packages etc.; a shared platform version set which contains code shared across many applications; and then a number of downstream application version sets.
  - solipsism 6y ago
    Thanks, but that's what was in the article. Why is this better than monorepo, specifically? What use cases does it allow that aren't possible in a monorepo? BTW, the typically talked about giant monorepos do support branching, and the branch is never the entire repo.
    - nickm12 6y ago
      A branch in a git repository defines a snapshot of the entire repository. You can't have two branches checked out simultaneously. If your repository has a file at a path, you can't have some part of the repository depending on that file at revision A and some other part of the repository depending on that file at revision B. Version Sets let you do that.
      - solipsism 6y ago
        Not everything is git. Subversion lets you do that. So does perforce. And of course Google's monorepo has its history in perforce. Again... People making this claim that Brazil is so much better than monorepos seem to know a lot about Brazil but not much about monorepos.
        nickm12 6y ago
        Brazil was originally built on a Perforce monorepo. It didn't start being git until 2013 or so and even then packages could migrate between the Perforce repository and git repository. So Brazil is not an alternative to a monorepo.
- donavanm 6y ago
  It starts with a distributed approach, from the bottom up. The primitives of Packages and Version Sets match how teams are organized and work. Package versions, VFIs, and package builds are the interfaces & immutable artifacts for teams and software stacks to interact. Conversely the the monorepo approach seems to spend its time trying to serialize and separate the innate codependencies of its approach.
  - solipsism 6y ago
    Package versions, VFIs, and package builds are the interfaces & immutable artifacts for teams and software stacks to interact. That doesn't explain anything, it's just a bunch of terms. Monorepos have lots of terms too. Why is it so hard to explain? I suspect it's because people claiming one is better than the other are only familiar with one, so they can't actually back up that claim.
nickm12 6y ago
My first couple of years at Amazon I was completely mystified by Brazil, version sets, and VFIs. Eventually the lightbulb clicked and when I moved on to a company with a monorepo I really missed Brazil. My main gripe with Brazil is that there are few ways to use it properly and a lot of ways to use it wrong and when you use it wrong you easily fall into dependency hell.
- qznc 6y ago
  You seem to be one of the rare people who used both, Brazil and monorepo. Could you elaborate? What did you realize when the lightbulb clicked? What do you miss outside of Amazon? How can Brazil be misused?
  - nickm12 6y ago
    So the main thing that clicked with Brazil was really understanding how to use versionsets effectively to model software stacks and in particular thinking about a package's dependencies as part of its public interface. A common anti-pattern I saw at Amazon was for some team to need some functionality, see a package in the "live" version set that had it, and grab it without thinking about how that package's dependencies aligned with their application's. Teams also frequently used the wrong dependency type (runtime, build, test). Things will "work" if you declare a test dependency as a runtime dependency, but you are setting yourself up for unnecessary version conflicts down the road (you can have multiple versions of a package in your test dependency closure, but not your runtime dependency closure). As for what I missed at the company with the monorepo there were two things: The first was that there was no way to deploy a version of the code that was exactly the same as what was in production plus some patch. The deployment system could only deploy the tip of the master branch, so if you needed to patch in prod you were going to bring with it all the other changes that had landed on the master branch, for better or for worse. There is no reason that a monorepo has to have this problem, though, it was a limitation of the deployment system. The main thing I didn't like about the monorepo though was that it was next to impossible for me to track the changes that were relevant to me. In Brazil, every package is its own git repo and it's very straightforward to just list the git history of a package and see what has changed. In the monorepo 99% of the change in the git history were completely irrelevant. Also, twice they needed to rewrite the history of the entire repo to expunge some secret that had accidentally been committed years ago. All the hashes changed and it was very disruptive. So what's good about Brazil is that it gives you the advantages of a monorepo without all the source code literally being in the same repo. It's also good because you can maintain multiple versions of the same package together and migrate different consuming apps separately. What's not so great about Brazil is that there is really a subtle art to factoring your software into packages and version sets and most times you get it wrong in a way that leads to seemingly unnecessary pain. The fact that you don't have to update all consumers together means that its really easy to just not update consumers and you pay the price for that eventually. There's another big pain point with Brazil which is that it was traditionally difficult to import open source software from open-source repositories (NPM, PyPy, Maven, etc.). This has gotten much better in recent years and there are efforts to improve it even more.

solipsism 6y ago

How is it an improvement over the monorepo? The article claims it is but doesn't say how.

nickm12 6y ago
The main power of Brazil is that every buildable unit ("package" in Brazil) can have multiple branches and then a software stack can be defined in terms of the set of branches ("version set" in Brazil) across all the software in the stack. The "version set" defines the complete build, test, and runtime closure of the application. In that sense, a version set is very much like a branch in a monorepo. The difference is that version sets are more composable and a version set contains only a subset of the repo. Like branches, version sets can have an upstream version set, so a common pattern is to have three layers of version sets: the central one (called "live" at Amazon) which contains your build tools, open source packages etc.; a shared platform version set which contains code shared across many applications; and then a number of downstream application version sets.
- solipsism 6y ago
  Thanks, but that's what was in the article. Why is this better than monorepo, specifically? What use cases does it allow that aren't possible in a monorepo? BTW, the typically talked about giant monorepos do support branching, and the branch is never the entire repo.
  - nickm12 6y ago
    A branch in a git repository defines a snapshot of the entire repository. You can't have two branches checked out simultaneously. If your repository has a file at a path, you can't have some part of the repository depending on that file at revision A and some other part of the repository depending on that file at revision B. Version Sets let you do that.
    - solipsism 6y ago
      Not everything is git. Subversion lets you do that. So does perforce. And of course Google's monorepo has its history in perforce. Again... People making this claim that Brazil is so much better than monorepos seem to know a lot about Brazil but not much about monorepos.
      - nickm12 6y ago
        Brazil was originally built on a Perforce monorepo. It didn't start being git until 2013 or so and even then packages could migrate between the Perforce repository and git repository. So Brazil is not an alternative to a monorepo.
donavanm 6y ago
It starts with a distributed approach, from the bottom up. The primitives of Packages and Version Sets match how teams are organized and work. Package versions, VFIs, and package builds are the interfaces & immutable artifacts for teams and software stacks to interact. Conversely the the monorepo approach seems to spend its time trying to serialize and separate the innate codependencies of its approach.
- solipsism 6y ago
  Package versions, VFIs, and package builds are the interfaces & immutable artifacts for teams and software stacks to interact. That doesn't explain anything, it's just a bunch of terms. Monorepos have lots of terms too. Why is it so hard to explain? I suspect it's because people claiming one is better than the other are only familiar with one, so they can't actually back up that claim.

nickm12 6y ago

My first couple of years at Amazon I was completely mystified by Brazil, version sets, and VFIs. Eventually the lightbulb clicked and when I moved on to a company with a monorepo I really missed Brazil. My main gripe with Brazil is that there are few ways to use it properly and a lot of ways to use it wrong and when you use it wrong you easily fall into dependency hell.

qznc 6y ago
You seem to be one of the rare people who used both, Brazil and monorepo. Could you elaborate? What did you realize when the lightbulb clicked? What do you miss outside of Amazon? How can Brazil be misused?
- nickm12 6y ago
  So the main thing that clicked with Brazil was really understanding how to use versionsets effectively to model software stacks and in particular thinking about a package's dependencies as part of its public interface. A common anti-pattern I saw at Amazon was for some team to need some functionality, see a package in the "live" version set that had it, and grab it without thinking about how that package's dependencies aligned with their application's. Teams also frequently used the wrong dependency type (runtime, build, test). Things will "work" if you declare a test dependency as a runtime dependency, but you are setting yourself up for unnecessary version conflicts down the road (you can have multiple versions of a package in your test dependency closure, but not your runtime dependency closure). As for what I missed at the company with the monorepo there were two things: The first was that there was no way to deploy a version of the code that was exactly the same as what was in production plus some patch. The deployment system could only deploy the tip of the master branch, so if you needed to patch in prod you were going to bring with it all the other changes that had landed on the master branch, for better or for worse. There is no reason that a monorepo has to have this problem, though, it was a limitation of the deployment system. The main thing I didn't like about the monorepo though was that it was next to impossible for me to track the changes that were relevant to me. In Brazil, every package is its own git repo and it's very straightforward to just list the git history of a package and see what has changed. In the monorepo 99% of the change in the git history were completely irrelevant. Also, twice they needed to rewrite the history of the entire repo to expunge some secret that had accidentally been committed years ago. All the hashes changed and it was very disruptive. So what's good about Brazil is that it gives you the advantages of a monorepo without all the source code literally being in the same repo. It's also good because you can maintain multiple versions of the same package together and migrate different consuming apps separately. What's not so great about Brazil is that there is really a subtle art to factoring your software into packages and version sets and most times you get it wrong in a way that leads to seemingly unnecessary pain. The fact that you don't have to update all consumers together means that its really easy to just not update consumers and you pay the price for that eventually. There's another big pain point with Brazil which is that it was traditionally difficult to import open source software from open-source repositories (NPM, PyPy, Maven, etc.). This has gotten much better in recent years and there are efforts to improve it even more.