5 ms·

I worked at Google (in a team using Perforce) and now work at a different company that uses multiple interdependent projects using Maven. Using a single monolit

by yonran 13y ago

I worked at Google (in a team using Perforce) and now work at a different company that uses multiple interdependent projects using Maven. Using a single monolithic codebase along with a build tool that statically builds everything at trunk has its advantages: * You immediately get improvements from upstream projects without having to get them manually. * You can unambiguously answer the question, What code am I using? with a single number. With multiple repositories, you have to list all the versions of each project that you are using. * Easy API refactoring. You don’t have to worry about coordinating version number bumps across different repositories/dependency manifests when you make major changes to inter-project APIs. With a monolithic repository, you fix all callers of an API using a single code commit. No need to edit the version numbers in your pom files. * Low cost to split a project into multiple separate projects. With multiple repositories or version numbers, you are reluctant to create new projects because the APIs will forever be harder to refactor (since you will have to worry about version numbers). * No diamond dependency problem of 2 dependencies using a different version of a base project. Everyone is using the same version of base. With a monolithic repository+build system, upstream callers are responsible to never make a commit that breaks downstream callers. I feel like it’s similar to the question of optimal currency areas. If your organization is growing in lock-step, then you can all happily share a single gold-standard repository with little friction. But if you can’t trust your upstream projects, you introduce versioning between the projects and have to deal with the mental burden of wondering whether to upgrade to the newest upstream project and whether you’re actually running the latest code. Edit: added a couple more.

Touche 13y ago
> * You immediately get improvements from upstream projects without having to get them manually. You also immediately get regressions. Not trying to be dismissive, but we fundamentally have different software philosophies if you think this point (which is the essence of most of your points) is a good thing that should be encouraged.
- eropple 13y ago
  Test better.
  - warmwaffles 13y ago
    Nice passive aggressive comment. Should try enlightening the commenter instead of berating them.
- yonran 13y ago
  You’re right, there are disadvantages to a single codebase and builds at trunk. You pick up regressions, and it is difficult for your team to develop on its own while holding everything else constant. But I don’t think the speed of receiving fixes is the essence of, or even primarily the source of, all my headaches with versions. The problem with mixing and matching versions within an organization is the enormous complexity that it introduces. Perhaps your downstream coworkers are still using an old version of your project, so they don’t want you to refactor their use of your API. Or perhaps you forgot to update your required upstream dependency when using a new function from a library, and your coworker’s program crashed because they’re still using the old dependency. Or perhaps someone forgot to bump the major version number when changing API or behavior, causing a previously built downstream project that is linking against the new upstream project to crash. Now, these problems are all solvable if you and your coworkers are very disciplined in updating your version numbers and your required dependency versions. But it means that you constantly have to be aware of what APIs you export and what versions of APIs you are calling. You constantly have to edit the project manifests to bump version numbers. You must think about whether your changes will be major or minor. You carefully read the Changelist before using the newest upstream projects. It is a mental burden. Contrast this to a monolithic codebase and build system. There are no version numbers in the dependency manifests to other projects in the company. If you want to change an API, you are responsible for fixing all the downstream users (rather than the other way around). Making a new project adds little mental overhead. If there is no impedance mismatch between the different teams of your company, it can make life much easier.
- stormbrew 13y ago
  You still immediately get regressions from separate repositories, you just don't find out until later. Possibly when you're no longer in the right mental space for dealing with it.
- thrownaway2424 13y ago
  Immediate regressions are good! If someone at Google breaks my code, I will know within half an hour at the latest and I will tell them to go fix it or just revert their changes myself. Immediate regressions also go perfectly with daily (or hourly!) releases. If there's a performance problem it will be identified early and I will only have thousands of changes to investigate instead of tens of millions. Imagine if I had a regression and I had to go to the other team saying "We just upgraded from the Foo you released 2 years ago to the one from last year and the performance sucks. Help!" I would not get any help. However I get plenty of support when I go to Foo-team to tell them that my Foo-per-second is 10% worse in the noon release compared to the midnight release. Having artifacts and stable interfaces and library releases and all that is very ivory tower hocus pocus stuff. In practice instant integration is better.
  - Touche 13y ago
    I don't just mean performance regressions. Someone upstream can change an API in a way that doesn't fit well with your use-case, goes in and "fixes" your code (makes sure all the tests pass) to fit the new API but makes it less maintainable in the process.
    - thrownaway2424 13y ago
      Teams (at Google) can't change calling code without the review and approval of the owners of the calling code, so what you state would not happen.
bluesnowmonkey 13y ago
Does this approach not isolate Google from the open source community? Most companies nowadays run on open source tools built from open source libraries written in open source languages. Version numbers and stable APIs are crucial.
exDM69 13y ago
> * You can unambiguously answer the question, What code am I using? with a single number. With multiple repositories, you have to list all the versions of each project that you are using. Git submodules may have a number of problems of its own, but it solves this one. There's always an unambiguous version number, which is the commit hash of the top repo. Every subrepository's commit hash is stored in the top repo and the top repo's version is an unambiguous version number of the entire code base. I wish that more effort would be put in Git submodules, it's pretty much an afterthought addition but I've heard that there have been recent improvements and future improvements may be coming...
- estebank 13y ago
  In 1.8.2 you can track a branch name instead of a commit hash. This has the benefit of allowing you to work always against the latest version, while foregoing the advantage of having a single unambiguous version number.
shoover 13y ago
Nice list. What then are the mechanics of making a breaking change in a library, file format, network protocol, etc. in the single repo system?
- yonran 13y ago
  Ideally, to make a breaking API change, you change the function and all the references within a single commit. Since this repository is only used for statically compiled programs, there is no need to keep the old API anymore. For protocols and file formats, Google universally uses protocol buffers with many optional fields. The protocol buffer library’s default is that when you read a protocol buffer, modify it, and write it back out, the fields that you didn’t understand are passed through. This means that middleman servers don’t need to be recompiled when you add new optional fields that they don’t use. But for the actual client and server, you generally don’t have the luxury of replacing them both at the same time. So you have to add the new field that is disabled using a flag, wait for it to rolled out to both the client and server, then enable the new field and disable the old field using the flag, then remove the flag and old field. It’s something that you coordinate with the release engineers. But it’s not formalized in the software version numbers.
  - shoover 13y ago
    Useful flexibility from protocol buffers there. To me it seems like having to change all references for a breaking API change could be a debilitating amount of work in some cases. Do you then make your breaking change to a branch and lobby for other teams to catch up before merging to the main branch? What about situations where you have a legion of stable legacy applications that may not be worth updating for any reason other than critical bugs?
    - yonran 13y ago
      Yes, it can be difficult to change a library used by everyone, because you need to get a code review and commit the change before merge conflicts start piling up. But you do it all in one commit; you don’t do refactoring in a branch as far as I remember. Occasionally one would hear from someone like Craig Silverstein touching hundreds of files. By the way, check out his talk on refactoring using clang <http://llvm.org/devmtg/2010-11/> http://llvm.org/devmtg/2010-11/>. If an application is still being used, it is always stored in the source tree, where the unit tests are automatically run. You do still have choices to lock its API or file formats: you can consider the API deprecated and tell everyone to use the new V2 API, or you can move the old program into a branch (but still in the source tree that everyone can see). But you want to branch as little as possible; large unmaintained branches quickly become unmaintainable.

by yonran 13y ago

Touche 13y ago

> * You immediately get improvements from upstream projects without having to get them manually. You also immediately get regressions. Not trying to be dismissive, but we fundamentally have different software philosophies if you think this point (which is the essence of most of your points) is a good thing that should be encouraged.

eropple 13y ago
Test better.
- warmwaffles 13y ago
  Nice passive aggressive comment. Should try enlightening the commenter instead of berating them.
yonran 13y ago
You’re right, there are disadvantages to a single codebase and builds at trunk. You pick up regressions, and it is difficult for your team to develop on its own while holding everything else constant. But I don’t think the speed of receiving fixes is the essence of, or even primarily the source of, all my headaches with versions. The problem with mixing and matching versions within an organization is the enormous complexity that it introduces. Perhaps your downstream coworkers are still using an old version of your project, so they don’t want you to refactor their use of your API. Or perhaps you forgot to update your required upstream dependency when using a new function from a library, and your coworker’s program crashed because they’re still using the old dependency. Or perhaps someone forgot to bump the major version number when changing API or behavior, causing a previously built downstream project that is linking against the new upstream project to crash. Now, these problems are all solvable if you and your coworkers are very disciplined in updating your version numbers and your required dependency versions. But it means that you constantly have to be aware of what APIs you export and what versions of APIs you are calling. You constantly have to edit the project manifests to bump version numbers. You must think about whether your changes will be major or minor. You carefully read the Changelist before using the newest upstream projects. It is a mental burden. Contrast this to a monolithic codebase and build system. There are no version numbers in the dependency manifests to other projects in the company. If you want to change an API, you are responsible for fixing all the downstream users (rather than the other way around). Making a new project adds little mental overhead. If there is no impedance mismatch between the different teams of your company, it can make life much easier.
stormbrew 13y ago
You still immediately get regressions from separate repositories, you just don't find out until later. Possibly when you're no longer in the right mental space for dealing with it.
thrownaway2424 13y ago
Immediate regressions are good! If someone at Google breaks my code, I will know within half an hour at the latest and I will tell them to go fix it or just revert their changes myself. Immediate regressions also go perfectly with daily (or hourly!) releases. If there's a performance problem it will be identified early and I will only have thousands of changes to investigate instead of tens of millions. Imagine if I had a regression and I had to go to the other team saying "We just upgraded from the Foo you released 2 years ago to the one from last year and the performance sucks. Help!" I would not get any help. However I get plenty of support when I go to Foo-team to tell them that my Foo-per-second is 10% worse in the noon release compared to the midnight release. Having artifacts and stable interfaces and library releases and all that is very ivory tower hocus pocus stuff. In practice instant integration is better.
- Touche 13y ago
  I don't just mean performance regressions. Someone upstream can change an API in a way that doesn't fit well with your use-case, goes in and "fixes" your code (makes sure all the tests pass) to fit the new API but makes it less maintainable in the process.
  - thrownaway2424 13y ago
    Teams (at Google) can't change calling code without the review and approval of the owners of the calling code, so what you state would not happen.

bluesnowmonkey 13y ago

Does this approach not isolate Google from the open source community? Most companies nowadays run on open source tools built from open source libraries written in open source languages. Version numbers and stable APIs are crucial.

exDM69 13y ago

> * You can unambiguously answer the question, What code am I using? with a single number. With multiple repositories, you have to list all the versions of each project that you are using. Git submodules may have a number of problems of its own, but it solves this one. There's always an unambiguous version number, which is the commit hash of the top repo. Every subrepository's commit hash is stored in the top repo and the top repo's version is an unambiguous version number of the entire code base. I wish that more effort would be put in Git submodules, it's pretty much an afterthought addition but I've heard that there have been recent improvements and future improvements may be coming...

estebank 13y ago
In 1.8.2 you can track a branch name instead of a commit hash. This has the benefit of allowing you to work always against the latest version, while foregoing the advantage of having a single unambiguous version number.

shoover 13y ago

Nice list. What then are the mechanics of making a breaking change in a library, file format, network protocol, etc. in the single repo system?

yonran 13y ago
Ideally, to make a breaking API change, you change the function and all the references within a single commit. Since this repository is only used for statically compiled programs, there is no need to keep the old API anymore. For protocols and file formats, Google universally uses protocol buffers with many optional fields. The protocol buffer library’s default is that when you read a protocol buffer, modify it, and write it back out, the fields that you didn’t understand are passed through. This means that middleman servers don’t need to be recompiled when you add new optional fields that they don’t use. But for the actual client and server, you generally don’t have the luxury of replacing them both at the same time. So you have to add the new field that is disabled using a flag, wait for it to rolled out to both the client and server, then enable the new field and disable the old field using the flag, then remove the flag and old field. It’s something that you coordinate with the release engineers. But it’s not formalized in the software version numbers.
- shoover 13y ago
  Useful flexibility from protocol buffers there. To me it seems like having to change all references for a breaking API change could be a debilitating amount of work in some cases. Do you then make your breaking change to a branch and lobby for other teams to catch up before merging to the main branch? What about situations where you have a legion of stable legacy applications that may not be worth updating for any reason other than critical bugs?
  - yonran 13y ago
    Yes, it can be difficult to change a library used by everyone, because you need to get a code review and commit the change before merge conflicts start piling up. But you do it all in one commit; you don’t do refactoring in a branch as far as I remember. Occasionally one would hear from someone like Craig Silverstein touching hundreds of files. By the way, check out his talk on refactoring using clang <http://llvm.org/devmtg/2010-11/> http://llvm.org/devmtg/2010-11/>. If an application is still being used, it is always stored in the source tree, where the unit tests are automatically run. You do still have choices to lock its API or file formats: you can consider the API deprecated and tell everyone to use the new V2 API, or you can move the old program into a branch (but still in the source tree that everyone can see). But you want to branch as little as possible; large unmaintained branches quickly become unmaintainable.