6 ms·

I still cannot wrap my head around this. What merit does forcing internal code to update a dependancy have? Having dependencies packaged like NuGet, and using a

I still cannot wrap my head around this. What merit does forcing internal code to update a dependancy have? Having dependencies packaged like NuGet, and using an internal repository, you can always remove a particular version of a dependency if you must, and a build will fail if it cannot automatically use the latest minor version.

regularfry 10y ago
It means that I, as a dependency author, can make a breaking change and push not only the updates to my library, but also the updates to all my dependents in a single commit. Using a dependency packaging system like NuGet is a workaround for me not knowing who all my clients are. In a monorepo, I can have perfect knowledge.
- humanrebar 10y ago
  > Using a dependency packaging system like NuGet is a workaround for me not knowing who all my clients are. Between published manifests and search indexed source code, you would know who all your clients are, even without a monorepo.
- michaelt 10y ago
  push not only the updates to my library, but also the updates to all my dependents in a single commit. And what's more, you can run all their automated tests and address any problems your changes introduce. Which presumably means regular forced updates aren't the recipe for instability they would otherwise be.
- gcb0 10y ago
  take debian. you write a new version. push the new package. everyone else's build will fail until they update. its as bad as you describe, but more organized :)
- cwilkes 10y ago
  Should you know who all your clients are? That seems to be a workaround for not having an adequate testing framework or a contract stating what your code will do. Do open source frameworks know everyone's code that is using it? Plus you do have perfect knowledge if your CI tools builds everything on your commit. Which it will do anyway if you have a monorepo.
  - robrenaud 10y ago
    > That seems to be a workaround for not having an adequate testing framework or a contract Sometimes you get that contract wrong, and you don't realize it until you've shipped it. If you want to evolve an API by introducing a breaking change, you can do it at Google. You just have to migrate the clients with the API change. It means your APIs can both improve and stay lean, without needing to carry around legacy cruft, which is great. It's not really an option in open source.
    - cwilkes 10y ago
      That's a good point about not carrying cruft forward. I've often seen on projects where there's crazy methods like sendEmailOn3rdBirthdayAfterFullMoon as someone somewhere needed it for a single project but who knows if it is still in use.
- hvidgaard 10y ago
  You will not have perfect knowledge. Even at Google they don't truely have a mono repo, as there is several other projects that live outside the monorepo. A build system will also detect changes and run test, produce a build, and notify maintainers, when you update a dependency. Monorepo is not needed for this.
DannyBee 10y ago
"I still cannot wrap my head around this. What merit does forcing internal code to update a dependancy have?" It allows a push model instead of a pull one. You can know everything you will break ahead of time. More generally, you can know the impact of a change (performance, etc) before you make it. Is it possible to do it without a mono-repo? Theoretically, yes, through some large federated something or other. These don't really exist ;) Basically, it lets you have a library team that moves the world forward without causing pain (at least, for those reasons :P).
- tamana 10y ago
  The latrge federated something or other doesn't exist because Google chose to build monorepo tools instead, and approximately no one else's code base is really big enough to matter. Why didn't/couldn't Google build the federated something or other?
  - DannyBee 10y ago
    The short answer is likely "It wasn't worth the cost/complexity/etc of that approach".
  - hvidgaard 10y ago
    It's an automated build system, it already exists. I think it's more likely that Googlers decided that they had the discipline for a mono repo - time will tell if that hold true.
- ISV_Damocles 10y ago
  Not theoretically, with [OpenGrok](https://opengrok.github.io/OpenGrok/ https://opengrok.github.io/OpenGrok/) you can register all of your git repositories and then search for the dependency declaration of your library across the entire company's source code, and then with that list you can use a simple bash script to autogenerate the git diffs to switch to the appropriate security patch for whichever minor version of the library they're using. You can also use this exact same approach to create tasks on teams to upgrade their library if you've EOLed the version they're using. You can be as narrow (requiring everyone on the latest version) or as broad (having a band of versions allowed, in case the change is significant enough to require updates to how it is used, or if there's a major component of the system using an older version for performance-regression reasons you're still addressing on the improved library) as you want. This actually simplifies your build system, as you can reuse the same standard open source tools that come along with your language of choice, which often assume they're running at the root of the project and can do what they want to the surrounding directory structure, and build-n-test cycles take mere minutes instead of hours, being isolated to only the component you're working with (and in the case of a library update across project, parallelized due to this isolation across build servers, so still much faster than the monorepo approach). This also makes the codebase healthier, in general. Libraries and services need to be designed to actually have a clear purpose and interface; they can't cheat by grabbing source files from each other while breaking the expected interface, so refactoring their internals has a near zero probability of causing downstream issues. Services can be rewritten by standing up a parallel service, shadowing traffic, and validating equal responses between them. You can also fork and repurpose service code for another project and know exactly what code is required by it immediately. These things are possible in monorepos, but they require a very high degree of diligence, and even moreso in a mixed-language environment. In general, breaking everything into a separate repository is madness (see the modern Node.js ecosystem), so new services should start out as "mini-monorepos" where everything new they need that isn't already an available library should be contained within the service. Once a few services seem to be doing the same sort of thing, then you look at the commonalities between their implementations, learn from past mistakes (that we all make), and make the library to share between them, and allow future services to not have to reimplement that. (This also has the advantage of making it easier to open source said library, since it would contain zero references to proprietary services, and requires no cleaning of the git history, and to get services to use your library, you already need to document it clearly.) Frankly, the world Google is in with their monorepo is an amazing feat of engineering built upon the initial tools they had available ~15 years ago, but it really isn't an ideal situation, and there are better, more flexible ways of doing things now. I'd be sad if our industry hadn't improved this over such a period of time.

regularfry 10y ago

It means that I, as a dependency author, can make a breaking change and push not only the updates to my library, but also the updates to all my dependents in a single commit. Using a dependency packaging system like NuGet is a workaround for me not knowing who all my clients are. In a monorepo, I can have perfect knowledge.

humanrebar 10y ago
> Using a dependency packaging system like NuGet is a workaround for me not knowing who all my clients are. Between published manifests and search indexed source code, you would know who all your clients are, even without a monorepo.
michaelt 10y ago
push not only the updates to my library, but also the updates to all my dependents in a single commit. And what's more, you can run all their automated tests and address any problems your changes introduce. Which presumably means regular forced updates aren't the recipe for instability they would otherwise be.
gcb0 10y ago
take debian. you write a new version. push the new package. everyone else's build will fail until they update. its as bad as you describe, but more organized :)
cwilkes 10y ago
Should you know who all your clients are? That seems to be a workaround for not having an adequate testing framework or a contract stating what your code will do. Do open source frameworks know everyone's code that is using it? Plus you do have perfect knowledge if your CI tools builds everything on your commit. Which it will do anyway if you have a monorepo.
- robrenaud 10y ago
  > That seems to be a workaround for not having an adequate testing framework or a contract Sometimes you get that contract wrong, and you don't realize it until you've shipped it. If you want to evolve an API by introducing a breaking change, you can do it at Google. You just have to migrate the clients with the API change. It means your APIs can both improve and stay lean, without needing to carry around legacy cruft, which is great. It's not really an option in open source.
  - cwilkes 10y ago
    That's a good point about not carrying cruft forward. I've often seen on projects where there's crazy methods like sendEmailOn3rdBirthdayAfterFullMoon as someone somewhere needed it for a single project but who knows if it is still in use.
hvidgaard 10y ago
You will not have perfect knowledge. Even at Google they don't truely have a mono repo, as there is several other projects that live outside the monorepo. A build system will also detect changes and run test, produce a build, and notify maintainers, when you update a dependency. Monorepo is not needed for this.

DannyBee 10y ago

"I still cannot wrap my head around this. What merit does forcing internal code to update a dependancy have?" It allows a push model instead of a pull one. You can know everything you will break ahead of time. More generally, you can know the impact of a change (performance, etc) before you make it. Is it possible to do it without a mono-repo? Theoretically, yes, through some large federated something or other. These don't really exist ;) Basically, it lets you have a library team that moves the world forward without causing pain (at least, for those reasons :P).

tamana 10y ago
The latrge federated something or other doesn't exist because Google chose to build monorepo tools instead, and approximately no one else's code base is really big enough to matter. Why didn't/couldn't Google build the federated something or other?
- DannyBee 10y ago
  The short answer is likely "It wasn't worth the cost/complexity/etc of that approach".
- hvidgaard 10y ago
  It's an automated build system, it already exists. I think it's more likely that Googlers decided that they had the discipline for a mono repo - time will tell if that hold true.
ISV_Damocles 10y ago
Not theoretically, with [OpenGrok](https://opengrok.github.io/OpenGrok/ https://opengrok.github.io/OpenGrok/) you can register all of your git repositories and then search for the dependency declaration of your library across the entire company's source code, and then with that list you can use a simple bash script to autogenerate the git diffs to switch to the appropriate security patch for whichever minor version of the library they're using. You can also use this exact same approach to create tasks on teams to upgrade their library if you've EOLed the version they're using. You can be as narrow (requiring everyone on the latest version) or as broad (having a band of versions allowed, in case the change is significant enough to require updates to how it is used, or if there's a major component of the system using an older version for performance-regression reasons you're still addressing on the improved library) as you want. This actually simplifies your build system, as you can reuse the same standard open source tools that come along with your language of choice, which often assume they're running at the root of the project and can do what they want to the surrounding directory structure, and build-n-test cycles take mere minutes instead of hours, being isolated to only the component you're working with (and in the case of a library update across project, parallelized due to this isolation across build servers, so still much faster than the monorepo approach). This also makes the codebase healthier, in general. Libraries and services need to be designed to actually have a clear purpose and interface; they can't cheat by grabbing source files from each other while breaking the expected interface, so refactoring their internals has a near zero probability of causing downstream issues. Services can be rewritten by standing up a parallel service, shadowing traffic, and validating equal responses between them. You can also fork and repurpose service code for another project and know exactly what code is required by it immediately. These things are possible in monorepos, but they require a very high degree of diligence, and even moreso in a mixed-language environment. In general, breaking everything into a separate repository is madness (see the modern Node.js ecosystem), so new services should start out as "mini-monorepos" where everything new they need that isn't already an available library should be contained within the service. Once a few services seem to be doing the same sort of thing, then you look at the commonalities between their implementations, learn from past mistakes (that we all make), and make the library to share between them, and allow future services to not have to reimplement that. (This also has the advantage of making it easier to open source said library, since it would contain zero references to proprietary services, and requires no cleaning of the git history, and to get services to use your library, you already need to document it clearly.) Frankly, the world Google is in with their monorepo is an amazing feat of engineering built upon the initial tools they had available ~15 years ago, but it really isn't an ideal situation, and there are better, more flexible ways of doing things now. I'd be sad if our industry hadn't improved this over such a period of time.