5 ms·

With multiple repos, how do you solve the issue that the private dependencies installed by package managers are not themselves under source control when you edi

by mnm1 9y ago

With multiple repos, how do you solve the issue that the private dependencies installed by package managers are not themselves under source control when you edit them? They are in /vendor or /node_modules or whatever. It's easy to pull them down and of course, for other people's dependencies, this is fine. But say I pull out one component of my app that's used by multiple apps. I set up my private repo or maybe the package manager works straight with git. But when it pulls the dependencies down, it doesn't pull the git subdirectories and so that code is not under source control. I make edits, test, then I have to copy that code back to its original repo, resolve any conflicts, and check it in. If I need to make a change, I have to do that again. If someone worked on that code, I have to pull that and merge it in separately. Even if I manually pull it myself in the right place, I'm still dealing with a git repo inside a git repo but the two are unrelated so many tools won't work with the inner repo. Short of writing a bunch of custom scripts, is there a standard way to handle this situation that I assume anyone with multiple repos that share internal, private dependencies has?

peterevans 9y ago
Any changes you need to make to internal libraries you install via package manager would need to be made to those separate repositories that house the libraries, and then those changes would have a release made (generally this is done through a git tag, and using semantic versioning). Once you update the version, your package manager will allow you to install that update to the other places you're using it, so all you need to change is the package file. It might seem like a pain to do it this way, especially if you're rapidly iterating on a library—it might be that your library is not really mature, or even used by more than one repository, so it may not even make sense to have that library in a separate repository to begin with! But once you do have a mature code base, semantic versioning is a really sane way of managing dependency updates for the N number of other projects which use your library.
- hinkley 9y ago
  And how do you test if a change in one of these repos fixes the problem you’re seeing? This is what we’re failing at with our multirepo. That and resectioning code to split or combine responsibilities in different ways. Something a monorepo makes trivial.
  - peterevans 9y ago
    Ideally, your library code that you're pulling in has some unit testing to demonstrate that things are working as they should be. (If not, consider adding unit testing! It's really useful!) If that is the case, then you can isolate the likelihood of an issue as either in the library (because unit tests fail there), or in the project consuming the library (because unit tests succeed in the library). Without testing, it's hard to have a ton of confidence in where the problem lies—which is exactly the problem you cite. And while a monorepo (or just a standalone repository with no separate libraries, which is frankly an easier setup to manage than monorepos or multirepos!) may make debugging a bit easier, it's not going to give you much more confidence in your code. Once your organization outgrows the paradigm where you just have N standalone projects with N completely separate code bases, and you do need to commit to either a multi-repo or a mono-repo configuration, it's really, really helpful to have unit testing to allow you to isolate where issues are.
    - hinkley 9y ago
      The class of problems I’m talking about are integration issues. Unit tests look good but when you put the pieces together... When the tests that matter cross version control boundaries you pay for it. Whether the costs outweigh the benefits is something you have to think about.
      - peterevans 9y ago
        What does your test coverage look like? Perhaps you're missing something there that would have caught that bug? Testing is, of course, no silver bullet. Tests are written by humans, and humans make mistakes—and it's pretty difficult to achieve 100% test coverage in a production system. The goal of testing is to have confidence in the code you've written. Tests often don't need to cross version control boundaries. You can use mock data—like would be produced by the library—on the consumer side, because you can delegate responsibility for testing of that library to the library repository itself. If your tests work great with the mock data, but things are still failing, then you can infer that the mock data and the actual data are different, and your bug is in the library.
- chrisweekly 9y ago
  Yes, tho I'd go further and say semver is table stakes -- and so is determinism (achieved on the library-consuming side by committing and maintaining the yarn.lock file, in the specific case of node projects).
- mnm1 9y ago
  No, semantic versioning and package management is really not a sane way to manage development. You're saying every time I make a code change, I should release a patch version BEFORE I can even test it, since I cannot test that dependency by itself. I'm sorry, but that's just insane, not to mention time consuming. And yes, my libraries are used by multiple apps. Why else would I need to split them out as dependencies? Actually, after thinking through this, the only solution I can think of is symlinks and those would work with a monorepo or multiple repos just as well.
  - peterevans 9y ago
    Sorry—I just saw this! But, as it was, I was not saying that you should release a patch version every time you make a code change. That would indeed be pretty arduous! What you should do is test your library independently from the consumer that uses it. If that's not possible, ask yourself why that is; maybe this code should not be sequestered into a library after all, or maybe it needs to be designed a bit differently to remove some of the coupling between the library and app that seems to be at issue. Ideally, your testing in your app should not be testing that the library works; it should be testing that your use of the library works. Too bad the real world is much messier than the ideal world. Good luck either way!
eadmund 9y ago
> Short of writing a bunch of custom scripts, is there a standard way to handle this situation that I assume anyone with multiple repos that share internal, private dependencies has? You write custom scripts to reïmplement everything you'd get for free with a monorepo! Having worked at organisations with a monorepo and with many repos, I can confidently say that any team which is using multiple repos is very probably wrong — and the more repos, the more likely wrong they are. If you have more repos than team members, you are almost certainly wrong. You end up spending far more time managing cross-repo dependencies and changes than you would merging changes in a monorepo. Multiple repos: not even once.
icebraining 9y ago
We used git submodules. The "app" repo didn't contain any code, just configuration and the submodules, which are essentially links to other repos (the app modules, in this case). Those libraries could be freely linked from multiple apps, of course. Disadvantage: it's not a package manager; it doesn't read all the dependencies from each library, resolve the duplicates and install them centrally. Instead of that, we only had submodules in the top repo (the app) - a bit like having a single requirements.txt file.

by mnm1 9y ago

peterevans 9y ago

Any changes you need to make to internal libraries you install via package manager would need to be made to those separate repositories that house the libraries, and then those changes would have a release made (generally this is done through a git tag, and using semantic versioning). Once you update the version, your package manager will allow you to install that update to the other places you're using it, so all you need to change is the package file. It might seem like a pain to do it this way, especially if you're rapidly iterating on a library—it might be that your library is not really mature, or even used by more than one repository, so it may not even make sense to have that library in a separate repository to begin with! But once you do have a mature code base, semantic versioning is a really sane way of managing dependency updates for the N number of other projects which use your library.

hinkley 9y ago
And how do you test if a change in one of these repos fixes the problem you’re seeing? This is what we’re failing at with our multirepo. That and resectioning code to split or combine responsibilities in different ways. Something a monorepo makes trivial.
- peterevans 9y ago
  Ideally, your library code that you're pulling in has some unit testing to demonstrate that things are working as they should be. (If not, consider adding unit testing! It's really useful!) If that is the case, then you can isolate the likelihood of an issue as either in the library (because unit tests fail there), or in the project consuming the library (because unit tests succeed in the library). Without testing, it's hard to have a ton of confidence in where the problem lies—which is exactly the problem you cite. And while a monorepo (or just a standalone repository with no separate libraries, which is frankly an easier setup to manage than monorepos or multirepos!) may make debugging a bit easier, it's not going to give you much more confidence in your code. Once your organization outgrows the paradigm where you just have N standalone projects with N completely separate code bases, and you do need to commit to either a multi-repo or a mono-repo configuration, it's really, really helpful to have unit testing to allow you to isolate where issues are.
  - hinkley 9y ago
    The class of problems I’m talking about are integration issues. Unit tests look good but when you put the pieces together... When the tests that matter cross version control boundaries you pay for it. Whether the costs outweigh the benefits is something you have to think about.
    - peterevans 9y ago
      What does your test coverage look like? Perhaps you're missing something there that would have caught that bug? Testing is, of course, no silver bullet. Tests are written by humans, and humans make mistakes—and it's pretty difficult to achieve 100% test coverage in a production system. The goal of testing is to have confidence in the code you've written. Tests often don't need to cross version control boundaries. You can use mock data—like would be produced by the library—on the consumer side, because you can delegate responsibility for testing of that library to the library repository itself. If your tests work great with the mock data, but things are still failing, then you can infer that the mock data and the actual data are different, and your bug is in the library.
chrisweekly 9y ago
Yes, tho I'd go further and say semver is table stakes -- and so is determinism (achieved on the library-consuming side by committing and maintaining the yarn.lock file, in the specific case of node projects).
mnm1 9y ago
No, semantic versioning and package management is really not a sane way to manage development. You're saying every time I make a code change, I should release a patch version BEFORE I can even test it, since I cannot test that dependency by itself. I'm sorry, but that's just insane, not to mention time consuming. And yes, my libraries are used by multiple apps. Why else would I need to split them out as dependencies? Actually, after thinking through this, the only solution I can think of is symlinks and those would work with a monorepo or multiple repos just as well.
- peterevans 9y ago
  Sorry—I just saw this! But, as it was, I was not saying that you should release a patch version every time you make a code change. That would indeed be pretty arduous! What you should do is test your library independently from the consumer that uses it. If that's not possible, ask yourself why that is; maybe this code should not be sequestered into a library after all, or maybe it needs to be designed a bit differently to remove some of the coupling between the library and app that seems to be at issue. Ideally, your testing in your app should not be testing that the library works; it should be testing that your use of the library works. Too bad the real world is much messier than the ideal world. Good luck either way!

eadmund 9y ago

> Short of writing a bunch of custom scripts, is there a standard way to handle this situation that I assume anyone with multiple repos that share internal, private dependencies has? You write custom scripts to reïmplement everything you'd get for free with a monorepo! Having worked at organisations with a monorepo and with many repos, I can confidently say that any team which is using multiple repos is very probably wrong — and the more repos, the more likely wrong they are. If you have more repos than team members, you are almost certainly wrong. You end up spending far more time managing cross-repo dependencies and changes than you would merging changes in a monorepo. Multiple repos: not even once.

icebraining 9y ago

We used git submodules. The "app" repo didn't contain any code, just configuration and the submodules, which are essentially links to other repos (the app modules, in this case). Those libraries could be freely linked from multiple apps, of course. Disadvantage: it's not a package manager; it doesn't read all the dependencies from each library, resolve the duplicates and install them centrally. Instead of that, we only had submodules in the top repo (the app) - a bit like having a single requirements.txt file.