5 ms·

Any changes you need to make to internal libraries you install via package manager would need to be made to those separate repositories that house the libraries

Any changes you need to make to internal libraries you install via package manager would need to be made to those separate repositories that house the libraries, and then those changes would have a release made (generally this is done through a git tag, and using semantic versioning). Once you update the version, your package manager will allow you to install that update to the other places you're using it, so all you need to change is the package file. It might seem like a pain to do it this way, especially if you're rapidly iterating on a library—it might be that your library is not really mature, or even used by more than one repository, so it may not even make sense to have that library in a separate repository to begin with! But once you do have a mature code base, semantic versioning is a really sane way of managing dependency updates for the N number of other projects which use your library.

hinkley 9y ago
And how do you test if a change in one of these repos fixes the problem you’re seeing? This is what we’re failing at with our multirepo. That and resectioning code to split or combine responsibilities in different ways. Something a monorepo makes trivial.
- peterevans 9y ago
  Ideally, your library code that you're pulling in has some unit testing to demonstrate that things are working as they should be. (If not, consider adding unit testing! It's really useful!) If that is the case, then you can isolate the likelihood of an issue as either in the library (because unit tests fail there), or in the project consuming the library (because unit tests succeed in the library). Without testing, it's hard to have a ton of confidence in where the problem lies—which is exactly the problem you cite. And while a monorepo (or just a standalone repository with no separate libraries, which is frankly an easier setup to manage than monorepos or multirepos!) may make debugging a bit easier, it's not going to give you much more confidence in your code. Once your organization outgrows the paradigm where you just have N standalone projects with N completely separate code bases, and you do need to commit to either a multi-repo or a mono-repo configuration, it's really, really helpful to have unit testing to allow you to isolate where issues are.
  - hinkley 9y ago
    The class of problems I’m talking about are integration issues. Unit tests look good but when you put the pieces together... When the tests that matter cross version control boundaries you pay for it. Whether the costs outweigh the benefits is something you have to think about.
    - peterevans 9y ago
      What does your test coverage look like? Perhaps you're missing something there that would have caught that bug? Testing is, of course, no silver bullet. Tests are written by humans, and humans make mistakes—and it's pretty difficult to achieve 100% test coverage in a production system. The goal of testing is to have confidence in the code you've written. Tests often don't need to cross version control boundaries. You can use mock data—like would be produced by the library—on the consumer side, because you can delegate responsibility for testing of that library to the library repository itself. If your tests work great with the mock data, but things are still failing, then you can infer that the mock data and the actual data are different, and your bug is in the library.
      - hinkley 9y ago
        For instance, I have a piece of code in a particularly gnarly modules that at this moment is disabled and has been for two sprints due to emergent behavior. First sprint it had adequate unit tests but not enough functional tests to exhibit a problem. Second sprint I fixed the testing deficiency and got the code to work end to end. Or so I thought. The first time I turned it on it preprod I couldn't turn it back off again because some piece of data that came from five function calls away was being shared, and nobody who participated in the PR recalled that fact. Most of the code I'm dealing with is in a single module. I have been chipping away at fixing the insane ball of mud as I can. My coworkers often aren't that lucky. They come to me for advice on how to deal with this sort of problem but crossing 2 or three modules. There's no low-friction way for them to fix any of this. They can't just refactor because of the coordination costs, and also the loss of historical information when you move a block of code across module boundaries or try to change module boundaries. This is the prime argument for monorepos in the literature - not making irreversible decisions on Law of Demeter problems. It's not my biggest reason, but it's sufficient for most people.
- idunno246 9y ago
  back in the day, eclipse would handle this fine - you normally have the jars, but if you need to edit the underlying project you can tell it that jar is this folder on disk(which is a different repo), and it loads it correctly as another project in the workspace replacing the jar.
- jayd16 9y ago
  Some (most?) package managers allow you to install to your local cache for local testing. Nuget and Maven allow this at least.
chrisweekly 9y ago
Yes, tho I'd go further and say semver is table stakes -- and so is determinism (achieved on the library-consuming side by committing and maintaining the yarn.lock file, in the specific case of node projects).
mnm1 9y ago
No, semantic versioning and package management is really not a sane way to manage development. You're saying every time I make a code change, I should release a patch version BEFORE I can even test it, since I cannot test that dependency by itself. I'm sorry, but that's just insane, not to mention time consuming. And yes, my libraries are used by multiple apps. Why else would I need to split them out as dependencies? Actually, after thinking through this, the only solution I can think of is symlinks and those would work with a monorepo or multiple repos just as well.
- peterevans 9y ago
  Sorry—I just saw this! But, as it was, I was not saying that you should release a patch version every time you make a code change. That would indeed be pretty arduous! What you should do is test your library independently from the consumer that uses it. If that's not possible, ask yourself why that is; maybe this code should not be sequestered into a library after all, or maybe it needs to be designed a bit differently to remove some of the coupling between the library and app that seems to be at issue. Ideally, your testing in your app should not be testing that the library works; it should be testing that your use of the library works. Too bad the real world is much messier than the ideal world. Good luck either way!

hinkley 9y ago

And how do you test if a change in one of these repos fixes the problem you’re seeing? This is what we’re failing at with our multirepo. That and resectioning code to split or combine responsibilities in different ways. Something a monorepo makes trivial.

peterevans 9y ago
Ideally, your library code that you're pulling in has some unit testing to demonstrate that things are working as they should be. (If not, consider adding unit testing! It's really useful!) If that is the case, then you can isolate the likelihood of an issue as either in the library (because unit tests fail there), or in the project consuming the library (because unit tests succeed in the library). Without testing, it's hard to have a ton of confidence in where the problem lies—which is exactly the problem you cite. And while a monorepo (or just a standalone repository with no separate libraries, which is frankly an easier setup to manage than monorepos or multirepos!) may make debugging a bit easier, it's not going to give you much more confidence in your code. Once your organization outgrows the paradigm where you just have N standalone projects with N completely separate code bases, and you do need to commit to either a multi-repo or a mono-repo configuration, it's really, really helpful to have unit testing to allow you to isolate where issues are.
- hinkley 9y ago
  The class of problems I’m talking about are integration issues. Unit tests look good but when you put the pieces together... When the tests that matter cross version control boundaries you pay for it. Whether the costs outweigh the benefits is something you have to think about.
  - peterevans 9y ago
    What does your test coverage look like? Perhaps you're missing something there that would have caught that bug? Testing is, of course, no silver bullet. Tests are written by humans, and humans make mistakes—and it's pretty difficult to achieve 100% test coverage in a production system. The goal of testing is to have confidence in the code you've written. Tests often don't need to cross version control boundaries. You can use mock data—like would be produced by the library—on the consumer side, because you can delegate responsibility for testing of that library to the library repository itself. If your tests work great with the mock data, but things are still failing, then you can infer that the mock data and the actual data are different, and your bug is in the library.
    - hinkley 9y ago
      For instance, I have a piece of code in a particularly gnarly modules that at this moment is disabled and has been for two sprints due to emergent behavior. First sprint it had adequate unit tests but not enough functional tests to exhibit a problem. Second sprint I fixed the testing deficiency and got the code to work end to end. Or so I thought. The first time I turned it on it preprod I couldn't turn it back off again because some piece of data that came from five function calls away was being shared, and nobody who participated in the PR recalled that fact. Most of the code I'm dealing with is in a single module. I have been chipping away at fixing the insane ball of mud as I can. My coworkers often aren't that lucky. They come to me for advice on how to deal with this sort of problem but crossing 2 or three modules. There's no low-friction way for them to fix any of this. They can't just refactor because of the coordination costs, and also the loss of historical information when you move a block of code across module boundaries or try to change module boundaries. This is the prime argument for monorepos in the literature - not making irreversible decisions on Law of Demeter problems. It's not my biggest reason, but it's sufficient for most people.
idunno246 9y ago
back in the day, eclipse would handle this fine - you normally have the jars, but if you need to edit the underlying project you can tell it that jar is this folder on disk(which is a different repo), and it loads it correctly as another project in the workspace replacing the jar.
jayd16 9y ago
Some (most?) package managers allow you to install to your local cache for local testing. Nuget and Maven allow this at least.

chrisweekly 9y ago

Yes, tho I'd go further and say semver is table stakes -- and so is determinism (achieved on the library-consuming side by committing and maintaining the yarn.lock file, in the specific case of node projects).

mnm1 9y ago

No, semantic versioning and package management is really not a sane way to manage development. You're saying every time I make a code change, I should release a patch version BEFORE I can even test it, since I cannot test that dependency by itself. I'm sorry, but that's just insane, not to mention time consuming. And yes, my libraries are used by multiple apps. Why else would I need to split them out as dependencies? Actually, after thinking through this, the only solution I can think of is symlinks and those would work with a monorepo or multiple repos just as well.

peterevans 9y ago
Sorry—I just saw this! But, as it was, I was not saying that you should release a patch version every time you make a code change. That would indeed be pretty arduous! What you should do is test your library independently from the consumer that uses it. If that's not possible, ask yourself why that is; maybe this code should not be sequestered into a library after all, or maybe it needs to be designed a bit differently to remove some of the coupling between the library and app that seems to be at issue. Ideally, your testing in your app should not be testing that the library works; it should be testing that your use of the library works. Too bad the real world is much messier than the ideal world. Good luck either way!