3 ms·

for repo in ls repos do thing done ?

by Groxx 9y ago

joshuamorton 9y ago
Diamond dependency like issues crop up. Imagine I have Repos A,B,C. A is a base repo. B and C depend on A, and C also depends on B. If I modify some API in A, and also update all the callsites in B and C, I also have to bump the version of A depended on by B and C, and also bump the version of B that C depends on, otherwise I'll get version mismatch/api compatibility breakages. To make this work that means that nothing can depend on latest, everything has to have frozen dependencies, and you either need to manually, or via some system, globally track all of the dependencies across repos, and atomically update all of them on every breaking change. In other words, you reinvent blaze/bazel at the repo level instead of the target level, and you have to add an additional tool that makes sure you're dependencies can never get mismatched. The monorepo sidesteps this issue by saying "everything must always build against latest".
- Groxx 9y ago
  "everything must always build against latest" is perfectly enforceable on multirepo too, it's just that nobody does it.
  - joshuamorton 9y ago
    >"everything must always build against latest" is perfectly enforceable on multirepo too, it's just that nobody does it No, you cannot. That's my entire point. Here's a minimal example: Repo one contains one file, provider.py: def five(): return 5 Repo two contains one file, consumer.py. import provider # assume path magic makes this work def test_five_is_produced(): assert provider.five() == 5 if __name__ == '__main__': test_five_is_produced() I also have an external build script that copies provider and consumer, from origin/master/HEAD into the same directory, and runs `python consumer.py`. Now I want to change `five` to actually be `number`, such that `number(n) == n`, ie. I really want a more generic impl. What sequence of changes can I commit such that tests will always pass, at any point in time? There is no way to atomically update both provider and consumer. There will be some period of time, perhaps only milliseconds, but some period of time, at which point I can run my build script and it will pick up incompatible versions of the two files. This is a reductive example, but the function `five` in this case takes the role of a more complex API of some kind.
    - Groxx 9y ago
      or you give your CI the ability to read transaction markers in your git repo. e.g. add a tag that says "must have [repo] at [sha]+". dependency management basically. you can even do this after the commits are created, so you can allow cycles and not just diamonds. but yes, cross-project commits are dramatically easier in a monorepo, I entirely agree with that - they essentially come "for free".
      - joshuamorton 9y ago
        Didn't you just reinvent versioning and frozen dependencies? What you described is not always building at latest, it's building at latest except when there are issues at which point you don't build at latest and instead build at a known good version. Consequences of this are, for example, that you cannot run all affected tests at every commit.
        Groxx 9y ago
        sure. I honestly don't see why that's a problem though, especially since "at every commit" can have clear markers for if it's expected to be buildable or not. My point here is that you're describing a known problem with known solutions, and saying it's impossible. I'm saying it requires work, as does all this in a monorepo. edit: to be technical: yes, you're correct, it can't always build at latest at every instant. Agreed. I don't see why that's necessary though. Simplifying, sure; necessary? No.
        joshuamorton 9y ago
        >sure. I honestly don't see why that's a problem though, especially since "at every commit" can have clear markers for if it's expected to be buildable or not. The value from this is the ability to always know exactly which thing caused which problem. If you know things are broken now, you can bisect from the last known good state, and find the change that introduced a breakage. With multi-repo, you can't do that, since it's not always a single change that introduces a breakage, but a combination. Ensuring that everything always builds at latest allows you to do a bunch of really cool magical bisection tricks. If you don't have that, you can't bisect to find breakages or regressions, because your "bisection" is 1. now 2 dimensional instead of 1 2. may/will have many false positives That puts you in a really rough spot when there's a breakage and you don't have the institutional knowledge to know what broke it.

for repo in ls repos do thing done ?

by Groxx 9y ago

for repo in ls repos do thing done ?

Diamond dependency like issues crop up. Imagine I have Repos A,B,C. A is a base repo. B and C depend on A, and C also depends on B. If I modify some API in A, and also update all the callsites in B and C, I also have to bump the version of A depended on by B and C, and also bump the version of B that C depends on, otherwise I'll get version mismatch/api compatibility breakages. To make this work that means that nothing can depend on latest, everything has to have frozen dependencies, and you either need to manually, or via some system, globally track all of the dependencies across repos, and atomically update all of them on every breaking change. In other words, you reinvent blaze/bazel at the repo level instead of the target level, and you have to add an additional tool that makes sure you're dependencies can never get mismatched. The monorepo sidesteps this issue by saying "everything must always build against latest".

Groxx 9y ago
"everything must always build against latest" is perfectly enforceable on multirepo too, it's just that nobody does it.
- joshuamorton 9y ago
  >"everything must always build against latest" is perfectly enforceable on multirepo too, it's just that nobody does it No, you cannot. That's my entire point. Here's a minimal example: Repo one contains one file, provider.py: def five(): return 5 Repo two contains one file, consumer.py. import provider # assume path magic makes this work def test_five_is_produced(): assert provider.five() == 5 if __name__ == '__main__': test_five_is_produced() I also have an external build script that copies provider and consumer, from origin/master/HEAD into the same directory, and runs `python consumer.py`. Now I want to change `five` to actually be `number`, such that `number(n) == n`, ie. I really want a more generic impl. What sequence of changes can I commit such that tests will always pass, at any point in time? There is no way to atomically update both provider and consumer. There will be some period of time, perhaps only milliseconds, but some period of time, at which point I can run my build script and it will pick up incompatible versions of the two files. This is a reductive example, but the function `five` in this case takes the role of a more complex API of some kind.
  - Groxx 9y ago
    or you give your CI the ability to read transaction markers in your git repo. e.g. add a tag that says "must have [repo] at [sha]+". dependency management basically. you can even do this after the commits are created, so you can allow cycles and not just diamonds. but yes, cross-project commits are dramatically easier in a monorepo, I entirely agree with that - they essentially come "for free".
    - joshuamorton 9y ago
      Didn't you just reinvent versioning and frozen dependencies? What you described is not always building at latest, it's building at latest except when there are issues at which point you don't build at latest and instead build at a known good version. Consequences of this are, for example, that you cannot run all affected tests at every commit.
      - Groxx 9y ago
        sure. I honestly don't see why that's a problem though, especially since "at every commit" can have clear markers for if it's expected to be buildable or not. My point here is that you're describing a known problem with known solutions, and saying it's impossible. I'm saying it requires work, as does all this in a monorepo. edit: to be technical: yes, you're correct, it can't always build at latest at every instant. Agreed. I don't see why that's necessary though. Simplifying, sure; necessary? No.
        joshuamorton 9y ago
        >sure. I honestly don't see why that's a problem though, especially since "at every commit" can have clear markers for if it's expected to be buildable or not. The value from this is the ability to always know exactly which thing caused which problem. If you know things are broken now, you can bisect from the last known good state, and find the change that introduced a breakage. With multi-repo, you can't do that, since it's not always a single change that introduces a breakage, but a combination. Ensuring that everything always builds at latest allows you to do a bunch of really cool magical bisection tricks. If you don't have that, you can't bisect to find breakages or regressions, because your "bisection" is 1. now 2 dimensional instead of 1 2. may/will have many false positives That puts you in a really rough spot when there's a breakage and you don't have the institutional knowledge to know what broke it.