6 ms·

Monorepo shortcomings 1 and 2 seem like bullshit to me. Perforce, the popular monorepo at most companies I've worked at, supports access control. Monorepos do

Monorepo shortcomings 1 and 2 seem like bullshit to me. Perforce, the popular monorepo at most companies I've worked at, supports access control. Monorepos do not prevent you from segmenting your code into modules and pushing binary/source packages into source control so that builds can avoid compiling everything(TiVo used to do this, and it worked well when you got the hang of it). I feel like these debates are often fueled by false arguments. Either way you go, you're going to want to build support tools and processes to tailor your VCS to your local needs.

jayd16 7y ago
This is all true BUT I think the monorepo as described here is the act of treating all your projects as directly referencing each other. Sure you could just use a manyrepo style of dependency tracking in a monorepo but I think that's not exactly what the author is exploring.
- jyounker 7y ago
  > This is all true BUT I think the monorepo as described here is the act of treating all your projects as directly referencing each other. From what I read that is a correct assessment. What the OP is proposing is something of a strawman argument. No advocate of monorepos I've ever met believe that a monorepo should imply a monolith. Generally they're advocating monorepos in order to develop microservices faster, and with less effort. Using a monorepo and the associated tooling side steps the pain that comes from complicated CI, the difficulty of sharing code, the difficulties of non-atomic cross-repo reviews, and the difficulties of making multi-app refactorings.
weberc2 7y ago
Can you elaborate on “monorepos do not prevent you from checking packages into source control” and how that helps to avoid recompiling everything? Why would you check a package into source control anyway? Surely source control is for source code? And I lean toward monorepos, btw, but there are still lots of obstacles and monorepo proponents don’t tend to acknowledge them or offer clear suggestions for how to solve or workaround them.
- jayd16 7y ago
  You can use something like a shared binary repo such as maven or you could just check in dependencies and not worry about an external server being available for builds. >Surely source control is for source code? This is just pedantry. Checking in binaries is a pragmatic solution that solves a lot of problems.
  - weberc2 7y ago
    I wasn’t trying to be a pedant, I’ve just never heard of anyone doing this. I was wondering how it helped solve the problem of not rebuilding everything.
    - jayd16 7y ago
      In short, the binaries are already built. Usually its faster to link to a prebuilt binary than to build from scratch.
      - weberc2 7y ago
        So where do these binaries get built and how does the system know which binaries to rebuild for a given change? If developers are building binaries and committing them directly, doesn’t that open up security or even correctness issues? How does this approach satisfy compliance concerns (how can the CTO or a manager sign off on the changes that went into the binary if it’s just something a random developer committed?)? How does this scale to tens of deployments per day? These are hard monorepo problems, and they keep being handwaved away.
        CuriousSkeptic 7y ago
        Not sure how people do this in practice. But in principle it seems rather straight forward. A compiler is just a program that takes some input and create some output. Both the compiler and the input can have a cryptographically secure hash. Putting both in a sealed box, like a docker image, with its own hash, gives you a program that takes no input and produces some output. If the box changes, run it in a trusted machine and save the output together with a signed declaration of which box version produced it
jsnell 7y ago
VCS access control are the wrong tool for solving the "people use code they shouldn't" complaint. First, VCS ACLs will massively reduce the benefits you're supposed to get from a monorepo. How will you do global refactors in that kind of a situation? How does a maintainer of a library figure out how the clients are actually using it? (The clients must have visibility into the library, but the opposite it unlikely to be true.) Second, let's say that I maintain a library with a supported public interface that's implemented in terms of an internal interface that nobody's supposed to use. How will VCS ACLs allow me to hide the implementation but not the interface? When they kick off a build, the compiler needs to be able to read the implementation parts to actually build the library. It can't be that the clients have access to read the headers but then link against a pre-build binary blob. At that point you don't have a monorepo, you've got multirepos stored in a monorepo. The actual solution are build system ACLs. Not ACLs for people, but ACLs for projects. Anyone can read the code, but you can say "only source files in directory X can include this header" or "only build files in directory Y can link against this object file".
- alexhutcheson 7y ago
  Bazel has this via the 'visibility' attribute on packages and build rules: https://docs.bazel.build/versions/master/skylark/build-style.html#visibility https://docs.bazel.build/versions/master/skylark/build-style...
- pshirshov 7y ago
  > VCS access control are the wrong tool for solving the "people use code they shouldn't" complaint. I agree > The actual solution are build system ACLs. Or, maybe, better languages enforcing better design. In most of the cases artifacts and libraries are not related to the domain, engineers create them just to establish artificial boundaries between code components, isolate irrelated things, enforce encapsulation and avoid accidental mixing of metalanguages. It would be lot better to have a smart compiler for this. A tool which can prevent us from mixing different abstraction layers, creating unneccessary horizontal links between our components, etc, etc. I have a couple of ideas how such a thing may look like.
- 01100011 7y ago
pshirshov 7y ago
> Monorepo shortcomings 1 and 2 seem like bullshit to me. It's a blogpost and the author didn't try to build a total and exhaustive formal system. These shortcomings are not absolute truth but actually they are true. I've seen this multiple times: a small projects evolves over years into a monster. Engineers add new components and reuse any other components they may need creating horizontal links. At some point they feel like they lost their productivity and they blame monorepo because it's easy to create horizontal links in a typical monorepo. So, they try to build a multirepo flow and they spend a lot of effort, time and money trying to make it working. At some point they feel that their productivity is even worse than it was before because now they need to orchestrate things so they merge everything back. Same applies not only to VCS flows, but to system design as well. When we discuss monolith/microservices controversy all the monorepo/multirepo arguments may be isomorphically translated to that domain. What is better, monolithic app or a bunch of microservices? A role-based app of course: https://github.com/7mind/slides/blob/master/02-roles/target/roles.pdf https://github.com/7mind/slides/blob/master/02-roles/target/...
- jyounker 7y ago
  Monorepo/multirepo and monolith/microservice are orthogonal concepts. When organizations don't understand that then they may end up building a distributed monolith in across multiple repos. (The "Distributed Big Ball of Mud" anti-pattern.) Monorepo advocates are typically advocating for microservices, but within a single code base. The way you provide access control is through code review and build system visibility. In order to modify another group's code you require their approval on the review for that section of the code base. (Using mechanisms like github/gitlab owners files or rules within upsource.) This still means that if one group needs to make extensive changes to another groups code, the path of least resistance may be to fork it into your own group's section of the repo. Build tools provide another point of control. If you're using a tool like bazel, the way you link to a component in another portion of the repo is through target names. The only targets your code will have access to are those that the owners has declared as being available for external builds.
  - pshirshov 7y ago
jumpingmice 7y ago
All of the supposed flaws of a monorepo in this article are actually flaws of git. This is a very common phenomenon. I often joke there are two kinds of developers: those who prefer monorepos and those who have never used perforce.

jayd16 7y ago

This is all true BUT I think the monorepo as described here is the act of treating all your projects as directly referencing each other. Sure you could just use a manyrepo style of dependency tracking in a monorepo but I think that's not exactly what the author is exploring.

jyounker 7y ago
> This is all true BUT I think the monorepo as described here is the act of treating all your projects as directly referencing each other. From what I read that is a correct assessment. What the OP is proposing is something of a strawman argument. No advocate of monorepos I've ever met believe that a monorepo should imply a monolith. Generally they're advocating monorepos in order to develop microservices faster, and with less effort. Using a monorepo and the associated tooling side steps the pain that comes from complicated CI, the difficulty of sharing code, the difficulties of non-atomic cross-repo reviews, and the difficulties of making multi-app refactorings.

weberc2 7y ago

Can you elaborate on “monorepos do not prevent you from checking packages into source control” and how that helps to avoid recompiling everything? Why would you check a package into source control anyway? Surely source control is for source code? And I lean toward monorepos, btw, but there are still lots of obstacles and monorepo proponents don’t tend to acknowledge them or offer clear suggestions for how to solve or workaround them.

jayd16 7y ago
You can use something like a shared binary repo such as maven or you could just check in dependencies and not worry about an external server being available for builds. >Surely source control is for source code? This is just pedantry. Checking in binaries is a pragmatic solution that solves a lot of problems.
- weberc2 7y ago
  I wasn’t trying to be a pedant, I’ve just never heard of anyone doing this. I was wondering how it helped solve the problem of not rebuilding everything.
  - jayd16 7y ago
    In short, the binaries are already built. Usually its faster to link to a prebuilt binary than to build from scratch.
    - weberc2 7y ago
      So where do these binaries get built and how does the system know which binaries to rebuild for a given change? If developers are building binaries and committing them directly, doesn’t that open up security or even correctness issues? How does this approach satisfy compliance concerns (how can the CTO or a manager sign off on the changes that went into the binary if it’s just something a random developer committed?)? How does this scale to tens of deployments per day? These are hard monorepo problems, and they keep being handwaved away.
      - CuriousSkeptic 7y ago
        Not sure how people do this in practice. But in principle it seems rather straight forward. A compiler is just a program that takes some input and create some output. Both the compiler and the input can have a cryptographically secure hash. Putting both in a sealed box, like a docker image, with its own hash, gives you a program that takes no input and produces some output. If the box changes, run it in a trusted machine and save the output together with a signed declaration of which box version produced it

jsnell 7y ago

VCS access control are the wrong tool for solving the "people use code they shouldn't" complaint. First, VCS ACLs will massively reduce the benefits you're supposed to get from a monorepo. How will you do global refactors in that kind of a situation? How does a maintainer of a library figure out how the clients are actually using it? (The clients must have visibility into the library, but the opposite it unlikely to be true.) Second, let's say that I maintain a library with a supported public interface that's implemented in terms of an internal interface that nobody's supposed to use. How will VCS ACLs allow me to hide the implementation but not the interface? When they kick off a build, the compiler needs to be able to read the implementation parts to actually build the library. It can't be that the clients have access to read the headers but then link against a pre-build binary blob. At that point you don't have a monorepo, you've got multirepos stored in a monorepo. The actual solution are build system ACLs. Not ACLs for people, but ACLs for projects. Anyone can read the code, but you can say "only source files in directory X can include this header" or "only build files in directory Y can link against this object file".

alexhutcheson 7y ago
Bazel has this via the 'visibility' attribute on packages and build rules: https://docs.bazel.build/versions/master/skylark/build-style.html#visibility https://docs.bazel.build/versions/master/skylark/build-style...
pshirshov 7y ago
> VCS access control are the wrong tool for solving the "people use code they shouldn't" complaint. I agree > The actual solution are build system ACLs. Or, maybe, better languages enforcing better design. In most of the cases artifacts and libraries are not related to the domain, engineers create them just to establish artificial boundaries between code components, isolate irrelated things, enforce encapsulation and avoid accidental mixing of metalanguages. It would be lot better to have a smart compiler for this. A tool which can prevent us from mixing different abstraction layers, creating unneccessary horizontal links between our components, etc, etc. I have a couple of ideas how such a thing may look like.
01100011 7y ago

pshirshov 7y ago

> Monorepo shortcomings 1 and 2 seem like bullshit to me. It's a blogpost and the author didn't try to build a total and exhaustive formal system. These shortcomings are not absolute truth but actually they are true. I've seen this multiple times: a small projects evolves over years into a monster. Engineers add new components and reuse any other components they may need creating horizontal links. At some point they feel like they lost their productivity and they blame monorepo because it's easy to create horizontal links in a typical monorepo. So, they try to build a multirepo flow and they spend a lot of effort, time and money trying to make it working. At some point they feel that their productivity is even worse than it was before because now they need to orchestrate things so they merge everything back. Same applies not only to VCS flows, but to system design as well. When we discuss monolith/microservices controversy all the monorepo/multirepo arguments may be isomorphically translated to that domain. What is better, monolithic app or a bunch of microservices? A role-based app of course: https://github.com/7mind/slides/blob/master/02-roles/target/roles.pdf https://github.com/7mind/slides/blob/master/02-roles/target/...

jyounker 7y ago
Monorepo/multirepo and monolith/microservice are orthogonal concepts. When organizations don't understand that then they may end up building a distributed monolith in across multiple repos. (The "Distributed Big Ball of Mud" anti-pattern.) Monorepo advocates are typically advocating for microservices, but within a single code base. The way you provide access control is through code review and build system visibility. In order to modify another group's code you require their approval on the review for that section of the code base. (Using mechanisms like github/gitlab owners files or rules within upsource.) This still means that if one group needs to make extensive changes to another groups code, the path of least resistance may be to fork it into your own group's section of the repo. Build tools provide another point of control. If you're using a tool like bazel, the way you link to a component in another portion of the repo is through target names. The only targets your code will have access to are those that the owners has declared as being available for external builds.
- pshirshov 7y ago

jumpingmice 7y ago

All of the supposed flaws of a monorepo in this article are actually flaws of git. This is a very common phenomenon. I often joke there are two kinds of developers: those who prefer monorepos and those who have never used perforce.