6 ms·

Every monorepo I've ever met (n=3) has some kind of radioactive DMZ that everybody is afraid to touch because it's not clear who owns it but it is clear from it

Every monorepo I've ever met (n=3) has some kind of radioactive DMZ that everybody is afraid to touch because it's not clear who owns it but it is clear from its quality that you don't want to be the last person who touched it because then maybe somebody will think that you own it. It's usually called "core" or somesuch. Separate repos for each team means that when two teams own components that need to interact, they have to expose a "public" interface to the other team--which is the kind of disciplined engineering work that we should be striving for. The monorepo-alternative is that you solve it in the DMZ where it feels less like engineering and more like some kind of multiparty political endeavor where PR reviewers of dubious stakeholder status are using the exercise to further agendas which are unrelated to the feature except that it somehow proves them right about whatever architectural point is recently contentious. Plus, it's always harder to remove something from the DMZ than to add it, so it's always growing and there's this sort of gravitational attractor which, eventually starts warping time such that PR's take longer to merge the closer they are to it. Better to just do the "hard" work of maintaining versioned interfaces with documented compatibility (backed by tests). You can always decide to collapse your codebase into a black hole later--but once you start on that path you may never escape.

zaphar 2y ago
Since we are indulging in generalizations from our past. With separate repos you end up with 10 "cores" that are radioctive DMZ's everybody is afraid to touch. And those "disciplined" public API's will be universally hated by everyone who consumes them. Neither a monorepo nor separate repos will result in people being disciplined. If you already have the discipline to do separate repositories correctly then you'll be fine with a monorepo. So I guess it's six on one hand, half dozen in the other.
- __MatrixMan__ 2y ago
  No I think there's a clear difference. I've seen this several times: Somebody changes teams and now they're no longer responsible for a bit of code, but then they learn that it is broken in some way, and now they're sneaking in commits that--on paper--should now be handled by somebody else. Dev's *like* to feel ownership of reasonably sized chunks of code. We like to arrange it in ways that is pleasing for us to work on later down the road. And once we've made those investments, we like to see them pay off by making quick easy changes that make users happy. Sharing a small codebase with three or four other people and finding ways to make each other's lives easier while supporting it is *fun* and it makes for better code too. But it only stays fun if you have enough autonomy that you can really own it--you and your small team. Footguns introduced need to be pointed at your feet. Automation introduced needs to save you time. If you've got the preferences of 50 other people to consider, and you know that whatever you do you're going to piss off some 10 of them or another... the fun goes away. This is simple: > we own this whole repo and only this 10% of it (the public interface) needs to make external stakeholders happy, otherwise we just care about making each other happy. ...and it has no space in it for there to be any code which is not clearly owned by somebody. In a monorepo, there are plenty of places for that.
  - zaphar 2y ago
    Except that monorepos are just as capable of identifying and gatekeeping those spaces as separate repos are. If you aren't going to use those gatekeeping mechanisms in a mono repo I doubt that the quality of your separate repo codebases are going to be any better. You can't solve this problem with technology it's a cultural issue and like all cultural issues any technical constraints you put in place to control it will be just be worked around in ways that make the problem worse not better.
    - __MatrixMan__ 2y ago
      I don't doubt that the mechanisms, if used properly, are capable. My point is that the culture that tends to grow up around monorepos is one where those mechanisms aren't used properly. > If you aren't going to use those gatekeeping mechanisms in a mono repo I doubt that the quality of your separate repo codebases are going to be any better Those are pretty different competencies right? Getting the monorepo in good shape requires a broad understanding of what is where and who is who (and the participation of your neighbors). Making your repo a nice place for visitors to contribute to... you can learn to to that by emulating repos found anywhere.
      - zaphar 2y ago
        My own experience does not suggest that your last statement is true at all in practice.
  - 0x6c6f6c 2y ago
    > Dev's *like* to feel ownership of reasonably sized chunks of code. I just have to say this has not been my experience at all. A lot of developers want nothing more than to come in and write whatever lines of code they're told to and get paid. Ownership is often the last thing they want.
    - __MatrixMan__ 2y ago
      Yes well, extrinsic motivation works for some I guess. I work much better when I chose the task based on my team's vision of where we want to take the code and our understanding of the needs of the people involved. If the problem got so glaring that we weren't already on it by the time a manager was writing it down, we screwed up. It's a feeling that's much harder to kindle in a monorepo, too much red tape and politics, nothing at all like a sandbox. And I can't be the only one who feels this way. Otherwise why would there be a big pile of free open source software?
gorgoiler 2y ago
I would advise annotating your experience anecdotes, which are surely valuable, with some information about team size, company size, corporate structure (tech team in a non tech corp, vs pure tech), age etc. The meat is in the detail, I find.
- __MatrixMan__ 2y ago
  That's good advice, thanks. In general I'll try to include that at the top level, but the edit window has passed. In these cases the companies were all about 200 employees, something like five to ten teams with hands in the code. Average team size: five to ten. All SaaS companies.
Boxxed 2y ago
You can have well-defined interfaces without splitting it into many repos just like how you can have well-defined interfaces without splitting it into microservices. In fact, I've seen enough garbage to know that forcing the issue via these mechanisms just makes bad software worse.
mhh__ 2y ago
Code ownership is a scam
- __MatrixMan__ 2y ago
  Only when somebody else pushes it on you.
- mceachen 2y ago
  It can be, but the idea of having “code experts” (or people with the most battle scars with this chunk of code) can absolutely be helpful, especially if the team is sufficiently large (and people aren’t afraid to talk to other people for guidance).
The_Colonel 2y ago
> Separate repos for each team means that when two teams own components that need to interact, they have to expose a "public" interface to the other team--which is the kind of disciplined engineering work that we should be striving for. This pattern has its own set of problems. Strict ownerships separation creates strong dependencies and capacity constraints on the project organization. A single team is rarely able to deliver a complete feature, because it will mean changes in a couple of services. If you go with a model where teams are allowed to build features in "foreign" services, you will still come to the situation that the owning team doesn't feel that responsible for something they haven't built / don't really understand. You can tweak it, involve the owning team more etc. but it has trade-offs / friction. The worst anti-pattern coming from this is "we have dependency on service X / team Y, but their code reviews are usually very long and pedantic, threatening our delivery. Can we do it locally in our service instead?" which results in people putting stuff where it doesn't belong, for organisational reasons.
- __MatrixMan__ 2y ago
  I'm not saying that the ownership should be strict, you should feel free to submit PRs to my repo, and I to yours. I just don't want there to be a no-man's-land. And if there must be an every-mans-land, let it be explicitly so, not some pile of stuff that didn't fit anywhere else.
  - jatins 2y ago
    > you should feel free to submit PRs to my repo, and I to yours Haven't seen it happen in practice. For some reason a separate repo induces more of "not my backyard" feeling that a separate folder
    - __MatrixMan__ 2y ago
      Really? GitHub is packed with commits authored by people besides the maintainers. Have your really not noticed? Or do you just stop looking for a CONTRIBUTING.md when you're at work?
  - 2y ago
materielle 2y ago
The problem is that DRY principles have be cargo culted way past the point of being useful. People see two pieces of duplicate, or even similar code in a monorepo, and they feel the urge create some sort of shared directory to de duplicate them. The problem, is this introduces more edges to your graph, which over time increases the monorepo complexity exponentially. And who is going to maintain that shared directory? Even in a mono-repo, you should probably start off by copy and pasting code around, until you figure out how to design and find ownership for an actual well thought through public api. And maybe for some code, that never happens. Also, like many things, people cargo cult big tech ideas without understanding why they work. In Google, the monorepo has two key features: 1) every directory has an owner by default 2) the owner can control build target visibility lists. That means that the ownership team controls who depends on them. By default, everything is private. Basically, every directory should either be a project, owned by a team. Or some common library code, owned by the library maintainers. The other thing, is library maintainers need to be discerning with what APIs and code they support. Again, a common refrain I here is “when it doubt, it’s better to put it in the company’s common library to promote code reuse”. That’s the exact wrong way to think about it. Library teams should be very careful about what APIs they support because they will be on the hook maintaining them forever. This is all to say, I can see how monorepos fail. Because people cherry-pick the ideas they like, but ignore the important things which make them work at Google: 1) default code owners 2) visibility lists that default to private 3) discerning library teams that aggressively reject code additions 4) don’t be too dry, if it can’t be exposed in a public api, just copy and paste, just like you would in a microservice.
brap 2y ago
This seems like an org/process problem and not a tech problem. You can end up in this situation with or without a monorepo.
- __MatrixMan__ 2y ago
  Yes but it's harder to do by accident. If code is getting into the product at all it must be in somebody's repo?
deskr 2y ago
Any old codebase will have their own version of Nina, Katya, Masha or even just an open reactor, monorepo or not. I once worked on a codebase that had had a directory that no one was allowed to touch because it was "about to be decommissioned". When I looked into it more closely, it had been "about to be decommissioned" for 7 years and it was holding so many things up.
actinium226 2y ago
Even if people are disciplined about versioned interfaces with documented compatibility (which is something I've personally never seen, except perhaps in a small number of very high quality open source repos), you still end up with deployment headaches. Making any sort of backwards incompatible change requires a high level of coordination and opens up the potential for mistakes that break service. With a monorepo you can deploy a backwards incompatible change all at once, or at the very least you can see in a single commit/PR what the changes were, instead of split across multiple repos.
- __MatrixMan__ 2y ago
  You can achieve the same thing with a single commit that changes only a bunch of version pointers, or by moving that kind of thing into a space governed by feature flags--both of which are easier to write tests for. How do you write a test for the rollbackability of a commit, if that test must be in the commit which it asserts can be rolled back? Does the test clone a second copy of the monorepo at the older version so that it has access to the "before" state? The hard parts of backwards compatibility have to do with accumulated state over time, they don't go away because you've bound some versions together.
rddbs 2y ago
The problem with giving each team a repo and an API surface is that you create API boundaries where your organizational boundaries are, not necessary where your service boundaries are. And as your organizational structure evolves over time, your repo and API boundaries lag behind since it’s so difficult to make large scale shifts to the code.
- __MatrixMan__ 2y ago
  I don't see that as a bad thing. By creating granules you constrain that evolution such that nobody ends up with half of a thing. Without those boundaries, people who don't understand the code may motivate organizational structures which don't make sense. That's where I've been for a few months: The work of prior gatekeepers now run through the middle of what we're responsible for. It feels like we bought a house online and when we showed up the kitchen is in one country and the bathroom is in another so we have to clear customs several times a day and we have to hold a summit with the friends of the previous owner if we want to change anything--even things in the interior. The architect of the reorg would never have done this if the natural boundaries had been a bit more apparent, i.e. as a list of repos to be assigned to the new teams. I'd prefer large scale shifts to come by replacing an old repo with a new one (or one with two, or two with one, or by open sourcing one which you no longer care to maintain). Maybe that slows down the organizational rate of change, but if the alternative is pretending to have changed in a way which isn't actually sustainable, then maybe slowing down is better.
- hburd 2y ago
  There is a name for this phenomenon, it is called Conway's Law: https://en.wikipedia.org/wiki/Conway%27s_law https://en.wikipedia.org/wiki/Conway%27s_law More recent commentators have noted a corollary - for software projects with a long lifetime of code reuse, such as Microsoft Windows, the structure of the code mirrors not only the communication structure of the organization which created the most recent release, but also the communication structures of every previous team which worked on that code.

zaphar 2y ago

Since we are indulging in generalizations from our past. With separate repos you end up with 10 "cores" that are radioctive DMZ's everybody is afraid to touch. And those "disciplined" public API's will be universally hated by everyone who consumes them. Neither a monorepo nor separate repos will result in people being disciplined. If you already have the discipline to do separate repositories correctly then you'll be fine with a monorepo. So I guess it's six on one hand, half dozen in the other.

__MatrixMan__ 2y ago
No I think there's a clear difference. I've seen this several times: Somebody changes teams and now they're no longer responsible for a bit of code, but then they learn that it is broken in some way, and now they're sneaking in commits that--on paper--should now be handled by somebody else. Dev's *like* to feel ownership of reasonably sized chunks of code. We like to arrange it in ways that is pleasing for us to work on later down the road. And once we've made those investments, we like to see them pay off by making quick easy changes that make users happy. Sharing a small codebase with three or four other people and finding ways to make each other's lives easier while supporting it is *fun* and it makes for better code too. But it only stays fun if you have enough autonomy that you can really own it--you and your small team. Footguns introduced need to be pointed at your feet. Automation introduced needs to save you time. If you've got the preferences of 50 other people to consider, and you know that whatever you do you're going to piss off some 10 of them or another... the fun goes away. This is simple: > we own this whole repo and only this 10% of it (the public interface) needs to make external stakeholders happy, otherwise we just care about making each other happy. ...and it has no space in it for there to be any code which is not clearly owned by somebody. In a monorepo, there are plenty of places for that.
- zaphar 2y ago
  Except that monorepos are just as capable of identifying and gatekeeping those spaces as separate repos are. If you aren't going to use those gatekeeping mechanisms in a mono repo I doubt that the quality of your separate repo codebases are going to be any better. You can't solve this problem with technology it's a cultural issue and like all cultural issues any technical constraints you put in place to control it will be just be worked around in ways that make the problem worse not better.
  - __MatrixMan__ 2y ago
    I don't doubt that the mechanisms, if used properly, are capable. My point is that the culture that tends to grow up around monorepos is one where those mechanisms aren't used properly. > If you aren't going to use those gatekeeping mechanisms in a mono repo I doubt that the quality of your separate repo codebases are going to be any better Those are pretty different competencies right? Getting the monorepo in good shape requires a broad understanding of what is where and who is who (and the participation of your neighbors). Making your repo a nice place for visitors to contribute to... you can learn to to that by emulating repos found anywhere.
    - zaphar 2y ago
      My own experience does not suggest that your last statement is true at all in practice.
- 0x6c6f6c 2y ago
  > Dev's *like* to feel ownership of reasonably sized chunks of code. I just have to say this has not been my experience at all. A lot of developers want nothing more than to come in and write whatever lines of code they're told to and get paid. Ownership is often the last thing they want.
  - __MatrixMan__ 2y ago
    Yes well, extrinsic motivation works for some I guess. I work much better when I chose the task based on my team's vision of where we want to take the code and our understanding of the needs of the people involved. If the problem got so glaring that we weren't already on it by the time a manager was writing it down, we screwed up. It's a feeling that's much harder to kindle in a monorepo, too much red tape and politics, nothing at all like a sandbox. And I can't be the only one who feels this way. Otherwise why would there be a big pile of free open source software?

gorgoiler 2y ago

I would advise annotating your experience anecdotes, which are surely valuable, with some information about team size, company size, corporate structure (tech team in a non tech corp, vs pure tech), age etc. The meat is in the detail, I find.

__MatrixMan__ 2y ago
That's good advice, thanks. In general I'll try to include that at the top level, but the edit window has passed. In these cases the companies were all about 200 employees, something like five to ten teams with hands in the code. Average team size: five to ten. All SaaS companies.

Boxxed 2y ago

You can have well-defined interfaces without splitting it into many repos just like how you can have well-defined interfaces without splitting it into microservices. In fact, I've seen enough garbage to know that forcing the issue via these mechanisms just makes bad software worse.

mhh__ 2y ago

Code ownership is a scam

__MatrixMan__ 2y ago
Only when somebody else pushes it on you.
mceachen 2y ago
It can be, but the idea of having “code experts” (or people with the most battle scars with this chunk of code) can absolutely be helpful, especially if the team is sufficiently large (and people aren’t afraid to talk to other people for guidance).

The_Colonel 2y ago

> Separate repos for each team means that when two teams own components that need to interact, they have to expose a "public" interface to the other team--which is the kind of disciplined engineering work that we should be striving for. This pattern has its own set of problems. Strict ownerships separation creates strong dependencies and capacity constraints on the project organization. A single team is rarely able to deliver a complete feature, because it will mean changes in a couple of services. If you go with a model where teams are allowed to build features in "foreign" services, you will still come to the situation that the owning team doesn't feel that responsible for something they haven't built / don't really understand. You can tweak it, involve the owning team more etc. but it has trade-offs / friction. The worst anti-pattern coming from this is "we have dependency on service X / team Y, but their code reviews are usually very long and pedantic, threatening our delivery. Can we do it locally in our service instead?" which results in people putting stuff where it doesn't belong, for organisational reasons.

__MatrixMan__ 2y ago
I'm not saying that the ownership should be strict, you should feel free to submit PRs to my repo, and I to yours. I just don't want there to be a no-man's-land. And if there must be an every-mans-land, let it be explicitly so, not some pile of stuff that didn't fit anywhere else.
- jatins 2y ago
  > you should feel free to submit PRs to my repo, and I to yours Haven't seen it happen in practice. For some reason a separate repo induces more of "not my backyard" feeling that a separate folder
  - __MatrixMan__ 2y ago
    Really? GitHub is packed with commits authored by people besides the maintainers. Have your really not noticed? Or do you just stop looking for a CONTRIBUTING.md when you're at work?
- 2y ago

materielle 2y ago

The problem is that DRY principles have be cargo culted way past the point of being useful. People see two pieces of duplicate, or even similar code in a monorepo, and they feel the urge create some sort of shared directory to de duplicate them. The problem, is this introduces more edges to your graph, which over time increases the monorepo complexity exponentially. And who is going to maintain that shared directory? Even in a mono-repo, you should probably start off by copy and pasting code around, until you figure out how to design and find ownership for an actual well thought through public api. And maybe for some code, that never happens. Also, like many things, people cargo cult big tech ideas without understanding why they work. In Google, the monorepo has two key features: 1) every directory has an owner by default 2) the owner can control build target visibility lists. That means that the ownership team controls who depends on them. By default, everything is private. Basically, every directory should either be a project, owned by a team. Or some common library code, owned by the library maintainers. The other thing, is library maintainers need to be discerning with what APIs and code they support. Again, a common refrain I here is “when it doubt, it’s better to put it in the company’s common library to promote code reuse”. That’s the exact wrong way to think about it. Library teams should be very careful about what APIs they support because they will be on the hook maintaining them forever. This is all to say, I can see how monorepos fail. Because people cherry-pick the ideas they like, but ignore the important things which make them work at Google: 1) default code owners 2) visibility lists that default to private 3) discerning library teams that aggressively reject code additions 4) don’t be too dry, if it can’t be exposed in a public api, just copy and paste, just like you would in a microservice.

brap 2y ago

This seems like an org/process problem and not a tech problem. You can end up in this situation with or without a monorepo.

__MatrixMan__ 2y ago
Yes but it's harder to do by accident. If code is getting into the product at all it must be in somebody's repo?

deskr 2y ago

Any old codebase will have their own version of Nina, Katya, Masha or even just an open reactor, monorepo or not. I once worked on a codebase that had had a directory that no one was allowed to touch because it was "about to be decommissioned". When I looked into it more closely, it had been "about to be decommissioned" for 7 years and it was holding so many things up.

actinium226 2y ago

Even if people are disciplined about versioned interfaces with documented compatibility (which is something I've personally never seen, except perhaps in a small number of very high quality open source repos), you still end up with deployment headaches. Making any sort of backwards incompatible change requires a high level of coordination and opens up the potential for mistakes that break service. With a monorepo you can deploy a backwards incompatible change all at once, or at the very least you can see in a single commit/PR what the changes were, instead of split across multiple repos.

__MatrixMan__ 2y ago
You can achieve the same thing with a single commit that changes only a bunch of version pointers, or by moving that kind of thing into a space governed by feature flags--both of which are easier to write tests for. How do you write a test for the rollbackability of a commit, if that test must be in the commit which it asserts can be rolled back? Does the test clone a second copy of the monorepo at the older version so that it has access to the "before" state? The hard parts of backwards compatibility have to do with accumulated state over time, they don't go away because you've bound some versions together.

rddbs 2y ago

The problem with giving each team a repo and an API surface is that you create API boundaries where your organizational boundaries are, not necessary where your service boundaries are. And as your organizational structure evolves over time, your repo and API boundaries lag behind since it’s so difficult to make large scale shifts to the code.

__MatrixMan__ 2y ago
I don't see that as a bad thing. By creating granules you constrain that evolution such that nobody ends up with half of a thing. Without those boundaries, people who don't understand the code may motivate organizational structures which don't make sense. That's where I've been for a few months: The work of prior gatekeepers now run through the middle of what we're responsible for. It feels like we bought a house online and when we showed up the kitchen is in one country and the bathroom is in another so we have to clear customs several times a day and we have to hold a summit with the friends of the previous owner if we want to change anything--even things in the interior. The architect of the reorg would never have done this if the natural boundaries had been a bit more apparent, i.e. as a list of repos to be assigned to the new teams. I'd prefer large scale shifts to come by replacing an old repo with a new one (or one with two, or two with one, or by open sourcing one which you no longer care to maintain). Maybe that slows down the organizational rate of change, but if the alternative is pretending to have changed in a way which isn't actually sustainable, then maybe slowing down is better.
hburd 2y ago
There is a name for this phenomenon, it is called Conway's Law: https://en.wikipedia.org/wiki/Conway%27s_law https://en.wikipedia.org/wiki/Conway%27s_law More recent commentators have noted a corollary - for software projects with a long lifetime of code reuse, such as Microsoft Windows, the structure of the code mirrors not only the communication structure of the organization which created the most recent release, but also the communication structures of every previous team which worked on that code.