5 ms·

Monorepos are changing how teams build software

8 points by gk1 5y ago

zaphar 5y ago
I love the concept of monorepos. With the proper tooling and support I think they can be force multipliers for high functioning engineering organizations. However almost none of the publicly available tooling for CI/CD, IDEs, Source code hosting, or any of the other force multipliers out there support it properly, so for most organizations I think it's a wash. It's a shame really since if they did support monorepos better or even just stop assuming too much about the repository then a whole lot of things could be unlocked for engineering teams. As it stands though going the monorepo route means you'll be building a lot of bespoke tooling to support it.
- MuffinFlavored 5y ago
  > With the proper tooling and support Could you elaborate/provide some examples?
  - Jensson 5y ago
    Google is the main example, their monorepo setup works and scales really well. It is basically a remote filesystem directory that automatically checks out the files you need to build your project and not other files.
    - akvadrako 5y ago
      Why would that be needed except for huge codebases at megacorps?
      - pxue 5y ago
        no one needs this except maybe google. even then if they started today with git maybe they wouldn't have used a monorepo. same as pointing to netflix and arguing you need 100+ microservices. no one needs this, maybe except netflix
        zaphar 5y ago
        Using a monorepo was a deliberate choice at Google. They modified their source control solution to support a monorepo every time it failed to scale sufficiently. If they had started with Git those same people would probably have modified Git to make it work as a monorepo. There are specific economies of scale Google gets with a monorepo that help it keep up with the sheer amount of cruft and technical debt they generate everyday. Google writes a lot of code and ensuring that security fixes get applied or legacy deprecated APIs get removed is for some people a full time job. Having a monorepo makes that job actually achievable rather than impossible. Source: I worked there and was familiar with many of the engineers working on the tooling for the monorepo.
  - zaphar 5y ago
    In a monorepo you usually only want to build portions of the codebase at any given time in your CI/CD. Most of the CI/CD tooling out there are accidentally hostile to this. As just one example it is often difficult to impossible to rebuild your code when a dependency changes. Build systems often don't expose this information usefully, so you'll find yourself either hand rolling something that looks at git changes and tries to infer what should be rebuilt and what shouldn't. Or you just rebuild everything in the entire repo every night. Or you can adopt Bazel everywhere which can be a hard sell in many companies but does give you better tools. Another example: Most source code hosting products don't give good support for path based permissions which you need in order to ensure that folks with the right context review code before it is merged in. Which pushes you to segregate your repositories along ownership and responsibility lines.
Mikeb85 5y ago
Are monorepos cool again? Dunno, all my software development experience is in monorepos (Rails, Gamedev). Or simply one-off scripts...
pythonbase 5y ago
From single PHP file to cover both back-end & front-end to multiple layers & repos to monorepo. Another full circle in SDLC.
JustLurking2022 5y ago
Having worked in companies with large monorepos and those without, I would never advise implementing one. The complexity as a function of repo size grows at something greater than linear rate. Code becomes tightly intertwined and the dependency graph becomes unwieldy, making refactoring a difficult proposition. Eventually, the codebase always grows to a size that makes it impractical to replicate on each developers' machine, so you start having to implement workarounds, and you realize that just keeping basic build operations working requires a substantial team to support. For the big players, they can eat the cost no problem, but it's still not money well spent and certainly not advisable for orgs without that level of resourcing.
- tmerr 5y ago
  > Code becomes tightly intertwined and the dependency graph becomes unwieldy One way to combat this is to use a build system with a notion of visibility, https://bazel.build/concepts/visibility https://bazel.build/concepts/visibility I think you might be over-extrapolating from your experience, but it's true going off the well-trodden road leads to surprises. And not that many smaller companies know the ins and outs of monorepos.
  - JustLurking2022 5y ago
    I have... as mentioned, worked in one of the large tech companies with a monorepo, and they all use fairly similar tooling. In practice, it doesn't solve the problem because most people default to making everything public. Nominally this avoids duplicating logic, but may result in more stuff being pulled in than was really necessary. These days, when NodeJS developers discuss how bad things have gotten with dependency management, I just laugh.
  - tored 5y ago
    How would you handle versioning in a monorepo? Just a new directory for major versions? If we have multiple teams you can’t really refactor other teams code and sometimes you need to do breaking changes thus I imagine that some versioning must exist.
smitty1e 5y ago
Ours is essentially a dumping ground, a disorganized network share. Oh, how great it would be to make the problem smaller.
ohCh6zos 5y ago
I've worked at companies with monorepos and it pushed me pretty hard in the opposite direction. My philosophy has been to go through the packaging and dependency process early and often, and split off as many things into their open repo as possible.
- tored 5y ago
  Any specific reason you came to that conclusion when working with monorepos?
  - ohCh6zos 5y ago
    I'm a bit scatterbrained on this topic, maybe I should sit down and try to organize my thoughts. Every company I've been at that has used monorepos has fallen into a hell of dependencies where changes break things unpredictably. The other big issue I've seen is companies with monorepos ended up not trusting their engineers. Everyone being able to see everything results in lowest common denominator code, and style fights that would never matter without one big repo.

8 points by gk1 5y ago

I love the concept of monorepos. With the proper tooling and support I think they can be force multipliers for high functioning engineering organizations. However almost none of the publicly available tooling for CI/CD, IDEs, Source code hosting, or any of the other force multipliers out there support it properly, so for most organizations I think it's a wash. It's a shame really since if they did support monorepos better or even just stop assuming too much about the repository then a whole lot of things could be unlocked for engineering teams. As it stands though going the monorepo route means you'll be building a lot of bespoke tooling to support it.

MuffinFlavored 5y ago
> With the proper tooling and support Could you elaborate/provide some examples?
- Jensson 5y ago
  Google is the main example, their monorepo setup works and scales really well. It is basically a remote filesystem directory that automatically checks out the files you need to build your project and not other files.
  - akvadrako 5y ago
    Why would that be needed except for huge codebases at megacorps?
    - pxue 5y ago
      no one needs this except maybe google. even then if they started today with git maybe they wouldn't have used a monorepo. same as pointing to netflix and arguing you need 100+ microservices. no one needs this, maybe except netflix
      - zaphar 5y ago
        Using a monorepo was a deliberate choice at Google. They modified their source control solution to support a monorepo every time it failed to scale sufficiently. If they had started with Git those same people would probably have modified Git to make it work as a monorepo. There are specific economies of scale Google gets with a monorepo that help it keep up with the sheer amount of cruft and technical debt they generate everyday. Google writes a lot of code and ensuring that security fixes get applied or legacy deprecated APIs get removed is for some people a full time job. Having a monorepo makes that job actually achievable rather than impossible. Source: I worked there and was familiar with many of the engineers working on the tooling for the monorepo.
- zaphar 5y ago
  In a monorepo you usually only want to build portions of the codebase at any given time in your CI/CD. Most of the CI/CD tooling out there are accidentally hostile to this. As just one example it is often difficult to impossible to rebuild your code when a dependency changes. Build systems often don't expose this information usefully, so you'll find yourself either hand rolling something that looks at git changes and tries to infer what should be rebuilt and what shouldn't. Or you just rebuild everything in the entire repo every night. Or you can adopt Bazel everywhere which can be a hard sell in many companies but does give you better tools. Another example: Most source code hosting products don't give good support for path based permissions which you need in order to ensure that folks with the right context review code before it is merged in. Which pushes you to segregate your repositories along ownership and responsibility lines.

Mikeb85 5y ago

Are monorepos cool again? Dunno, all my software development experience is in monorepos (Rails, Gamedev). Or simply one-off scripts...

pythonbase 5y ago

From single PHP file to cover both back-end & front-end to multiple layers & repos to monorepo. Another full circle in SDLC.

JustLurking2022 5y ago

Having worked in companies with large monorepos and those without, I would never advise implementing one. The complexity as a function of repo size grows at something greater than linear rate. Code becomes tightly intertwined and the dependency graph becomes unwieldy, making refactoring a difficult proposition. Eventually, the codebase always grows to a size that makes it impractical to replicate on each developers' machine, so you start having to implement workarounds, and you realize that just keeping basic build operations working requires a substantial team to support. For the big players, they can eat the cost no problem, but it's still not money well spent and certainly not advisable for orgs without that level of resourcing.

tmerr 5y ago
> Code becomes tightly intertwined and the dependency graph becomes unwieldy One way to combat this is to use a build system with a notion of visibility, https://bazel.build/concepts/visibility https://bazel.build/concepts/visibility I think you might be over-extrapolating from your experience, but it's true going off the well-trodden road leads to surprises. And not that many smaller companies know the ins and outs of monorepos.
- JustLurking2022 5y ago
  I have... as mentioned, worked in one of the large tech companies with a monorepo, and they all use fairly similar tooling. In practice, it doesn't solve the problem because most people default to making everything public. Nominally this avoids duplicating logic, but may result in more stuff being pulled in than was really necessary. These days, when NodeJS developers discuss how bad things have gotten with dependency management, I just laugh.
- tored 5y ago
  How would you handle versioning in a monorepo? Just a new directory for major versions? If we have multiple teams you can’t really refactor other teams code and sometimes you need to do breaking changes thus I imagine that some versioning must exist.

smitty1e 5y ago

Ours is essentially a dumping ground, a disorganized network share. Oh, how great it would be to make the problem smaller.

ohCh6zos 5y ago

I've worked at companies with monorepos and it pushed me pretty hard in the opposite direction. My philosophy has been to go through the packaging and dependency process early and often, and split off as many things into their open repo as possible.

tored 5y ago
Any specific reason you came to that conclusion when working with monorepos?
- ohCh6zos 5y ago
  I'm a bit scatterbrained on this topic, maybe I should sit down and try to organize my thoughts. Every company I've been at that has used monorepos has fallen into a hell of dependencies where changes break things unpredictably. The other big issue I've seen is companies with monorepos ended up not trusting their engineers. Everyone being able to see everything results in lowest common denominator code, and style fights that would never matter without one big repo.