5 ms·

Maybe I'm not cool enough to understand this, but I don't see the draw for monorepos. Imagine if you're a tool owner, and you want to make a change that present

by whack 8y ago

Maybe I'm not cool enough to understand this, but I don't see the draw for monorepos. Imagine if you're a tool owner, and you want to make a change that presents significant improvements for 99.9% of people, but causes significant problems for 0.1% of your users. In a versioned world, you can release your change as a new version, and allow your users to self-select if/when/how they want to migrate to the new version. But in a monorepo, you have to either trample over the 0.1%, or let the 0.1% hold everyone else hostage. Conversely, imagine if you're using some tools developed by a far off team within the company. Every time the tooling team decides to make a change, it will immediately and irrevocably propagate into your stack, whether you like it or not. If you were at a startup and had a production critical project, would you hardcode specific versions for all your dependencies, and carefully test everything before moving to newer versions? Or would you just set everything to LATEST and hope that none of your dependencies decide to break you the next day? Working with a monorepo is essentially like the latter.

perfunctory 8y ago
> Working with a monorepo is essentially like the latter. Not really. In the dependencies analogy the author of the dependency has no way to test the dependee(s). While with monorepo this is exactly what you do, "the tooling team" will "carefully test everything" before "propagate into your stack" (and it doesn't have to be irrevocable).
- whack 8y ago
  In practice, at any medium/large organization, the tooling team doesn't know your system, and its nuances, nearly well enough to "carefully test everything". Having a solid automated test suite does help. But I personally would like to be in control of when my project updates its dependencies, instead of being forced to always pull everything from LATEST.
  - perfunctory 8y ago
    > the tooling team doesn't know your system they don't know it because they don't use monorepo. monorepo makes "solid automated tests" easier since basically there is only one version to test. The instinct against pulling everything from LATEST, developed in the traditional world, is perfectly understandable. However in monorepo "your" project is also tooling-team's project. "being forced" becomes "being helped". It's shared responsibility.
  - joshuamorton 8y ago
    You are, by having tests. At Google the contract is essentially infrastructure teams (and generally, your dependencies) will not break your unit tests (or will contact you well in advance to handle changes). But if you don't have a test, they might. They don't have to carefully test everything. You do. And if you don't, breakages are entirely your responsibility, because you didn't have a test for them.
IMAYousaf 8y ago
Hello. What sort of tooling differences would one expect for a monorepo vs. multiple repos? Is that a factor of something intrinsic about having one big repo, or is that a factor of the scale of the type of organization that Google is? Thanks.
anyfoo 8y ago
I've worked both at Google (only as an intern, though) and at other very very big companies with gargantuan code bases. At that scale, with software that is constantly in flux, pretty much the last thing you want is having to keep compatibility between several versions of a component. It's bad enough if you have to do it for external reasons, but if the only reason is so that "others in the company have a choice" then... no, just no. You might think this ought to be trivial by having clear API contracts, but that's a) not how things work in practice if all code is effectively owned by the same, overarching entity and, more importantly, b) now you have an enormous effort to transition between incompatible API revisions instead of just being able to do lockstep changes, for no real gain. Even if you manage to pull that off (again, for what benefit?), it will bite you that 1.324.2564 behaves subtly different from 1.324.5234 even though the intent was just to add a new option and they otherwise ought to have no extensional changes in behavior.
- whack 8y ago
  It took me a while to figure out that you're disagreeing with me, because your last paragraph is a perfect example of why monorepos are so dangerous. Imagine a tooling team on a different continent that makes some changes this afternoon. Like you said, their intent is just to add a new option, and it ought to have no extensional changes in behavior, but it still ends up behaving subtly different. The next morning, all your services end up broken as a result. In a versioned world, you can still freeze your dependency at 1.324.5234, and migrate only when you want to, and when you're feeling confident about it. In a monorepo world, you don't have a choice. You've been forcefully migrated as soon as the tooling team decides to make the change on their end. They had the best of intentions, but that doesn't always translate to a good outcome. FWIW, I'm currently working at a large famous company that uses a monorepo. Color me not-impressed. I do think that having a single repository for an entire team/project is a good idea. Hundreds of different projects and teams who've never seen one another? Not so much.
  - shub 8y ago
    > Like you said, their intent is just to add a new option, and it ought to have no extensional changes in behavior, but it still ends up behaving subtly different. The next morning, all your services end up broken as a result. Someone makes a commit to library code and production magically breaks? How does that happen?
perfunctory 8y ago
>In a versioned world, you can release your change as a new version, and allow your users to self-select Repeat this process multiple times and you end up with configuration/settings hell. Been there done that. It's not black and white but "trampling over the 0.1%" could be a sensible business/architectural decision. For example how do you imagine "google maps" users selecting when/how to migrate?
- whack 8y ago
  I was referring only to static dependencies, like Guava for example. Static dependencies don't require ongoing "upkeep", so you really should allow your users to use an older version of your library/code, if that's what they really want to do. When it comes to live services that you're actually running on a daily basis, like Google maps, forcing users to migrate makes a lot more sense.
  - perfunctory 8y ago
    I see. I guess static dependencies are not as prone to ongoing upkeep as services, but they are not totally immune. Think Heartbleed.
    - whack 8y ago
      Just because your older versions are available to users, doesn't mean you have to continuously maintain them. For example, I know plenty of teams still using Java 7, even though Oracle has EOLed it. The teams know the risk that they are taking, and have the flexibility to decide when they want to migrate to a newer version, and that's how it should be. I would never want to live in a world where Oracle can force me to migrate to Java 10 within a day of release.
  - googlemike 8y ago
    What do you define as a static dependency?
- deleted 8y ago
  [deleted]
andrewfong 8y ago
Not saying this is how Google does it, but a monorepo doesn't prevent you from having multiple versions of the same dependency. Ideally, with a monorepo, you could update 99% of your sub-packages to the latest version while still leaving the one alone.
- rossjudson 8y ago
  There are a few exceptions to the "one version" rule; the monorepo and the build system support multiple versions just fine. It's just that we don't want them.
crazygringo 8y ago
> But in a monorepo, you have to either trample over the 0.1%, or let the 0.1% hold everyone else hostage. Nope. In a monorepo (like at Google), you're responsible for not breaking anyone else's code, as evidenced by their tests still passing. So you never trample over the 0.1%. Instead you fix your code, or you fix their code for them -- which was probably due to your own bugs or undefined behavior in the first place. Or else you don't push. And if you break their code because they didn't have tests? That's their problem, and better for them to learn their lesson sooner that later, because they're breaking engineering standards that they've been told since the day they joined. A monorepo depends, fundamentally, on all code having complete test coverage.
- u801e 8y ago
  > So you never trample over the 0.1%. Instead you fix your code, or you fix their code for them -- which was probably due to your own bugs or undefined behavior in the first place. Or else you don't push. Given the size of a monrepo, is it possible to run the entire test suite in one's development environment, or do they have another endpoint to push to to run tests on a dedicated server?
  - YokoZar 8y ago
    > Given the size of a monrepo, is it possible to run the entire test suite in one's development environment, or do they have another endpoint to push to to run tests on a dedicated server? Eventually you hit a point where you need systems to run the tests for you. Making this work is part of the investment in infrastructure and tooling you need to do as a big serious company.
  - summerlight 8y ago
    Google has a CI infrastructure which runs most of the affected tests for each commit (which they call "CL") on thousand of machines in parallel. Though even for Google, running the entire test suite every time is prohibitively expensive so they have a way to merge and run multiple CLs in a single batch run every 3 hours, which is useful for testing a CL that may affect hundreds of thousands of build/test targets. If you're interested, this paper may give you an idea how Google is doing test. https://static.googleusercontent.com/media/research.google.com/ko//pubs/archive/45861.pdf https://static.googleusercontent.com/media/research.google.c...

by whack 8y ago

perfunctory 8y ago

> Working with a monorepo is essentially like the latter. Not really. In the dependencies analogy the author of the dependency has no way to test the dependee(s). While with monorepo this is exactly what you do, "the tooling team" will "carefully test everything" before "propagate into your stack" (and it doesn't have to be irrevocable).

whack 8y ago
In practice, at any medium/large organization, the tooling team doesn't know your system, and its nuances, nearly well enough to "carefully test everything". Having a solid automated test suite does help. But I personally would like to be in control of when my project updates its dependencies, instead of being forced to always pull everything from LATEST.
- perfunctory 8y ago
  > the tooling team doesn't know your system they don't know it because they don't use monorepo. monorepo makes "solid automated tests" easier since basically there is only one version to test. The instinct against pulling everything from LATEST, developed in the traditional world, is perfectly understandable. However in monorepo "your" project is also tooling-team's project. "being forced" becomes "being helped". It's shared responsibility.
- joshuamorton 8y ago
  You are, by having tests. At Google the contract is essentially infrastructure teams (and generally, your dependencies) will not break your unit tests (or will contact you well in advance to handle changes). But if you don't have a test, they might. They don't have to carefully test everything. You do. And if you don't, breakages are entirely your responsibility, because you didn't have a test for them.

IMAYousaf 8y ago

Hello. What sort of tooling differences would one expect for a monorepo vs. multiple repos? Is that a factor of something intrinsic about having one big repo, or is that a factor of the scale of the type of organization that Google is? Thanks.

anyfoo 8y ago

I've worked both at Google (only as an intern, though) and at other very very big companies with gargantuan code bases. At that scale, with software that is constantly in flux, pretty much the last thing you want is having to keep compatibility between several versions of a component. It's bad enough if you have to do it for external reasons, but if the only reason is so that "others in the company have a choice" then... no, just no. You might think this ought to be trivial by having clear API contracts, but that's a) not how things work in practice if all code is effectively owned by the same, overarching entity and, more importantly, b) now you have an enormous effort to transition between incompatible API revisions instead of just being able to do lockstep changes, for no real gain. Even if you manage to pull that off (again, for what benefit?), it will bite you that 1.324.2564 behaves subtly different from 1.324.5234 even though the intent was just to add a new option and they otherwise ought to have no extensional changes in behavior.

whack 8y ago
It took me a while to figure out that you're disagreeing with me, because your last paragraph is a perfect example of why monorepos are so dangerous. Imagine a tooling team on a different continent that makes some changes this afternoon. Like you said, their intent is just to add a new option, and it ought to have no extensional changes in behavior, but it still ends up behaving subtly different. The next morning, all your services end up broken as a result. In a versioned world, you can still freeze your dependency at 1.324.5234, and migrate only when you want to, and when you're feeling confident about it. In a monorepo world, you don't have a choice. You've been forcefully migrated as soon as the tooling team decides to make the change on their end. They had the best of intentions, but that doesn't always translate to a good outcome. FWIW, I'm currently working at a large famous company that uses a monorepo. Color me not-impressed. I do think that having a single repository for an entire team/project is a good idea. Hundreds of different projects and teams who've never seen one another? Not so much.
- shub 8y ago
  > Like you said, their intent is just to add a new option, and it ought to have no extensional changes in behavior, but it still ends up behaving subtly different. The next morning, all your services end up broken as a result. Someone makes a commit to library code and production magically breaks? How does that happen?

perfunctory 8y ago

>In a versioned world, you can release your change as a new version, and allow your users to self-select Repeat this process multiple times and you end up with configuration/settings hell. Been there done that. It's not black and white but "trampling over the 0.1%" could be a sensible business/architectural decision. For example how do you imagine "google maps" users selecting when/how to migrate?

whack 8y ago
I was referring only to static dependencies, like Guava for example. Static dependencies don't require ongoing "upkeep", so you really should allow your users to use an older version of your library/code, if that's what they really want to do. When it comes to live services that you're actually running on a daily basis, like Google maps, forcing users to migrate makes a lot more sense.
- perfunctory 8y ago
  I see. I guess static dependencies are not as prone to ongoing upkeep as services, but they are not totally immune. Think Heartbleed.
  - whack 8y ago
    Just because your older versions are available to users, doesn't mean you have to continuously maintain them. For example, I know plenty of teams still using Java 7, even though Oracle has EOLed it. The teams know the risk that they are taking, and have the flexibility to decide when they want to migrate to a newer version, and that's how it should be. I would never want to live in a world where Oracle can force me to migrate to Java 10 within a day of release.
- googlemike 8y ago
  What do you define as a static dependency?
deleted 8y ago
[deleted]

andrewfong 8y ago

Not saying this is how Google does it, but a monorepo doesn't prevent you from having multiple versions of the same dependency. Ideally, with a monorepo, you could update 99% of your sub-packages to the latest version while still leaving the one alone.

rossjudson 8y ago
There are a few exceptions to the "one version" rule; the monorepo and the build system support multiple versions just fine. It's just that we don't want them.

crazygringo 8y ago

> But in a monorepo, you have to either trample over the 0.1%, or let the 0.1% hold everyone else hostage. Nope. In a monorepo (like at Google), you're responsible for not breaking anyone else's code, as evidenced by their tests still passing. So you never trample over the 0.1%. Instead you fix your code, or you fix their code for them -- which was probably due to your own bugs or undefined behavior in the first place. Or else you don't push. And if you break their code because they didn't have tests? That's their problem, and better for them to learn their lesson sooner that later, because they're breaking engineering standards that they've been told since the day they joined. A monorepo depends, fundamentally, on all code having complete test coverage.

u801e 8y ago
> So you never trample over the 0.1%. Instead you fix your code, or you fix their code for them -- which was probably due to your own bugs or undefined behavior in the first place. Or else you don't push. Given the size of a monrepo, is it possible to run the entire test suite in one's development environment, or do they have another endpoint to push to to run tests on a dedicated server?
- YokoZar 8y ago
  > Given the size of a monrepo, is it possible to run the entire test suite in one's development environment, or do they have another endpoint to push to to run tests on a dedicated server? Eventually you hit a point where you need systems to run the tests for you. Making this work is part of the investment in infrastructure and tooling you need to do as a big serious company.
- summerlight 8y ago
  Google has a CI infrastructure which runs most of the affected tests for each commit (which they call "CL") on thousand of machines in parallel. Though even for Google, running the entire test suite every time is prohibitively expensive so they have a way to merge and run multiple CLs in a single batch run every 3 hours, which is useful for testing a CL that may affect hundreds of thousands of build/test targets. If you're interested, this paper may give you an idea how Google is doing test. https://static.googleusercontent.com/media/research.google.com/ko//pubs/archive/45861.pdf https://static.googleusercontent.com/media/research.google.c...