11 ms·

A monorepo misconception – atomic cross-project commits

102 points by askl 5y ago

majikandy 5y ago
Interesting idea and it certainly makes sense to ensure working software at all times. However it doesn’t have to be one or the other. Sometimes a single commit (or PR if you prefer multiple commits) can update the api and all clients at the same time. Sometimes client callers are external to the repo/company. In which case backward compatibility strategy is needed. There is no need to abandon a concept of a single repo just because you might not use one of the main benefits all the time.
w_t_payne 5y ago
Either way, you've got to do the tooling work to make your chosen approach work. There's no free lunch here.
marceloabsousa 5y ago
Mostly agree with the article. To me the problem is about dependency management. I see all the time codebases hugely fragmented at the level of git which is totally ad-hoc. After a while, teams face a lot of issues the most annoying being one change in the product involving N PRs with N code reviews and N CIs. This fragmentation of knowledge also pop-ups in weird integration bugs that could be solved with better tool enforcing clearer processes. The ultimate goal of Reviewpad (https://reviewpad.com https://reviewpad.com) is to allow the benefits of the monorepo approach independently of how the codebase is fragmented at the git level. We are starting from the perspective of code review and debugging (e.g. code reviews with multiple PRs, or code reviews across multiple projects). For people doing microservices like us, the ability to have a single code review for both library and clients has been quite positive so far.
lhnz 5y ago
> It's particularly easy to see that the "atomic changes > across the whole repo" story is rubbish when you move > away from libraries, and also consider code that has > any kind of more complicated deployment lifecycle, > for example the interactions between services and > client binaries that communicate over an RPC interface. This seems exactly wrong to me. Getting rid of complicated deployment lifecycles is exactly the job that people use monorepos to solve. Wasn't that one of the reasons they are used at Google, Facebook, etc? As well as being able to do large-scale refactors in one place, of course. You should be able to merge a PR to cause a whole system to deploy: clients, backend services, database schemas, libraries, etc. This doesn't preclude wanting to break up commits into meaningful chunks or to add non-breaking-change migration patterns into libraries -- but consider this: is it meaningful for a change to be broken into separate commits just because it is being done to independent services? What benefit does cutting commits this way give you? What you want to avoid is needing to do separate PRs into many backend service and client repos, since: (1) when the review is split into 10+ places it's easier for reviewers to miss problems arising due to integration, (2) needing to avoid breaking changes can sometimes require developers to follow multi-stage upgrade processes that are so difficult that they cause mistakes, and (3) when there are separate PRs into different repositories these tend to start independent CI processes that will not test the system as-it-will-be (unless you test in production or have a very good E2E suite -- which would be a good idea in this situation). I will say that, even in a monorepo, a big change might still happen gradually behind feature flags. But I think that generally it's nice to be able to deploy breaking changes in a more atomic fashion.
- nicative 5y ago
  Even within the same repo, it is very likely that the old version of your code will coexist with the new version during the deploy roll out. Often having different commits and deploys is a requirement. For instance, imagine that you add a column in the database and also is using it in the backend service. You probably have add the column first and then commit and deploy the usage of the column later, because you can't easily guarantee that the new code won't be used before the column is added. Same would apply for a new field in the API contract.
  - thundergolfer 5y ago
    The old version won't coexist when the change is contained within a single binary, which seems like it would be true in a bunch of cases. In our monorepo we have to treat database changes with care, like you mention, as well as HTTP client/server API changes, but a bunch of stuff can be refactored cleanly without concern for backwards compatibility.
    - withinboredom 5y ago
      Do you only have a single instance of the binary running across the whole org? And during deployment to you stop the running instance before starting the new one?
      - jvolkman 5y ago
        Any change that won't cross deployable binary boundaries (think docker container) can be made atomically without care about subsequent deployment schedules. So this doesn't work for DB changes or client/server API changes as mentioned by OP, but does work for changes to shared libraries that get packaged into the deployment artifacts. For example, changing the interface in an internal shared library, or updating the version of a shared external library.
        kohlerm 5y ago
        Seems like a common misconception to me that people seem to believe that you can never change an interface. You actually can as long as it is not published to be used outside of your repositories.
argonaut 5y ago
There are plenty of smaller changes that are just way easier with a single commit. A small API call with just two or three consumers, a documentation change, etc. The multistage approach is certainly good for big changes.
- stkdump 5y ago
  Also, if you do all changes at once, you can also run the whole testsuite and find problems quicker. Sure, you could still later split the work up into many commits. But even when problems come up that you didn't see in the tests, it is easier to revert the work if it indeed is a single commit. The real reason why I still sometimes prefer many small commits is to reduce the chance of merge conflicts.
  - kohlerm 5y ago
    You can do automatic bisecting https://git-scm.com/docs/git-bisect https://git-scm.com/docs/git-bisect which you cannot easily do with multiple repos
cjfd 5y ago
The arguments seem exceedingly weak here. Having a monorepo does not prevent a workflow like (1) add new function signature; (2) let all calls use the new function signature; (3) remove old function signature. However, in some cases a multirepo makes sure you could only do it this way. If a function is called all over the place the three-step program is the only somewhat safe way to do it but if a function is called not exceedingly many times one can do the atomic refactor, which is an easier process. That one sometimes needs the three-step process and sometimes can do an atomic refactor is in no way contradictory as this article seems to claim. It depends on the size of the change and the risk involve what you would do in a particular case. Also, a monorepo forcing trunk-based development is great. Long-living feature branches are hell. I would even say to avoid feature branches entirely and use feature switches instead whenever one can get away with it. Every branch always runs the risk of making refactoring more difficult.
- Cthulhu_ 5y ago
  Re: branches, I agree in principle - we've had feature branches at my current job that were open for a year, reasoning being that it impacts a lot of our application core and we cannot permit issues impacting existing customers, etc etc. But I do like short-lived (max 1 sprint) branches for my own smaller scale projects because they group changes together. I name my branches after an issue number, rebase freely on that branch, and merge with a `--no-ff` flag when I'm done. My history is a neat branch / work / merge construct. Not sure if this is just fear, but I believe trunk- and feature switch based history will end up an incoherent mess of multiple projects being worked on at the same time, commits passing through each other. I'm sure they can be filtered out, but still.
- kohlerm 5y ago
  It is not the monorepo which forces trunkbased development it is the fact that you use one build for the monorepo. E.g. you can do trunk based development with multiple repos by forcing everyone to use a common build. You could also use a monorepo and still everyone does their own build consuming libraries via mechanisms such as Mavens Repositories or published npm packages
bob1029 5y ago
It looks like more effective change control is a major concern of the author. We use a monorepo for our organization and have found that feature flags are the best way to manage the problem of exposing new functionality in ways that won't piss off our customers. We can let them tell us when they are ready and we flip the switch. Once a flag goes true for every customer, make a note to drop it. This is important because these things will accumulate if you embrace this ideology.
Cthulhu_ 5y ago
The API changes are mainly a problem (I think) if you have a monorepo, but not a mono-deploy; you can only get the full benefits of a monorepo if all your changes are deployed simultaneously and atomically. Changing a model that is shared between different services (or a client / server) should be atomic. In practice, you need to think about backwards compatibility, and work in the three-step process outlined in the article (build and deprecate, switch over, remove deprecated code across a number of deployments / time). If you don't have atomic deployments, that's one less argument in favor of monorepos.
- deleted 5y ago
  [deleted]
- jyounker 5y ago
  You're using one specific definition of an API. Another definition of an API is the contract between a library and the calling code. The latter sort of API changes happen frequently (at least in places with monorepos), and they don't require phased deployments.
thundergolfer 5y ago
> The example of renaming a function with thousands of callers, for example, is probably better handled by just temporarily aliasing the function, or by temporarily defining the new function in terms of the old. Why exactly? The best I can think of is that you may annoy people because they'd need to rebase/merge after your change lands and takes away a function they were using. If you're literally just doing a rename, and you're using something like Java, why not just go ahead and do a global rename?
- jsnell 5y ago
  Because actually getting that commit pushed won't be just clicking "rename method". You'll need to run and pass the presubmits of every single project you made code changes in - and the more projects you're changing at once, the less likely it is that the build is green for all of them at the same time. Then you'll need to get code review approvals from the code owners of each of the clients in one go. Hopefully no new users pop up during this process. If some do, you'll need to iterate again. Then once some trivial refactoring inevitably causes some kind of breakage ("oh, somebody was using reflection on this class"), you'll need to revert the change. That'll be another set of full code reviews from every single owner. Let's hope that in the meanwhile, nobody pushed any commits that depend on your new code. None of this is a problem if you have a library and two clients. But the story being told is not "we can safely make changes in three projects at once", it's "we can safely make changes in hundreds or thousands of projects". The former is kind of uninteresting. The latter is a fairy tale.
  - jvolkman 5y ago
    The latter does happen at Google at least. But it requires tooling to reliably make the change (monorepo-aware refactoring tools, basically), and tooling to run all affected tests. Such large changes at Google are often accompanied by a doc outlining the specific process and possible impacts of the change, and many times are given "global approval" by someone that has that ability rather than requiring individual approvals from each affected team.
    - 5y ago
xcambar 5y ago
The process described in the article (allow old+new behavior on server, migrate clients, deprecate old behavior on server) works spectacularly well, especially when used with a decent level of observability and informafion in the form of logging messages. We are using variations on this theme extensively at my company, in a large spectrum of projects, with great satisfaction. Crucially, this method is orthogonal to using monorepos. It is simply a safety net for and good stewardship of your APIs. Definitely a best practice in my tool belt.
- kohlerm 5y ago
  That is just a best practice for published interfaces (allowed to be used outside of your repositories). E.G. using a different build, main example being APIs available for customers) Not all interfaces need (really "must not") to be published. For "private" interfaces it is an advantage to be able to refactor them easily and also being able to rollback those changes easily if needed. Without a mono-repo that becomes difficult. You basically need an additional layer to "stitch" commits together. You could question why that makes sense because it would just emulate mono-repo behavior.
  - bluGill 5y ago
    If an interface spans repos it isn't private. I assume you are not creating some strawman every source code file is in a separate repository. That would be insane and nobody does that. If you are in a mutli-repo world, then how your break your repos up is an important decision. There are interfaces that are allowed to use within one repo that you cannot use in others, which allows things that should be coupled to be coupled, while forcing things that should be more isolated to be isolated. This is the power of the multi-repo: the separation of what is private vs public is enforced. (which isn't to say you should go to multi-repo - there are pros and cons of both approaches, in the area of where and interface can be used multi-repo gives you more control, but there are other ways to achieve the same goal)
    - kohlerm 5y ago
      Not sure I understand what you say. But a repo for a library that is used only internally in an application does not need to be published to "customers"(real external customers or internal customers such as another microservice). Therefore interfaces of this library can be changed "at will" as long as all internal users are adapted.
kohlerm 5y ago
I do not really agree with the conclusion. Google does large scale automatic refactorings. Those clearly benefit from atomic commits, because it is easy to roll them back in that case. As other have mentioned in smaller projects you might want to be able to do some (internal) refactorings easily and being able to roll them back easily is a big advantage
- skybrian 5y ago
  I’m not sure what you consider the conclusion to be, but large-scale refactorings at Google do happen in the way described in the article, with the bulk of the migration done using many commits done in parallel. Being able to rollback is indeed important and reducing the sizes of commits makes it less painful. If one project has a problem, unrelated projects won’t see the churn. Or at least that’s how it was done when I was there. It’s been a while.
  - kohlerm 5y ago
    I am saying large-scale refactoring can and should (in same cases) happen (and are doable). The author claims you should always do it in steps. I disagree with this opinion. If the refactoring is automatic, why would I do it in steps?
    - skybrian 5y ago
      The more projects you touch, the more approvals you need and the more tests you need to run. Some tests take a long time to run and some are flaky. And the longer to takes to do all this, the more likely there is to be a merge conflict due to other people’s commits. If you divide it up, most commits will land quickly and without issue, and then you can deal with the rest.
    - bluGill 5y ago
      Is your automatic refactoring completely 100% bug free in all edge conditions? I personally don't have that much confidence.
  - joatmon-snoo 5y ago
draw_down 5y ago
This is still easier to do with monorepo. One reason is easily, reliably finding all the usage sites.
bla3 5y ago
The advantage of a monorepo in this particular case is that it makes easy things easy: if you want to remove a parameter of a function in some library and that function has just a few callers in dependent executables, you can just do that in a single commit. Without a monorepo, you have to do the full-blown iterative rollout described in the OP even for small changes, if they cross VCS boundaries.
- draw_down 5y ago
  Exactly, I don’t understand what is so difficult to get here.
- lkschubert8 5y ago
  Wouldn't you still have to do the iterative thing to avoid temporary outages during deployments? Or do you try to synchronize all deployments?
  - sparsely 5y ago
    I don't think synchronized deployments is really possible - you'd have to either still do the iterative thing, or possibly have some versioning system in place
    - bluGill 5y ago
      It is possible for trivial cases. What I do in my basement for example - though even there I have come to prefer keeping things intentionally unsynchronized: it ensures that after updates I have some system that still works.
  - bpicolo 5y ago
    It takes the guesswork out of library migrations. API migrations still need forwards/backwards compat hygiene, unless you blue/green your entire infrastructure to compatible versions, which is possible but not necessarily practical
  - rohitpaulk 5y ago
shawnz 5y ago
Single "big bang" atomic commits that update both the client and server with new features usually aren't practical, I can agree with that. I think the real intention of the "atomic commits" idea is a total ordering of commits between both the client and server code. Both a "big bang" strategy as well as the author's incremental change strategy can benefit from that arrangement. The key is that at any given point in the repository's history, you can be sure that the client and server both work together. The point is not that each commit atomically completes an entire new feature, only that each commit atomically moves the repository to a state where the client and server still work together. In that sense, the author's incremental commits actually do have that kind of atomicity.
nhoughto 5y ago
Don’t agree with the conclusion or reasoning, sure you might need to take a multi Stage approach for many reasons. But if you’ve got your shit together atomic change is possible, in many other approaches it’s not even nearly possible. Whether your org can put together the maturity / investment / whatever to make it a reality is upto you. But the possibility is the sell, not the guarantee. If you can make it work (and google do on a crazy scale) it’s an incredible power, incredible, like change your entire approach to tech debt incredible. Also has costs at scale like the theory about how google keeps shutting down products because keeping them in tree is expensive.
- jyounker 5y ago
  Google keeps shutting down projects because of internal politics and reward structures.
KaiserPro 5y ago
Monorepos offer one thing, and precisely one advantage, "visibility" because everything is in the same place, you can, in theory can find stuff. However in practice its also a great way to hide things. But the major issue is that people confuse monorepos for a release system. Monorepos do not replace the need for an artifact cache/store, or indeed versioned libraries. They also don't dictate that you can't have them, it just makes it easier _not to_. You can do what facebook do, which is essentially have an unknowable npm like dependency graph and just yolo it. They sorta make it work by having lots of unit tests. However that only works on systems that have a continuous update path (ie not user hardware.) It is possible to have "atomic changes" across many libraries. it makes it easier to regex stuff, but also impossible to test or predict what would happen. Its very rare that you'd want to alter >0.1% of the files in your monorepo at one time. But thats not the fault of a monorepo, thats a product of having millions of files.
kerblang 5y ago
Sort of off-topic, but before building a shared library, I strongly recommend reading up on Jeff Atwood's "Rule of Threes". It's harder than most people think, and I've seen them do far more harm than good because authors aren't disciplined & lack domain expertise. The biggest mistake is thinking "Something is always better than nothing" when in fact that something can easily become your biggest liability. If you must do it, keep it as minimalist and focused as possible, and don't hand the task to junior-level people; hand it to someone who recognizes that it's 10x harder than it looks. https://blog.codinghorror.com/the-delusion-of-reuse https://blog.codinghorror.com/the-delusion-of-reuse https://blog.codinghorror.com/rule-of-three/ https://blog.codinghorror.com/rule-of-three/
corpMaverick 5y ago
On the subject of libraries on large code bases. I think one should be careful on deciding what goes into a library and what goes into an API/Service. Both allow you to share code. But there is a difference on who controls the deployment in prod. For example if you need to change a business rule, you can change the code, and when the API/service is deployed the rule change takes effect. However in a library you make the change and deploy the library but it is up to the apps that include the library to decide when the change is deployed in prod. As the owner of the library, you no longer have control on when it is deployed in prod.
mumblemumble 5y ago
So I'm kind of stuck thinking, if you want to have cross-cutting changes happening in many small steps rather than one big one, why not choose a repository layout that makes it the path of least resistance, rather than one that makes it require extra effort?
- bluGill 5y ago
  There is no perfect answer. Both mono-repo and multi-repo have pros and cons. Once you make a choice you have to deal with the cons that come with your choice (you are allowed to change, but then you get a different set of cons to deal with).
joatmon-snoo 5y ago
ITT: people who have never dealt with things going _wrong_ in a large codebase.
sam0x17 5y ago
The article seems to be written from a perspective where integration and unit testing don't exist. IMO the main advantage of monorepos is that you can have end-to-end tests within the same project structure. In that case you wouldn't need to do these incremental changes hoping things don't break in production, because your test suite would prove to you that it works.
- ec109685 5y ago
  And even if you do the changes incrementally, it lets you validate you are doing each step correctly and not breaking backwards compatibility along the way.
icythere 5y ago
As an ops engineer I really expect not to use multiple repositories. Seriously the company don't hire 10 ops engineers + 1 developer: They hire 10 developers + 1 ops engineer. Instead of fixing a problem once, the ops engineer now have to solve a problem 10 times! It's against any good promise at hiring time! In my past job, I wrote a script to update all deployment manifests in tens of repositories. All commits arrived at the git server at the same time. Needless to say the whole team had to stop all work to wait for the ci/cd triggers @@@
caffeine 5y ago
He talked about per-directory ACLs … does anyone know an open source system that implements this? I can’t seem to find one

102 points by askl 5y ago

Interesting idea and it certainly makes sense to ensure working software at all times. However it doesn’t have to be one or the other. Sometimes a single commit (or PR if you prefer multiple commits) can update the api and all clients at the same time. Sometimes client callers are external to the repo/company. In which case backward compatibility strategy is needed. There is no need to abandon a concept of a single repo just because you might not use one of the main benefits all the time.

w_t_payne 5y ago

Either way, you've got to do the tooling work to make your chosen approach work. There's no free lunch here.

marceloabsousa 5y ago

Mostly agree with the article. To me the problem is about dependency management. I see all the time codebases hugely fragmented at the level of git which is totally ad-hoc. After a while, teams face a lot of issues the most annoying being one change in the product involving N PRs with N code reviews and N CIs. This fragmentation of knowledge also pop-ups in weird integration bugs that could be solved with better tool enforcing clearer processes. The ultimate goal of Reviewpad (https://reviewpad.com https://reviewpad.com) is to allow the benefits of the monorepo approach independently of how the codebase is fragmented at the git level. We are starting from the perspective of code review and debugging (e.g. code reviews with multiple PRs, or code reviews across multiple projects). For people doing microservices like us, the ability to have a single code review for both library and clients has been quite positive so far.

lhnz 5y ago

> It's particularly easy to see that the "atomic changes > across the whole repo" story is rubbish when you move > away from libraries, and also consider code that has > any kind of more complicated deployment lifecycle, > for example the interactions between services and > client binaries that communicate over an RPC interface. This seems exactly wrong to me. Getting rid of complicated deployment lifecycles is exactly the job that people use monorepos to solve. Wasn't that one of the reasons they are used at Google, Facebook, etc? As well as being able to do large-scale refactors in one place, of course. You should be able to merge a PR to cause a whole system to deploy: clients, backend services, database schemas, libraries, etc. This doesn't preclude wanting to break up commits into meaningful chunks or to add non-breaking-change migration patterns into libraries -- but consider this: is it meaningful for a change to be broken into separate commits just because it is being done to independent services? What benefit does cutting commits this way give you? What you want to avoid is needing to do separate PRs into many backend service and client repos, since: (1) when the review is split into 10+ places it's easier for reviewers to miss problems arising due to integration, (2) needing to avoid breaking changes can sometimes require developers to follow multi-stage upgrade processes that are so difficult that they cause mistakes, and (3) when there are separate PRs into different repositories these tend to start independent CI processes that will not test the system as-it-will-be (unless you test in production or have a very good E2E suite -- which would be a good idea in this situation). I will say that, even in a monorepo, a big change might still happen gradually behind feature flags. But I think that generally it's nice to be able to deploy breaking changes in a more atomic fashion.

nicative 5y ago
Even within the same repo, it is very likely that the old version of your code will coexist with the new version during the deploy roll out. Often having different commits and deploys is a requirement. For instance, imagine that you add a column in the database and also is using it in the backend service. You probably have add the column first and then commit and deploy the usage of the column later, because you can't easily guarantee that the new code won't be used before the column is added. Same would apply for a new field in the API contract.
- thundergolfer 5y ago
  The old version won't coexist when the change is contained within a single binary, which seems like it would be true in a bunch of cases. In our monorepo we have to treat database changes with care, like you mention, as well as HTTP client/server API changes, but a bunch of stuff can be refactored cleanly without concern for backwards compatibility.
  - withinboredom 5y ago
    Do you only have a single instance of the binary running across the whole org? And during deployment to you stop the running instance before starting the new one?
    - jvolkman 5y ago
      Any change that won't cross deployable binary boundaries (think docker container) can be made atomically without care about subsequent deployment schedules. So this doesn't work for DB changes or client/server API changes as mentioned by OP, but does work for changes to shared libraries that get packaged into the deployment artifacts. For example, changing the interface in an internal shared library, or updating the version of a shared external library.
      - kohlerm 5y ago
        Seems like a common misconception to me that people seem to believe that you can never change an interface. You actually can as long as it is not published to be used outside of your repositories.

argonaut 5y ago

There are plenty of smaller changes that are just way easier with a single commit. A small API call with just two or three consumers, a documentation change, etc. The multistage approach is certainly good for big changes.

stkdump 5y ago
Also, if you do all changes at once, you can also run the whole testsuite and find problems quicker. Sure, you could still later split the work up into many commits. But even when problems come up that you didn't see in the tests, it is easier to revert the work if it indeed is a single commit. The real reason why I still sometimes prefer many small commits is to reduce the chance of merge conflicts.
- kohlerm 5y ago
  You can do automatic bisecting https://git-scm.com/docs/git-bisect https://git-scm.com/docs/git-bisect which you cannot easily do with multiple repos

cjfd 5y ago

The arguments seem exceedingly weak here. Having a monorepo does not prevent a workflow like (1) add new function signature; (2) let all calls use the new function signature; (3) remove old function signature. However, in some cases a multirepo makes sure you could only do it this way. If a function is called all over the place the three-step program is the only somewhat safe way to do it but if a function is called not exceedingly many times one can do the atomic refactor, which is an easier process. That one sometimes needs the three-step process and sometimes can do an atomic refactor is in no way contradictory as this article seems to claim. It depends on the size of the change and the risk involve what you would do in a particular case. Also, a monorepo forcing trunk-based development is great. Long-living feature branches are hell. I would even say to avoid feature branches entirely and use feature switches instead whenever one can get away with it. Every branch always runs the risk of making refactoring more difficult.

Cthulhu_ 5y ago
Re: branches, I agree in principle - we've had feature branches at my current job that were open for a year, reasoning being that it impacts a lot of our application core and we cannot permit issues impacting existing customers, etc etc. But I do like short-lived (max 1 sprint) branches for my own smaller scale projects because they group changes together. I name my branches after an issue number, rebase freely on that branch, and merge with a `--no-ff` flag when I'm done. My history is a neat branch / work / merge construct. Not sure if this is just fear, but I believe trunk- and feature switch based history will end up an incoherent mess of multiple projects being worked on at the same time, commits passing through each other. I'm sure they can be filtered out, but still.
kohlerm 5y ago
It is not the monorepo which forces trunkbased development it is the fact that you use one build for the monorepo. E.g. you can do trunk based development with multiple repos by forcing everyone to use a common build. You could also use a monorepo and still everyone does their own build consuming libraries via mechanisms such as Mavens Repositories or published npm packages

bob1029 5y ago

It looks like more effective change control is a major concern of the author. We use a monorepo for our organization and have found that feature flags are the best way to manage the problem of exposing new functionality in ways that won't piss off our customers. We can let them tell us when they are ready and we flip the switch. Once a flag goes true for every customer, make a note to drop it. This is important because these things will accumulate if you embrace this ideology.

Cthulhu_ 5y ago

The API changes are mainly a problem (I think) if you have a monorepo, but not a mono-deploy; you can only get the full benefits of a monorepo if all your changes are deployed simultaneously and atomically. Changing a model that is shared between different services (or a client / server) should be atomic. In practice, you need to think about backwards compatibility, and work in the three-step process outlined in the article (build and deprecate, switch over, remove deprecated code across a number of deployments / time). If you don't have atomic deployments, that's one less argument in favor of monorepos.

deleted 5y ago
[deleted]
jyounker 5y ago
You're using one specific definition of an API. Another definition of an API is the contract between a library and the calling code. The latter sort of API changes happen frequently (at least in places with monorepos), and they don't require phased deployments.

thundergolfer 5y ago

> The example of renaming a function with thousands of callers, for example, is probably better handled by just temporarily aliasing the function, or by temporarily defining the new function in terms of the old. Why exactly? The best I can think of is that you may annoy people because they'd need to rebase/merge after your change lands and takes away a function they were using. If you're literally just doing a rename, and you're using something like Java, why not just go ahead and do a global rename?

jsnell 5y ago
Because actually getting that commit pushed won't be just clicking "rename method". You'll need to run and pass the presubmits of every single project you made code changes in - and the more projects you're changing at once, the less likely it is that the build is green for all of them at the same time. Then you'll need to get code review approvals from the code owners of each of the clients in one go. Hopefully no new users pop up during this process. If some do, you'll need to iterate again. Then once some trivial refactoring inevitably causes some kind of breakage ("oh, somebody was using reflection on this class"), you'll need to revert the change. That'll be another set of full code reviews from every single owner. Let's hope that in the meanwhile, nobody pushed any commits that depend on your new code. None of this is a problem if you have a library and two clients. But the story being told is not "we can safely make changes in three projects at once", it's "we can safely make changes in hundreds or thousands of projects". The former is kind of uninteresting. The latter is a fairy tale.
- jvolkman 5y ago
  The latter does happen at Google at least. But it requires tooling to reliably make the change (monorepo-aware refactoring tools, basically), and tooling to run all affected tests. Such large changes at Google are often accompanied by a doc outlining the specific process and possible impacts of the change, and many times are given "global approval" by someone that has that ability rather than requiring individual approvals from each affected team.
  - 5y ago

xcambar 5y ago

The process described in the article (allow old+new behavior on server, migrate clients, deprecate old behavior on server) works spectacularly well, especially when used with a decent level of observability and informafion in the form of logging messages. We are using variations on this theme extensively at my company, in a large spectrum of projects, with great satisfaction. Crucially, this method is orthogonal to using monorepos. It is simply a safety net for and good stewardship of your APIs. Definitely a best practice in my tool belt.

kohlerm 5y ago
That is just a best practice for published interfaces (allowed to be used outside of your repositories). E.G. using a different build, main example being APIs available for customers) Not all interfaces need (really "must not") to be published. For "private" interfaces it is an advantage to be able to refactor them easily and also being able to rollback those changes easily if needed. Without a mono-repo that becomes difficult. You basically need an additional layer to "stitch" commits together. You could question why that makes sense because it would just emulate mono-repo behavior.
- bluGill 5y ago
  If an interface spans repos it isn't private. I assume you are not creating some strawman every source code file is in a separate repository. That would be insane and nobody does that. If you are in a mutli-repo world, then how your break your repos up is an important decision. There are interfaces that are allowed to use within one repo that you cannot use in others, which allows things that should be coupled to be coupled, while forcing things that should be more isolated to be isolated. This is the power of the multi-repo: the separation of what is private vs public is enforced. (which isn't to say you should go to multi-repo - there are pros and cons of both approaches, in the area of where and interface can be used multi-repo gives you more control, but there are other ways to achieve the same goal)
  - kohlerm 5y ago
    Not sure I understand what you say. But a repo for a library that is used only internally in an application does not need to be published to "customers"(real external customers or internal customers such as another microservice). Therefore interfaces of this library can be changed "at will" as long as all internal users are adapted.

kohlerm 5y ago

I do not really agree with the conclusion. Google does large scale automatic refactorings. Those clearly benefit from atomic commits, because it is easy to roll them back in that case. As other have mentioned in smaller projects you might want to be able to do some (internal) refactorings easily and being able to roll them back easily is a big advantage

skybrian 5y ago
I’m not sure what you consider the conclusion to be, but large-scale refactorings at Google do happen in the way described in the article, with the bulk of the migration done using many commits done in parallel. Being able to rollback is indeed important and reducing the sizes of commits makes it less painful. If one project has a problem, unrelated projects won’t see the churn. Or at least that’s how it was done when I was there. It’s been a while.
- kohlerm 5y ago
  I am saying large-scale refactoring can and should (in same cases) happen (and are doable). The author claims you should always do it in steps. I disagree with this opinion. If the refactoring is automatic, why would I do it in steps?
  - skybrian 5y ago
    The more projects you touch, the more approvals you need and the more tests you need to run. Some tests take a long time to run and some are flaky. And the longer to takes to do all this, the more likely there is to be a merge conflict due to other people’s commits. If you divide it up, most commits will land quickly and without issue, and then you can deal with the rest.
  - bluGill 5y ago
    Is your automatic refactoring completely 100% bug free in all edge conditions? I personally don't have that much confidence.
- joatmon-snoo 5y ago

draw_down 5y ago

This is still easier to do with monorepo. One reason is easily, reliably finding all the usage sites.

bla3 5y ago

The advantage of a monorepo in this particular case is that it makes easy things easy: if you want to remove a parameter of a function in some library and that function has just a few callers in dependent executables, you can just do that in a single commit. Without a monorepo, you have to do the full-blown iterative rollout described in the OP even for small changes, if they cross VCS boundaries.

draw_down 5y ago
Exactly, I don’t understand what is so difficult to get here.
lkschubert8 5y ago
Wouldn't you still have to do the iterative thing to avoid temporary outages during deployments? Or do you try to synchronize all deployments?
- sparsely 5y ago
  I don't think synchronized deployments is really possible - you'd have to either still do the iterative thing, or possibly have some versioning system in place
  - bluGill 5y ago
    It is possible for trivial cases. What I do in my basement for example - though even there I have come to prefer keeping things intentionally unsynchronized: it ensures that after updates I have some system that still works.
- bpicolo 5y ago
  It takes the guesswork out of library migrations. API migrations still need forwards/backwards compat hygiene, unless you blue/green your entire infrastructure to compatible versions, which is possible but not necessarily practical
- rohitpaulk 5y ago

shawnz 5y ago

Single "big bang" atomic commits that update both the client and server with new features usually aren't practical, I can agree with that. I think the real intention of the "atomic commits" idea is a total ordering of commits between both the client and server code. Both a "big bang" strategy as well as the author's incremental change strategy can benefit from that arrangement. The key is that at any given point in the repository's history, you can be sure that the client and server both work together. The point is not that each commit atomically completes an entire new feature, only that each commit atomically moves the repository to a state where the client and server still work together. In that sense, the author's incremental commits actually do have that kind of atomicity.

nhoughto 5y ago

Don’t agree with the conclusion or reasoning, sure you might need to take a multi Stage approach for many reasons. But if you’ve got your shit together atomic change is possible, in many other approaches it’s not even nearly possible. Whether your org can put together the maturity / investment / whatever to make it a reality is upto you. But the possibility is the sell, not the guarantee. If you can make it work (and google do on a crazy scale) it’s an incredible power, incredible, like change your entire approach to tech debt incredible. Also has costs at scale like the theory about how google keeps shutting down products because keeping them in tree is expensive.

jyounker 5y ago
Google keeps shutting down projects because of internal politics and reward structures.

KaiserPro 5y ago

Monorepos offer one thing, and precisely one advantage, "visibility" because everything is in the same place, you can, in theory can find stuff. However in practice its also a great way to hide things. But the major issue is that people confuse monorepos for a release system. Monorepos do not replace the need for an artifact cache/store, or indeed versioned libraries. They also don't dictate that you can't have them, it just makes it easier _not to_. You can do what facebook do, which is essentially have an unknowable npm like dependency graph and just yolo it. They sorta make it work by having lots of unit tests. However that only works on systems that have a continuous update path (ie not user hardware.) It is possible to have "atomic changes" across many libraries. it makes it easier to regex stuff, but also impossible to test or predict what would happen. Its very rare that you'd want to alter >0.1% of the files in your monorepo at one time. But thats not the fault of a monorepo, thats a product of having millions of files.

kerblang 5y ago

Sort of off-topic, but before building a shared library, I strongly recommend reading up on Jeff Atwood's "Rule of Threes". It's harder than most people think, and I've seen them do far more harm than good because authors aren't disciplined & lack domain expertise. The biggest mistake is thinking "Something is always better than nothing" when in fact that something can easily become your biggest liability. If you must do it, keep it as minimalist and focused as possible, and don't hand the task to junior-level people; hand it to someone who recognizes that it's 10x harder than it looks. https://blog.codinghorror.com/the-delusion-of-reuse https://blog.codinghorror.com/the-delusion-of-reuse https://blog.codinghorror.com/rule-of-three/ https://blog.codinghorror.com/rule-of-three/

corpMaverick 5y ago

On the subject of libraries on large code bases. I think one should be careful on deciding what goes into a library and what goes into an API/Service. Both allow you to share code. But there is a difference on who controls the deployment in prod. For example if you need to change a business rule, you can change the code, and when the API/service is deployed the rule change takes effect. However in a library you make the change and deploy the library but it is up to the apps that include the library to decide when the change is deployed in prod. As the owner of the library, you no longer have control on when it is deployed in prod.

mumblemumble 5y ago

So I'm kind of stuck thinking, if you want to have cross-cutting changes happening in many small steps rather than one big one, why not choose a repository layout that makes it the path of least resistance, rather than one that makes it require extra effort?

bluGill 5y ago
There is no perfect answer. Both mono-repo and multi-repo have pros and cons. Once you make a choice you have to deal with the cons that come with your choice (you are allowed to change, but then you get a different set of cons to deal with).

joatmon-snoo 5y ago

ITT: people who have never dealt with things going _wrong_ in a large codebase.

sam0x17 5y ago

The article seems to be written from a perspective where integration and unit testing don't exist. IMO the main advantage of monorepos is that you can have end-to-end tests within the same project structure. In that case you wouldn't need to do these incremental changes hoping things don't break in production, because your test suite would prove to you that it works.

ec109685 5y ago
And even if you do the changes incrementally, it lets you validate you are doing each step correctly and not breaking backwards compatibility along the way.

icythere 5y ago

As an ops engineer I really expect not to use multiple repositories. Seriously the company don't hire 10 ops engineers + 1 developer: They hire 10 developers + 1 ops engineer. Instead of fixing a problem once, the ops engineer now have to solve a problem 10 times! It's against any good promise at hiring time! In my past job, I wrote a script to update all deployment manifests in tens of repositories. All commits arrived at the git server at the same time. Needless to say the whole team had to stop all work to wait for the ci/cd triggers @@@

caffeine 5y ago

He talked about per-directory ACLs … does anyone know an open source system that implements this? I can’t seem to find one