4 ms·

I'm not familiar with how monorepos work in practice, but it seems obvious to me that it's going to complicate everyday tasks. Ready to commit? Whoops, another

by wtracy 8y ago

I'm not familiar with how monorepos work in practice, but it seems obvious to me that it's going to complicate everyday tasks. Ready to commit? Whoops, another team made a bunch of commits to their project, and you need to rebase your project before you can commit. (I'm having flashbacks to Clearcase already.) Need to roll back the last two commits you made? Sure, that takes two seconds--oh, wait, another team made multiple commits that got interleaved with yours. Have fun cherry picking the files you want to revert. Of course, I'm apparently a curmodgeon, because as soon as someone starts talking about running a find/replace globally across multiple projects, I want to grab something sharp.

headcanon 8y ago
It seems facebook built tooling to address that issue. I doubt they would tolerate that lifestyle either: https://code.fb.com/web/rapid-release-at-massive-scale/ https://code.fb.com/web/rapid-release-at-massive-scale/
joshuamorton 8y ago
Why would you need to rebase or cherrypick unless you and the other team were touching the same files?
- notacoward 8y ago
  Problem is, places that use monorepos also tend to have whole teams full of people who feel entitled to f#ck with stuff across the entire repo, often without going through the normal review or other processes for each component. Thus it's not uncommon to commit code early in the day so that you can build packages for system test, find a problem in system test, then come back later the same day to find one of those randos has already "fixed" your code to adhere to some new standard they decided on at lunchtime. Now reverting only your own commit is a mess, and reverting theirs as well invites a s###storm of epic proportions because they're higher-caste than you.
  - eadmund 8y ago
    > places that use monorepos also tend to have whole teams full of people who feel entitled to f#ck with stuff across the entire repo That's only true to the extent that the statement 'places that use multirepos also tend to be full of people who feel entitled to f#ck with stuff across the entire codebase' is. Bad colleagues can cause damage either way. I appreciate that with a single repo, bad colleagues are at least forced to have passing tests after their changes, rather than leaving it to me to pick up the pieces.
    - notacoward 8y ago
      That hasn't been my experience. Yes, it's a culture thing rather than a technology thing, but with a monorepo the "core" or "foundation" or "developer experience" teams tend to act like they're the owners of all the code and everyone else is just visiting. With multiple repos that's reversed. Each repo has its owner, and the broad-mandate teams are at least aware of their visitor status. That cultural difference has practical consequences, which IMO favor separate repos. The busybodies and style pedants can go jump in a lava lake. > with a single repo, bad colleagues are at least forced to have passing tests Passing unit tests, big whoop. Maybe sometimes light functional tests. Integration/system/stress tests that have to run across an entire cluster for non-trivial time to get meaningful results and thus can't easily be kicked off from a commit hook? Not a chance. The coverage that results is no better (or even different) than what you'd get with separate repos.
      - joshuamorton 8y ago
        > with a monorepo the "core" or "foundation" or "developer experience" teams tend to act like they're the owners of all the code and everyone else is just visiting. Fwiw, this hasn't at all been my experience with this kind of thing at Google. Certainly developer experience teams and language teams will make broad changes that affect everyone but Those changes are trivial: the change maker has to make a convincing argument that the change can't break anyone, if they can't, it will require local approval from the owner. They don't happen all that often: ~once a month for any given leaf directory. They require a special approvals process where you have to answer why the change is necessary, why the churn is worth it, and why it really is safe, and convince a group of approvers that this is the case. If they do break something, the change-maker has to roll it back and then either reconsider, or fix the issues and try again. >The busybodies and style pedants can go jump in a lava lake. Consistent style is important. Among other things, it means that unmaintained code (ie. the stuff written 3 years ago that does its job) still gets updates to be consistent with everything else, so that you don't end up with ancient spaghetti that breaks every modern style rule (I mean you still get that, but less). It also allows deprecation. If there's a core library that does something "wrong", they can deprecate the badly behaved stuff and make sure everyone is off of it: if its not in trunk, it isn't being used. >The coverage that results is no better (or even different) than what you'd get with separate repos. This is untrue. If I'm the `core` team who maintains the `core` repo that everyone depends on, and I make a change that breaks you but my tests pass, you don't know it. Then when you version bump, fixing yourself is your problem. With a monorepo, it's my responsibility to fix you before I can make my change.
username90 8y ago
Why would those things be issues? At Google there multiple new commits per second and it works mostly fine, you just need a VCS which is made for a monorepo.
lclarkmichalek 8y ago
> Ready to commit? Whoops, another team made a bunch of commits to their project, and you need to rebase your project before you can commit. (I'm having flashbacks to Clearcase already.) If they merge cleanly, it's not an issue. If they don't, you need to fix the merge conflict. The work you need to do is proportional to the number of merge conflicts, which isn't special to monorepos. > Need to roll back the last two commits you made? Sure, that takes two seconds--oh, wait, another team made multiple commits that got interleaved with yours. Have fun cherry picking the files you want to revert. Again, only an issue if the changes are on the same files. It can be a bit of a pain to revert a stack of diffs, but if it's just a random commit with no other relevent commits to the file, very easy.
- wtracy 8y ago
  Yeah, the rebasing complaint isn't fair if you're using a modern VCS. I used to work on a large team at Cisco Systems that used Clearcase. Clearcase does not do merges. If anything has changed in master, you have to check out again, which obliterates all local changes. (I have never met a developer who liked Clearcase. It was built to simplify life for system administrators and to tick the right boxes for management, not to be useful for developers.) My general VCS experience is that you can't roll back a commit without also rolling back all subsequent commits, related or not. I'm glad to hear that modern systems have fixed that. (It looks like even Subversion does that now, cool!)
ajuc 8y ago
If you don't touch the same files the merge is trivial. If you do - then it's a good thing that you see there's a conflict right away, rather than notice versioning problems between your separate subsystems in integration testing (or even worse - production).
svat 8y ago
See Hyrum Wright's talks on Large-Scale Changes at Google: https://www.youtube.com/watch?v=ZpvvmvITOrk https://www.youtube.com/watch?v=ZpvvmvITOrk (CppCon2014), https://www.youtube.com/watch?v=TrC6ROeV4GI https://www.youtube.com/watch?v=TrC6ROeV4GI (CppCon 2018), and paper by Googlers on ClangMR (http://www.hyrumwright.org/papers/icsm2013.pdf http://www.hyrumwright.org/papers/icsm2013.pdf). It's great! And if you'd like to learn more about how monorepos work in practice there's a couple of papers: https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext?mobile=false https://cacm.acm.org/magazines/2016/7/204032-why-google-stor... and https://ai.google/research/pubs/pub47040 https://ai.google/research/pubs/pub47040 (Also worth reading: http://danluu.com/monorepo/ http://danluu.com/monorepo/)
eadmund 8y ago
> Whoops, another team made a bunch of commits to their project, and you need to rebase your project before you can commit. If their commits didn't affect you, then … it merges cleanly, and you don't care. If their commits do affect you, then … now you know. With multiple repositories, you will have no idea that they broke something you rely on. Even better, with a monorepo, if they break something your existing code relies on, they' have broken the build, and they have to fix it. With multiple repos, another person or team is free to break things you rely on without even knowing it, and you won't know it until you update your dependencies hours, days, weeks or months later, wondering why everything is suddenly different. If another team's work is tightly-enough interwoven with yours that their daily commits affect you, then you're all on the same team (and/or your architecture needs work).
yowlingcat 8y ago
You're not a curmudgeon, you're just wrong. As another comment states, you'd have to fix the merge conflict anyways and if there's no merge conflict, your rebase isn't an issue. If you're getting merge conflicts and need to rebase your project and that's challenging, it turns out to be an...organizational issue.
bostik 8y ago
A lot of the pain you describe boils down to insufficient tooling and bad code organisation. Outside of mega-corps, usually only a few people (couple of teams at best) are working on a given section of code at a time. Coordinating changes between maybe 12-15 people is quite feasible. Most of the time it's enough to keep code nicely segregated by paths - something like $team/$XXX or $scope/$team/$YYY should work. On top of that, you need two things to enable a nice workflow: * server-side (or otherwise programmatic) merges/rebases only; no human should ever need to push to master directly. That's the job of the [pre-]CI machinery. * comprehensive pre-merge testing before the server-side merge. Do all your development in branches, and have bot+CI test _all_ unmerged branches against MASTER+YOURBRANCH on every push to YOURBRANCH. Because no human is involved when merging to master, the tip of the master has not moved due to external factors. Also, rerun tests if master has indeed moved thanks to bot having merged another branch ahead of yours. Fix any test breakages in the branch. To make the second item work, you have to realise that the testing steps need to be rapid enough. Usually it should be enough to test the new-to-be-branch against merge problems, code convention errors and have a run through all the usual unit tests. Most of the time you can leave any larger cross-service or integration/end-to-end tests for code that has landed in master already.[ß] Once code has landed in master, CI can pick it up and produce the release artifacts. The tooling needs to be good enough to know how to avoid useless work (doing useless checkouts and running tests against dozens of branches will get expensive). It also needs to provide very good and easily actionable feedback. You want clear test results, with quick jumps to failure(s) and robust logging. To give some context of where the above is coming from... We have a monorepo with more than 130 projects, and about 7% of the codebase changes in any given month. (Except for December. Understandably.) We also clock more than 40k pre-merge test runs a month. Once a merge request has been approved, it is often available for shipping in 20 minutes. When builds against master never fail due to code conflicts, development velocity is maintained better even with multiple teams working on the same piece of code. ß: this is a tradeoff between development friction and test coverage. A sufficiently thorough integration test can take anything from a just a few minutes to couple of hours. You want to run them against batches of changes, and in case of introduced failures, incentivise teams to put in new unit tests to cover as much of the uncovered error scope as possible.

by wtracy 8y ago

headcanon 8y ago

It seems facebook built tooling to address that issue. I doubt they would tolerate that lifestyle either: https://code.fb.com/web/rapid-release-at-massive-scale/ https://code.fb.com/web/rapid-release-at-massive-scale/

joshuamorton 8y ago

Why would you need to rebase or cherrypick unless you and the other team were touching the same files?

notacoward 8y ago
Problem is, places that use monorepos also tend to have whole teams full of people who feel entitled to f#ck with stuff across the entire repo, often without going through the normal review or other processes for each component. Thus it's not uncommon to commit code early in the day so that you can build packages for system test, find a problem in system test, then come back later the same day to find one of those randos has already "fixed" your code to adhere to some new standard they decided on at lunchtime. Now reverting only your own commit is a mess, and reverting theirs as well invites a s###storm of epic proportions because they're higher-caste than you.
- eadmund 8y ago
  > places that use monorepos also tend to have whole teams full of people who feel entitled to f#ck with stuff across the entire repo That's only true to the extent that the statement 'places that use multirepos also tend to be full of people who feel entitled to f#ck with stuff across the entire codebase' is. Bad colleagues can cause damage either way. I appreciate that with a single repo, bad colleagues are at least forced to have passing tests after their changes, rather than leaving it to me to pick up the pieces.
  - notacoward 8y ago
    That hasn't been my experience. Yes, it's a culture thing rather than a technology thing, but with a monorepo the "core" or "foundation" or "developer experience" teams tend to act like they're the owners of all the code and everyone else is just visiting. With multiple repos that's reversed. Each repo has its owner, and the broad-mandate teams are at least aware of their visitor status. That cultural difference has practical consequences, which IMO favor separate repos. The busybodies and style pedants can go jump in a lava lake. > with a single repo, bad colleagues are at least forced to have passing tests Passing unit tests, big whoop. Maybe sometimes light functional tests. Integration/system/stress tests that have to run across an entire cluster for non-trivial time to get meaningful results and thus can't easily be kicked off from a commit hook? Not a chance. The coverage that results is no better (or even different) than what you'd get with separate repos.
    - joshuamorton 8y ago
      > with a monorepo the "core" or "foundation" or "developer experience" teams tend to act like they're the owners of all the code and everyone else is just visiting. Fwiw, this hasn't at all been my experience with this kind of thing at Google. Certainly developer experience teams and language teams will make broad changes that affect everyone but Those changes are trivial: the change maker has to make a convincing argument that the change can't break anyone, if they can't, it will require local approval from the owner. They don't happen all that often: ~once a month for any given leaf directory. They require a special approvals process where you have to answer why the change is necessary, why the churn is worth it, and why it really is safe, and convince a group of approvers that this is the case. If they do break something, the change-maker has to roll it back and then either reconsider, or fix the issues and try again. >The busybodies and style pedants can go jump in a lava lake. Consistent style is important. Among other things, it means that unmaintained code (ie. the stuff written 3 years ago that does its job) still gets updates to be consistent with everything else, so that you don't end up with ancient spaghetti that breaks every modern style rule (I mean you still get that, but less). It also allows deprecation. If there's a core library that does something "wrong", they can deprecate the badly behaved stuff and make sure everyone is off of it: if its not in trunk, it isn't being used. >The coverage that results is no better (or even different) than what you'd get with separate repos. This is untrue. If I'm the `core` team who maintains the `core` repo that everyone depends on, and I make a change that breaks you but my tests pass, you don't know it. Then when you version bump, fixing yourself is your problem. With a monorepo, it's my responsibility to fix you before I can make my change.

username90 8y ago

Why would those things be issues? At Google there multiple new commits per second and it works mostly fine, you just need a VCS which is made for a monorepo.

lclarkmichalek 8y ago

> Ready to commit? Whoops, another team made a bunch of commits to their project, and you need to rebase your project before you can commit. (I'm having flashbacks to Clearcase already.) If they merge cleanly, it's not an issue. If they don't, you need to fix the merge conflict. The work you need to do is proportional to the number of merge conflicts, which isn't special to monorepos. > Need to roll back the last two commits you made? Sure, that takes two seconds--oh, wait, another team made multiple commits that got interleaved with yours. Have fun cherry picking the files you want to revert. Again, only an issue if the changes are on the same files. It can be a bit of a pain to revert a stack of diffs, but if it's just a random commit with no other relevent commits to the file, very easy.

wtracy 8y ago
Yeah, the rebasing complaint isn't fair if you're using a modern VCS. I used to work on a large team at Cisco Systems that used Clearcase. Clearcase does not do merges. If anything has changed in master, you have to check out again, which obliterates all local changes. (I have never met a developer who liked Clearcase. It was built to simplify life for system administrators and to tick the right boxes for management, not to be useful for developers.) My general VCS experience is that you can't roll back a commit without also rolling back all subsequent commits, related or not. I'm glad to hear that modern systems have fixed that. (It looks like even Subversion does that now, cool!)

ajuc 8y ago

If you don't touch the same files the merge is trivial. If you do - then it's a good thing that you see there's a conflict right away, rather than notice versioning problems between your separate subsystems in integration testing (or even worse - production).

svat 8y ago

See Hyrum Wright's talks on Large-Scale Changes at Google: https://www.youtube.com/watch?v=ZpvvmvITOrk https://www.youtube.com/watch?v=ZpvvmvITOrk (CppCon2014), https://www.youtube.com/watch?v=TrC6ROeV4GI https://www.youtube.com/watch?v=TrC6ROeV4GI (CppCon 2018), and paper by Googlers on ClangMR (http://www.hyrumwright.org/papers/icsm2013.pdf http://www.hyrumwright.org/papers/icsm2013.pdf). It's great! And if you'd like to learn more about how monorepos work in practice there's a couple of papers: https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext?mobile=false https://cacm.acm.org/magazines/2016/7/204032-why-google-stor... and https://ai.google/research/pubs/pub47040 https://ai.google/research/pubs/pub47040 (Also worth reading: http://danluu.com/monorepo/ http://danluu.com/monorepo/)

eadmund 8y ago

> Whoops, another team made a bunch of commits to their project, and you need to rebase your project before you can commit. If their commits didn't affect you, then … it merges cleanly, and you don't care. If their commits do affect you, then … now you know. With multiple repositories, you will have no idea that they broke something you rely on. Even better, with a monorepo, if they break something your existing code relies on, they' have broken the build, and they have to fix it. With multiple repos, another person or team is free to break things you rely on without even knowing it, and you won't know it until you update your dependencies hours, days, weeks or months later, wondering why everything is suddenly different. If another team's work is tightly-enough interwoven with yours that their daily commits affect you, then you're all on the same team (and/or your architecture needs work).

yowlingcat 8y ago

You're not a curmudgeon, you're just wrong. As another comment states, you'd have to fix the merge conflict anyways and if there's no merge conflict, your rebase isn't an issue. If you're getting merge conflicts and need to rebase your project and that's challenging, it turns out to be an...organizational issue.

bostik 8y ago

A lot of the pain you describe boils down to insufficient tooling and bad code organisation. Outside of mega-corps, usually only a few people (couple of teams at best) are working on a given section of code at a time. Coordinating changes between maybe 12-15 people is quite feasible. Most of the time it's enough to keep code nicely segregated by paths - something like $team/$XXX or $scope/$team/$YYY should work. On top of that, you need two things to enable a nice workflow: * server-side (or otherwise programmatic) merges/rebases only; no human should ever need to push to master directly. That's the job of the [pre-]CI machinery. * comprehensive pre-merge testing before the server-side merge. Do all your development in branches, and have bot+CI test _all_ unmerged branches against MASTER+YOURBRANCH on every push to YOURBRANCH. Because no human is involved when merging to master, the tip of the master has not moved due to external factors. Also, rerun tests if master has indeed moved thanks to bot having merged another branch ahead of yours. Fix any test breakages in the branch. To make the second item work, you have to realise that the testing steps need to be rapid enough. Usually it should be enough to test the new-to-be-branch against merge problems, code convention errors and have a run through all the usual unit tests. Most of the time you can leave any larger cross-service or integration/end-to-end tests for code that has landed in master already.[ß] Once code has landed in master, CI can pick it up and produce the release artifacts. The tooling needs to be good enough to know how to avoid useless work (doing useless checkouts and running tests against dozens of branches will get expensive). It also needs to provide very good and easily actionable feedback. You want clear test results, with quick jumps to failure(s) and robust logging. To give some context of where the above is coming from... We have a monorepo with more than 130 projects, and about 7% of the codebase changes in any given month. (Except for December. Understandably.) We also clock more than 40k pre-merge test runs a month. Once a merge request has been approved, it is often available for shipping in 20 minutes. When builds against master never fail due to code conflicts, development velocity is maintained better even with multiple teams working on the same piece of code. ß: this is a tradeoff between development friction and test coverage. A sufficiently thorough integration test can take anything from a just a few minutes to couple of hours. You want to run them against batches of changes, and in case of introduced failures, incentivise teams to put in new unit tests to cover as much of the uncovered error scope as possible.