7 ms·

> Moving to a monorepo didn't change much, and what minor changes it made have been positive. I'm not sure that this statement in the summary jives with this s

• notwhereyouare 2y ago
  ironically was gonna come and comment on that same second block of text. We went from monorepo to multi-repo at work and it's been a huge set back and disappointment with the devs because it's what our contractors recommended. I've asked for a code deploy and everything and it's failed in prod due to a missing check in
  • CharlieDigital 2y ago
    > ...because it's what our contractors recommended It's sad when this happens instead of taking input from the team on how to actually improve productivity/quality. A startup I joined started with a multi-repo because the senior team came from a FAANG where this was common practice to have multiple services and a repo for each service. Problem was that it was a startup with one team of 6 devs and each of the pieces was connected by REST APIs. So now any change to one service required deploying that service and pulling down the OpenAPI spec to regenerate client bindings. It was so clumsy and easy to make simple mistakes. I refactored the whole thing in one weekend into a monorepo , collapsed the handful of services into one service, and we never looked back. That refactoring and a later paper out of Google actually inspired me to write this article as a practical guide to building a "modular monolith": https://chrlschn.dev/blog/2024/01/a-practical-guide-to-modular-monoliths/ https://chrlschn.dev/blog/2024/01/a-practical-guide-to-modul...
    • eddd-ddde 2y ago
      At least google and meta are heavy into monorepos, I'm really curious what company is using a _repo per service_. That's insane.
      • dewey 2y ago
        It's almost never a good idea to get inspired by what Google / Meta / Huge Company is doing as most of the times you don't have their problems and they have custom toolings and teams making everything work on that scale.
        • CharlieDigital 2y ago
          In this case, I'd say it's the opposite: monorepo as an approach works amazingly well for small teams all the ways up to huge orgs (with the right tooling to support it). The difference is that past a certain level of complexity, the org will most certainly need specialized tooling to support massive codebases to make CI/CD (build, test, deploy, etc.) times sane. On the other hand, multi-repos may work for massive orgs, but is always going to add friction for small orgs.
          • dewey 2y ago
            In this case I wasn't even referring to mono repo or not, but more about the idea of taking inspiration from very large companies for your own not-large-company problems.
          • influx 2y ago
            I’ve used one of the Meta monorepos (yeah there’s not just one!) and it’s super painful at that scale.
            • CRConrad 2y ago
              > ...one of the Meta monorepos (yeah there’s not just one! Eeh... Might be quite meta-, but really doesn't feel very mono-.
  • jayd16 2y ago
    If prod went down because of a missing check in, there are other problems.
    • notwhereyouare 2y ago
      did I say prod went down? I just said it failed in prod. it was a logging change and only half the logging went out. To me, that's a failure
• taeric 2y ago
  My only counter argument here, is when those 4 things deploy independently. Sometimes, people will get tricked into thinking a code change is atomic because it is in one commit, when it will lead to a mixed fleet because of deployment realities. In that world, having them separate is easier to work with, as you may have to revert one of the deployments separately from the others.
  • derefr 2y ago
    That's just an argument for not doing "implicit GitOps", treating the tip of your monorepo's main branch as the source-of-truth on the correct deployment state of your entire system. ("Implicit GitOps" sorta-kinda works when you have a 1:1 correspondence between repos and deployable components — though not always! — but it isn't tenable for a monorepo.) What instead, then? Explicit GitOps. Explicit, reified release specifications (think k8s resource manifests, or Erlang .relup files), one per separately-deploy-cadenced component. If you have a monorepo, then these live also as a dir in the monorepo. CD happens only when these files change. With this approach, a single PR can atomically merge code and update one or more release specifications (triggering CD for those components), if and when that is a sensible thing to do. But there can also be separate PRs for updating the code vs. "integrating and deploying changes" to a component, if-and-when that is sensible.
    • scubbo 2y ago
      ...I can't believe I'd never thought about the fact that a "Deployment Repo" can, in fact, just be a directory within the Code Repo. Interesting thought - thanks!
    • taeric 2y ago
      I mean... sure? Yes, if you add extra structure on top of your code that is there to model the deployments, then you get a bit closer to modeling your deployments. Isn't that the exact argument for why you might want multiple repositories, as well?
    • ramchip 2y ago
      > With this approach, a single PR can atomically merge code and update one or more release specifications (triggering CD for those components), if and when that is a sensible thing to do. How do you avoid the chicken-and-egg problem? Like if the k8s manifest contains a container tag, and the tag is created by CI when the PR is merged to main, it would seem you can’t add code and deploy that code in the same PR.
• audunw 2y ago
  There’s nothing preventing you from having a single pull request in for merging branches over multiple repos. There’s nothing preventing you from having a parent repo with a lock file that gives you a single linear set of commits tracking the state of multiple repos. That is, if you’re not tied to using just Github of course. Big monorepos and multiple repo solutions require some tooling to deal with scaling issues. What surprises me is the attitude that monorepos are the right solution to these challenges. For some projects it makes sense yes, but it’s clear to me that we should have a solution that allows repositories to be composed/combined in elegant ways. Multi-repository pull requests should be a first class feature of any serious source code management system. If you start two projects separately and then later find out you need to combine their history and work with them as if they were one repository, you shouldn’t be forced to restructure the repositories.
  • pelletier 2y ago
    > Multi-repository pull requests should be a first class feature of any serious source code management system. Do you have examples of source code management systems that provide this feature and do you have experience with them? repo-centric approach of GitHub often feels limiting.
    • jvolkman 2y ago
      Apparently Gerrit supports this with topics: https://gerrit-review.googlesource.com/Documentation/cross-repository-changes.html https://gerrit-review.googlesource.com/Documentation/cross-r...
    • jamesfinlayson 2y ago
      At a company I used to be at they used GitHub Enterprise and some repos definitely seemed to have linked repos or linked commits (I don't remember exactly but there was some way of linking for repos that depended on each other).
    • zelphirkalt 2y ago
      It is very easy to build this yourself. Just use your dependency manager of choice that has git support and create a config of versions of the repos and lock file that. Whenever you have a change across multiple services, simply change that config in one go, so that every commit of that config describes a deployable state of the whole project.
• danudey 2y ago
  I prefer microservices/microrepos _conceptually_, but we had the same experience as your quoted text - making changes to four repos, and backporting those changes to the previous two release branches, means twelve separate PRs to make a change. Having a centralized configuration library (a shared Makefile that we can pull down into our repo and include into the local Makefile) helps, until you have to make a backwards-incompatible change to that Makefile and then post PRs to every branch of every repo that uses that Makefile. Now we have almost the entirety of our projects back into one repository and everything is simpler; one PR per release branch, three PRs (typically) for any change that needs backporting. Vastly simpler process and much less room for error.
  • esperent 2y ago
    Isn't there some middle ground between microservices and monorepos? For example, a few large clearly defined projects that talk to each other via versioned APIs?
    • svieira 2y ago
      Mono-repo is not the same as mono-service. You can deploy multiple services out of a single repository.
    • 0xFACEFEED 2y ago
      Assuming you're just referring to repos: not really IMO. As soon as you split 1 repo into 2 repos you need to start building tooling to support your 2 repos. If your infrastructure is sufficiently robust with 2 repos then you might as well have 3 or 4 or 10. If it's built to _only_ support 2 repos (or 3 or 4) then it's brittle out of the gate. The value of a monorepo is that you completely eliminate certain classes of problems and take on other classes of problems. Classic trade off. Folks that prefer monorepos take the position that multirepo problems are much harder than monorepo problems most of the time.
      • Lvl999Noob 2y ago
        The classic 0, 1, or N. You can have 0 repos (no-code or outsourced or just some unversioned blob of code). You can have 1 repo (a mono-repo). Or you can have N number of repos (individual micro services). Limiting yourself to a number that's higher than 1 but still finite is just asking for pain. (Of course implementation limits can apply but there shouldn't be logical limits)
• wongarsu 2y ago
  It's not as much of a pain if your tooling supports git repos as dependencies. For example a typical multi-repo PR for us with rust is 1) PR against library 2) PR against application that points dependency to PR's branch, makes changes 3) PR review 4) PR 1 is approved and merged 5) PR 2 is changed to point to new master branch of commit 6) PR 2 is approved and merged Same idea if you use some kind of versioning and release system. It's still a bit of a pain with all the PRs and coordination involved, but at every step every branch is consistent and buildable, you just check it out and hit build. This is obviously more difficult if you have a more loosely coupled architecture like microservices. But that's self-inflicted pain
  • deleted 2y ago
    [deleted]
• ericyd 2y ago
  I felt the same, the author seemed to downplay the success while every effect listed in the article felt like a huge improvement.
• eikenberry 2y ago
  I thought one of the whole points behind separate (non-mono)repos was to help enforce loose coupling and if you came to a point where a single feature change required PRs on 4 separate repos then that was an indicator that your project needed refactoring as it was becoming to tightly coupled. The example in the article could have been interpreted to mean that they should refactor the functionality for interacting with the ML model into it's own repo so it could encapsulate this aspect of the project. Instead they doubled down on the tighter coupling by putting them in a monorepo (which itself encourages tighter coupling).
  • marcosdumay 2y ago
    The issue is that you can't "enforce" loose coupling. The causality is reversed here. Your software artifacts will have loose coupling if you divided them well enough on their creation. As soon as they are created, you can't do anything else to change it, except for joining or splitting them.
    • eikenberry 2y ago
      It's not about enforcement, it's about encouragement and path of least resistance. In monorepos the path of least resistance is tight coupling (unless discouraged in other ways). In 'microrepros' (?) there is added resistance to tight coupling as was presented in the article. This encourages people down the correct path but cannot enforce it (again, as was presented in the article).
      • MajimasEyepatch 2y ago
        It leads people down a different path, but that's not necessarily the correct path. For example, between the lack of discoverability of separate repos and the hurdle of modifying multiple separate repos, teams might be tempted to duplicate functionality and data instead of figuring out the appropriate relationships between services. Some amount of coupling is normal and reasonable. There are ways to manage it well.
    • ted_dunning 2y ago
• Attummm 2y ago
  The issue you faced stemmed from the previous best practice of "everything in its own repository." This approach caused major issues. Such as versioning challenges and data model inconsistencies you mentioned. The situations it could lead to are comedy sketches, but it's a real pain especially when you’re part of a team struggling with these problems. And it’s almost impossible to convince a team to change direction once they’ve committed to it. Now, though, it seems the pendulum has swung in the opposite direction, from “everything in its own repo” to “everything in one repo.” This, too, will create its own set of problems, which also can be comedic, but frustrating to experience. For instance, what happens when someone accidentally pushes a certificate or API key and you need to force an update upstream? Coordinating that with 50 developers spread across 8 projects, all in a single repo. Instead we could also face the problems we currently face and start out wirn a balanced approach. Start with one repository, or split frontend and backend if needed. For data pipelines that share models with the API, keep them in the same repository, creating a single source of truth for the data model. This method has often led to other developers telling me about the supposed benefits of “everything in its own repo.” Just as I pushed back then, I feel the need to push back now against the monorepo trend. The same can be said for monoliths and microservices, where the middle ground is often overlooked in discussions about best practices. They all reminded me of the concept of “no silver bullet”[0]. Any decision will face its own unique challenges. But silver bullet solution can create artificial challenges that are wasteful, painful, and most of all unnecessary. [0]https://en.m.wikipedia.org/wiki/No_Silver_Bullet https://en.m.wikipedia.org/wiki/No_Silver_Bullet
  • lolinder 2y ago
    > what happens when someone accidentally pushes a certificate or API key and you need to force an update upstream The correct approach here is typically to invalidate the certificate or API key. A force push usually doesn't work. If you're using GitHub, the dangerous commit lives on effectively forever in an awkward "not in a repository" state. Even if you're not on GitHub and your system actually garbage collects, the repo has been cloned onto enough build machines and dev machines that you're better off just treating the key or cert as compromised than trying to track down all the places where it might have been copied.
• chipdart 2y ago
  > IMO, this is a huge quality of life improvement and prevents a lot of mistakes from not having the right revision synced down across different repos. This alone is a HUGE improvement where a dev doesn't accidentally end up with one repo in this branch and forgot to pull this other repo at the same branch and get weird issues due to this basic hassle. This scenario only happens if you're working with a distributed monolith, you care nothing about breaking existing APIs, and you own all producers and consumers. When this happens, your problem is obviously not how you maintain projects independently in separate repos. Your problems is how you're failing to preserve API contracts, and instead you go to the extreme of insisting in breaking all contracts at once. It's also a huge red flag when you talk about "accidentally end up with one repo in this branch". Don't you know what you are pushing to production? As always, monorepo talks are just a thin veil of optimization placed over a huge mess of major operational and technical problems.
  • The_Colonel 2y ago
    > Your problems is how you're failing to preserve API contracts, and instead you go to the extreme of insisting in breaking all contracts at once. Preserving contracts has a significant cost. Why would you try to preserve contracts if you're in fact in control of all producers and consumers?
    • chipdart 2y ago
      > Preserving contracts has a significant cost. Not really. You just need to break out a new version when you put out a breaking change. This can mean anything between duplicating the code that implements your interface to then apply a change and checking a version flag when invoking an operation to determine which version to run. Do you struggle to not break your code? To me it's like the main responsibility of any developer, isn't it? > Why would you try to preserve contracts if you're in fact in control of all producers and consumers? Because you clearly aren't in control, if you feel compelled to dump anything and everything in the same repository to try to prevent things from breaking. Also, as others have pointed out, storing code in the same repository gives you no assurance that all your services will be deployed at the same exact moment in time. Consider for example multi-region deployments, or even deployments to multiple availability zones. Those aren't atomic. Why does anyone believe that storing all code in the same repository is enough to make these operations atomic?
• eduction 2y ago
  Jibes. Not jives. The word is jibes.
• throwawayk7h 2y ago
  Why can't we have one PR, many repos?
• friendzis 2y ago
  > > In the previous, separate repository world, this would've been four separate pull requests in four separate repositories, and with comments linking them together for posterity. Tooling issue. Bitbucket will happily link PRs if instructed so. > prevents a lot of mistakes from not having the right revision synced down across different repos. Tooling issue. Depending on setup, "tooling" could be few lines in your favorite scripting language. --- Monorepo steps around some tooling issues, but sacrifices access control, checkout speed
• HelloNurse 2y ago
  This "huge quality of life improvement" is "minor" compared to the lack of significant differences in more important aspects of version control that become neither easier nor more complex: for example resolving conflicts (files that used to be in another repository remain uninvolved) or creating, merging and killing branches (still related to the same features, bugfixes, modules, schedules etc.)
• mountainriver 2y ago
  It’s a game of tradeoffs. Yes the situation described does sometimes happen. If you are intentional about how you break apart your repos you can avoid this most of the time. They don’t need to be micro repos, they can just be regular sized and driven by the data model. Some benefits include: * Separate repos evolve better than a monorepo. Particularly in early stage companies where pivots are frequent. * They keep tools just doing one thing well and prevent tying everything in the codebase together. * Dependencies are more flexible, this is particularly useful with ML libraries. * A lot of repo tooling is simpler with medium sized single purpose projects * Stronger sense of ownership boundaries