6 ms·

Stop growing gigantic monolithic repositories. This is a very stupid way to do things. Evidence: it requires you to rewrite Mercurial. Essentially nobody except

by pekk 10y ago

Stop growing gigantic monolithic repositories. This is a very stupid way to do things. Evidence: it requires you to rewrite Mercurial. Essentially nobody except Facebook has this problem, and that isn't because Facebook is so far ahead of everyone else in its engineering practices.

argonaut 10y ago
Google, Facebook, and Twitter all use monolithic repos (I'm taking monolithic to mean 50+% is in one repo, not necessarily everything in one repo), so no, not just Facebook. I'm going to assume there are very good reasons for doing so (and there are - you can look up past discussions on HN).
sshumaker 10y ago
Wrong. Google has the exact same problem, only they chose to rewrite Perforce. The scale of these codebases is way outside what most people have experienced and it might behoove people to realize that a lot of conventional wisdom might not apply. E.g. last time I checked, Google's codebase was >2Billion lines of code. Google has 30,000 engineers working on a "monolithic" codebase and remain relatively productive despite (or perhaps because of it). There are whole sets of different problems at this scale.
- emu 10y ago
  See https://www.youtube.com/watch?v=W71BTkUbdqE https://www.youtube.com/watch?v=W71BTkUbdqE for a lot more details on what Google does.
  - kyrra 10y ago
    And Google has been contributing to Mercurial. Like subtree: https://bitbucket.org/Google/narrowhg https://bitbucket.org/Google/narrowhg
- pvelagal 10y ago
  2 billion lines of code @ 80 characters per line (with 2 bytes per character) = 320 GB of code, fits on my mac SSD. Wonder if 30,0000 engineers , sending 30,000 requests per second to server(s) with 320GB of code (can be put in memory) is a huge problem (edit) in terms of scale, especially when these same engineers build systems for handling billions of requests ?
  - steffan 10y ago
    You're not considering that it's not just the current 2 billion lines of code - it's the history of the repo, branches, forks, possibly binaries thrown in.
    - pvelagal 10y ago
      I never built any source code system, so not an expert, but isnt branching snapshotting the current version of file(s) (strings), and newer versions are about calculating deltas. Arent, most of the Operations in a source code mgmt system, are diffs and merges of strings. Basically, my point is, scale problems associated with these systems are not at the same level as other scalability problems handled by google's core competency areas such as search which deal with billions of requests or peta bytes of data. Not that they are easy, but that orders of magnitude smaller.
- amaks 10y ago
  It's also very beneficial to the code health as it allows to make global refactoring, e.g. adopting new language features, improving shared libraries, etc.
joeblubaugh 10y ago
I've worked in places where we depended on external repos (Maven, Gradle, other Ivy-based things) and places where all code has to live in a mono-repo. Thus far I've preferred the mono-repo mostly for dependency management reasons. Whether you have a lot of internal dependencies or external dependencies, you get similar benefits: - All the changes to internal dependencies are in your revision history, across the company. Shared internal library upgrades are picked up quickly and propagate throughout applications with low delay. - It's easier to have shared versions of external dependencies "automatically" instead of establishing policies that need a human to enforce. This makes it easier to roll out new versions and bug fixes. - It's easier to do system-wide improvements in code quality. Replacing common bad code with better implementations is something I've seen across code bases at both Google and Twitter and they have been beneficial. I think nobody should be trying to implement a system like this if you've got fewer than ~600 SW engineers, though. Small groups don't have as much drift or system-wide refactorings that give you benefit.
- joneholland 10y ago
  At Expedia, we were a gigantic perforce repository, but the trend has moved to hundreds of smaller repositories and applications hosted in enterprise github. Package management is simple, we push common packages into sonatype nexus with semantic versioning. Personally I prefer strong versioning and allowing teams to upgrade common libraries at their own pace, but we've had a few times where we needed to quickly upgrade every repository (i.e., a security patch). You have a few options: 1. You can unpublish the old dependency, breaking all builds until they upgrade. (Frowned upon) 2. You can write a script that identifies all repository owners that depend on you and send out a upgrade by X date email. 3. You can script pull request creation and submit hundreds of pulls to upgrade the dependency. So far, things have worked out well, and I never want to go back to the monorepo.
newobj 10y ago
I've worked at both FB (monorepo) and Amazon (massively federated repo where you only check out tiny slices at a time; a huge package dependency system pulls things together), and I can testify that monorepo is great, and federated repo with dependencies-via-packaging is awful. It's slower to work with, harder to coordinate wide scale changes, slower to work with, the cognitive load is much higher, and it's slower to work with. Did I mention it's higher friction and slower to work with?
- olalonde 10y ago
  What do people mean by monorepo exactly (e.g. at Facebook)? For instance, FB's open source projects aren't all hosted in the same "open source monorepo", why not? And what goes into the monorepos exactly? All the closed source code? As a "one-man-team" who uses at least 15 different repos, it's hard for me to imagine how a massive company would manage things within a single repo and no package management. Also, aren't any of those monorepo companies concerned that a single rogue employee or stolen laptop could leak their entire source code? I can only guess I'm misunderstanding what is meant by monorepo and how they're used...
  - encoderer 10y ago
    Friends tell me that very sensitive code (ranking algos, etc) is not in the repo. Personally, I also favor a single repo. You manage it the way you manage separate packages: with organization and some discipline. The magic is that command line tools like grep, sed & awk--and static analysis tools for refactoring--work really well. You can change a method signature and it just works. I've been part of monolith-breaking before (most recently at Trulia) and it definitely adds friction to the development process to work across such a rich graph of package dependencies.
    - honkhonkpants 10y ago
      Sensitive code _is_ in the repo, you just can't read it. This allows you to change library code, run the tests of the sensitive application code, and know that it worked, without being able to read the sensitive code.
honkhonkpants 10y ago
I love it when random commenters point out the obvious problems of the world's wealthiest, most successful companies.
- 1stop 10y ago
  I know, do you reckon Facebook engineers are sitting awkwardly in a meeting with this comment page up: "Why didn't we think of this!!!?"
- logor 10y ago
  Implying you have to have lots of money to be correct in your software management approach. Try again. Maybe it's because these unknowns can't fathom working with such tightly wound systems that after all the code reviews are done your changes are irrelevant and need to be updated again and go through another review. In before straw man, but other than pointing out day to day issues, maybe you should try to understand the argument against trying to fit everything all in one place. Do you really want to be the maintainer of all those third party libraries you imported? Do you not allow third party and adopt "not invented here"? Try again
  - honkhonkpants 10y ago
    The point is: whatever your opinions are of Google's or Facebook's source code management process, and whatever you think the defects are of that process, and no matter what improvements you think could be made, these defects and shortcomings have demonstrably not stood in the way of these companies completing the largest software projects in history.

by pekk 10y ago

argonaut 10y ago

Google, Facebook, and Twitter all use monolithic repos (I'm taking monolithic to mean 50+% is in one repo, not necessarily everything in one repo), so no, not just Facebook. I'm going to assume there are very good reasons for doing so (and there are - you can look up past discussions on HN).

sshumaker 10y ago

Wrong. Google has the exact same problem, only they chose to rewrite Perforce. The scale of these codebases is way outside what most people have experienced and it might behoove people to realize that a lot of conventional wisdom might not apply. E.g. last time I checked, Google's codebase was >2Billion lines of code. Google has 30,000 engineers working on a "monolithic" codebase and remain relatively productive despite (or perhaps because of it). There are whole sets of different problems at this scale.

emu 10y ago
See https://www.youtube.com/watch?v=W71BTkUbdqE https://www.youtube.com/watch?v=W71BTkUbdqE for a lot more details on what Google does.
- kyrra 10y ago
  And Google has been contributing to Mercurial. Like subtree: https://bitbucket.org/Google/narrowhg https://bitbucket.org/Google/narrowhg
pvelagal 10y ago
2 billion lines of code @ 80 characters per line (with 2 bytes per character) = 320 GB of code, fits on my mac SSD. Wonder if 30,0000 engineers , sending 30,000 requests per second to server(s) with 320GB of code (can be put in memory) is a huge problem (edit) in terms of scale, especially when these same engineers build systems for handling billions of requests ?
- steffan 10y ago
  You're not considering that it's not just the current 2 billion lines of code - it's the history of the repo, branches, forks, possibly binaries thrown in.
  - pvelagal 10y ago
    I never built any source code system, so not an expert, but isnt branching snapshotting the current version of file(s) (strings), and newer versions are about calculating deltas. Arent, most of the Operations in a source code mgmt system, are diffs and merges of strings. Basically, my point is, scale problems associated with these systems are not at the same level as other scalability problems handled by google's core competency areas such as search which deal with billions of requests or peta bytes of data. Not that they are easy, but that orders of magnitude smaller.
amaks 10y ago
It's also very beneficial to the code health as it allows to make global refactoring, e.g. adopting new language features, improving shared libraries, etc.

joeblubaugh 10y ago

I've worked in places where we depended on external repos (Maven, Gradle, other Ivy-based things) and places where all code has to live in a mono-repo. Thus far I've preferred the mono-repo mostly for dependency management reasons. Whether you have a lot of internal dependencies or external dependencies, you get similar benefits: - All the changes to internal dependencies are in your revision history, across the company. Shared internal library upgrades are picked up quickly and propagate throughout applications with low delay. - It's easier to have shared versions of external dependencies "automatically" instead of establishing policies that need a human to enforce. This makes it easier to roll out new versions and bug fixes. - It's easier to do system-wide improvements in code quality. Replacing common bad code with better implementations is something I've seen across code bases at both Google and Twitter and they have been beneficial. I think nobody should be trying to implement a system like this if you've got fewer than ~600 SW engineers, though. Small groups don't have as much drift or system-wide refactorings that give you benefit.

joneholland 10y ago
At Expedia, we were a gigantic perforce repository, but the trend has moved to hundreds of smaller repositories and applications hosted in enterprise github. Package management is simple, we push common packages into sonatype nexus with semantic versioning. Personally I prefer strong versioning and allowing teams to upgrade common libraries at their own pace, but we've had a few times where we needed to quickly upgrade every repository (i.e., a security patch). You have a few options: 1. You can unpublish the old dependency, breaking all builds until they upgrade. (Frowned upon) 2. You can write a script that identifies all repository owners that depend on you and send out a upgrade by X date email. 3. You can script pull request creation and submit hundreds of pulls to upgrade the dependency. So far, things have worked out well, and I never want to go back to the monorepo.

newobj 10y ago

I've worked at both FB (monorepo) and Amazon (massively federated repo where you only check out tiny slices at a time; a huge package dependency system pulls things together), and I can testify that monorepo is great, and federated repo with dependencies-via-packaging is awful. It's slower to work with, harder to coordinate wide scale changes, slower to work with, the cognitive load is much higher, and it's slower to work with. Did I mention it's higher friction and slower to work with?

olalonde 10y ago
What do people mean by monorepo exactly (e.g. at Facebook)? For instance, FB's open source projects aren't all hosted in the same "open source monorepo", why not? And what goes into the monorepos exactly? All the closed source code? As a "one-man-team" who uses at least 15 different repos, it's hard for me to imagine how a massive company would manage things within a single repo and no package management. Also, aren't any of those monorepo companies concerned that a single rogue employee or stolen laptop could leak their entire source code? I can only guess I'm misunderstanding what is meant by monorepo and how they're used...
- encoderer 10y ago
  Friends tell me that very sensitive code (ranking algos, etc) is not in the repo. Personally, I also favor a single repo. You manage it the way you manage separate packages: with organization and some discipline. The magic is that command line tools like grep, sed & awk--and static analysis tools for refactoring--work really well. You can change a method signature and it just works. I've been part of monolith-breaking before (most recently at Trulia) and it definitely adds friction to the development process to work across such a rich graph of package dependencies.
  - honkhonkpants 10y ago
    Sensitive code _is_ in the repo, you just can't read it. This allows you to change library code, run the tests of the sensitive application code, and know that it worked, without being able to read the sensitive code.

honkhonkpants 10y ago

I love it when random commenters point out the obvious problems of the world's wealthiest, most successful companies.

1stop 10y ago
I know, do you reckon Facebook engineers are sitting awkwardly in a meeting with this comment page up: "Why didn't we think of this!!!?"
logor 10y ago
Implying you have to have lots of money to be correct in your software management approach. Try again. Maybe it's because these unknowns can't fathom working with such tightly wound systems that after all the code reviews are done your changes are irrelevant and need to be updated again and go through another review. In before straw man, but other than pointing out day to day issues, maybe you should try to understand the argument against trying to fit everything all in one place. Do you really want to be the maintainer of all those third party libraries you imported? Do you not allow third party and adopt "not invented here"? Try again
- honkhonkpants 10y ago
  The point is: whatever your opinions are of Google's or Facebook's source code management process, and whatever you think the defects are of that process, and no matter what improvements you think could be made, these defects and shortcomings have demonstrably not stood in the way of these companies completing the largest software projects in history.