8 ms·

_Programming_ is more pleasant when you break your codebase into isolated components. Instead of trying to force git to adapt to these monolithic repositories,

_Programming_ is more pleasant when you break your codebase into isolated components. Instead of trying to force git to adapt to these monolithic repositories, we should be breaking them down into manageable pieces for developers to work on. If you still need 10 years of history, just keep the old repository around for archival purposes. [Edit] Not that I'm against GVFS, there is nothing wrong with not wanting to download the entire repository in some cases.

aseipp 10y ago
Many monolithic repositories still have very clear isolation and component separation. There's a difference between "I'm doing this out of good taste" and "I'm doing this because this tool makes my life insufferable, otherwise". For dedicated organizations, with active development -- singular repos have some keen benefits, IMO. Git is absolutely bad at handling even moderately large repositories (unless it is 100% text based for its entire life), and its support for handling submodules completely sucks. I didn't even work on a multi-million line codebase. Few hundred kLOC. But with like 20 submodules though (library components), and the amount of organizational overhead is annoying as shit. (Not to mention the tools we developed to stop the active ~20 developers from footgunning themselves when they committed non-existent pointers, explaining carefully how they worked, etc). We have a binary or two, but we are anxious and careful to control and update them when we do (they were added before our time, and every update at this point on such a long-lived repo can visibly affect performance. Splitting it out is even more annoying ultimately and loses our history, and does not reduce the size of the original repo). Git is just bad here. Almost everyone is a volunteer. This kind of shit is a waste of time for everyone, frankly. What if git was 5x as slow as it was now? Would that mean we should create 5x more repositories, divide every repository up into 5 more? Delete your historic repositores with 5x frequency? Would trying to improve that mean we're trying to "force git to adapt for monolithic repositories", as you claim? There are very real limits to this kind of reasoning, bending backwards for your tools. Tools like Bitkeeper are far better at this and have a way better UI, out of the box. The "Product Lines" feature means subrepositories are much more transparent for all users (with full push/pull bidirectional interoperation, which is absolutely critical, and it does crazy things like -- gasp, cloning all subrepositores by default). But BitKeeper can also handle 50GB repositories fairly easily and quickly, including large binary files. No multi-repos necessary, no LFS. It "just works". That's excellent and it removes concerns and worries you might have later. I also think you understate the true value of continuous history. It's not something to toss aside so quickly, and any extra effort to use it means you lose a lot of insight. Maybe if you don't have to dig old code constantly (or your company throws away/rewrites its product constantly I guess). In my current job, for the OSS project I worked on -- it's got VCS history dating back to 1995. Yes, I have traced changes, design decisions, old relics, and even bugs back nearly ~20 years. Why? Because it was easy and there, and the best way to discover why something was done. It is absolutely one of the most important discovery and historic tools, for this reason. Many developers -- including myself, use it all the time. New developers cannot possibly understand the rational of an obscure change 10 years ago by someone with a PhD without this kind of help, at least not easily. Some files may not have been touched in like 8 years! This repository is also relatively quick to clone, luckily. The 20 submodules make it a damper. One of my last jobs was working on a product that was 15 years old, and had only one repository. They stored binaries in it, but it had had many of them updated over the years (small JAR files, mostly, one copy of GCC). It took 6 hours to checkout, but honestly that was the main thing that sucked - performance. However, all of the same benefits applied. And that one truly was continuous -- you could get a look at the repo, as it was built for customers, years ago, to look at the behavior. Atomic refactorings across years of code, over a million lines of code, and 50+ developers and libraries was possible in a single commit. That's a good thing. But the performance is a tool deficiency, not something to be hand waved away. That second repository, the 15 year one, was like, 50GB total. That's peanuts in today's terms. Other VCSs would have been much more tolerable (including BK, or probably something like Perforce, in terms of pure speed).
- chris_wot 10y ago
  In my current job, for the OSS project I worked on -- it's got VCS history dating back to 1995. Yes, I have traced changes, design decisions, old relics, and even bugs back nearly ~20 years. Why? Because it was easy and there, and the best way to discover why something was done. It is absolutely one of the most important discovery and historic tools, for this reason. Many developers -- including myself, use it all the time. I have to second this. When I was working on the VCL component of LibreOffice I often had to backtrack through positively ancient history to try to understand decisions. Unfortunately, history before 2000 is not available, and even worse was the absolutely stupid decision someone made to merge changes but not keep commit history for branches - consequently there are poor summaries of multiple changes for a single code commit. It can be incredibly frustrating!
  - luckydude 10y ago
    I'll third this. I hired a guy, a really bright guy, and put him to work. He was debugging some problem, I think a windows problem, and found the file that contained the code with the problem, looked at it, and said "this code is gross, I'm gonna rewrite it". Did I mention he was bright? He was, and he took a look at the history in bk revtool (a graphical tool, shows you the dag, you can left/right click on a pair of nodes and see diffs, double click on a node and see that version of the file). He double clicked on the very first rev and lo and behold, there was the code he was about to type in. He said it was exactly how he imagined it. Hmm, says he. I wonder why it changed. He started clicking around and went "ohh, so on this OS they have that problem". Some more clicks "oh, and windows has this problem with wrapping pids". Etc. Had he not had that history (and had it not be really fast to click around and see it), he would have rewritten the file to be back where we started, losing all the bug fixes. History matters. Being able to see it quickly and easily matters.
- iainmerrick 10y ago
  Git is absolutely bad at handling even moderately large repositories (unless it is 100% text based for its entire life), and its support for handling submodules completely sucks. [...] Tools like Bitkeeper are far better at this and have a way better UI, out of the box. Wait, are you saying... Git is a half-assed copy of the just the bits of BitKeeper that were needed for the Linux kernel?!? Written by people who don't care about UI? (I'm mostly joking, of course, Git is great in many ways. But it's so terrible in others. The living embodiment of "worse is better".)
iainmerrick 10y ago
I have done a (very) little bit of work in Chromium/WebKit, which at the time was split across two repos. The dance you had to do to make a change spanning both repos (make a commit in repo A that adds a hook for the thing you need but still compiles, make repo B use it, maybe a third commit to tidy up A) was incredibly tedious and I'm sure introduced extra bugs. When the Chromium team forked WebKit (renaming it Blink) they merged it into Chromium, specifically citing ease of development. To some extent I think the problems with split repositories are just down to bad design. WebKit was never a nice clean API, it was just a mess of whatever hacks Apple and Google needed for their respective browsers. The Chromium/WebKit split wasn't made on sensible engineering grounds, it just reflected the Google/Apple administrative divisions (Conway's law: "organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations") I still feel like modular code is better, and properly designed modular code can work well in modular repositories. But good modular code can work in monorepos too. Monorepos have the advantage that you can do huge codebase-wide refactorings atomically. Clang seems to use that approach -- all the internal APIs are constantly in flux.
- jamesmiller5 10y ago
  Gerrit, the review system Chromium uses does support the exact feature of committing across repo boundaries using "Topics" . You can verify a "Topic" is atomically submittable across repositories before submission and then commit it to those repos all at the same time. https://www.gerritcodereview.com/releases/2.12.md#New-Features https://www.gerritcodereview.com/releases/2.12.md#New-Featur...
  - iainmerrick 10y ago
    Sounds good! Although it says: "This setting should be considered experimental, and is disabled by default" I assume all the repos involved have to be managed by the same Gerrit instance too. In that case it's not clear to me what you actually gain from multiple repos (besides working around Git's scaling issues).
    - 10y ago

aseipp 10y ago

Many monolithic repositories still have very clear isolation and component separation. There's a difference between "I'm doing this out of good taste" and "I'm doing this because this tool makes my life insufferable, otherwise". For dedicated organizations, with active development -- singular repos have some keen benefits, IMO. Git is absolutely bad at handling even moderately large repositories (unless it is 100% text based for its entire life), and its support for handling submodules completely sucks. I didn't even work on a multi-million line codebase. Few hundred kLOC. But with like 20 submodules though (library components), and the amount of organizational overhead is annoying as shit. (Not to mention the tools we developed to stop the active ~20 developers from footgunning themselves when they committed non-existent pointers, explaining carefully how they worked, etc). We have a binary or two, but we are anxious and careful to control and update them when we do (they were added before our time, and every update at this point on such a long-lived repo can visibly affect performance. Splitting it out is even more annoying ultimately and loses our history, and does not reduce the size of the original repo). Git is just bad here. Almost everyone is a volunteer. This kind of shit is a waste of time for everyone, frankly. What if git was 5x as slow as it was now? Would that mean we should create 5x more repositories, divide every repository up into 5 more? Delete your historic repositores with 5x frequency? Would trying to improve that mean we're trying to "force git to adapt for monolithic repositories", as you claim? There are very real limits to this kind of reasoning, bending backwards for your tools. Tools like Bitkeeper are far better at this and have a way better UI, out of the box. The "Product Lines" feature means subrepositories are much more transparent for all users (with full push/pull bidirectional interoperation, which is absolutely critical, and it does crazy things like -- gasp, cloning all subrepositores by default). But BitKeeper can also handle 50GB repositories fairly easily and quickly, including large binary files. No multi-repos necessary, no LFS. It "just works". That's excellent and it removes concerns and worries you might have later. I also think you understate the true value of continuous history. It's not something to toss aside so quickly, and any extra effort to use it means you lose a lot of insight. Maybe if you don't have to dig old code constantly (or your company throws away/rewrites its product constantly I guess). In my current job, for the OSS project I worked on -- it's got VCS history dating back to 1995. Yes, I have traced changes, design decisions, old relics, and even bugs back nearly ~20 years. Why? Because it was easy and there, and the best way to discover why something was done. It is absolutely one of the most important discovery and historic tools, for this reason. Many developers -- including myself, use it all the time. New developers cannot possibly understand the rational of an obscure change 10 years ago by someone with a PhD without this kind of help, at least not easily. Some files may not have been touched in like 8 years! This repository is also relatively quick to clone, luckily. The 20 submodules make it a damper. One of my last jobs was working on a product that was 15 years old, and had only one repository. They stored binaries in it, but it had had many of them updated over the years (small JAR files, mostly, one copy of GCC). It took 6 hours to checkout, but honestly that was the main thing that sucked - performance. However, all of the same benefits applied. And that one truly was continuous -- you could get a look at the repo, as it was built for customers, years ago, to look at the behavior. Atomic refactorings across years of code, over a million lines of code, and 50+ developers and libraries was possible in a single commit. That's a good thing. But the performance is a tool deficiency, not something to be hand waved away. That second repository, the 15 year one, was like, 50GB total. That's peanuts in today's terms. Other VCSs would have been much more tolerable (including BK, or probably something like Perforce, in terms of pure speed).

chris_wot 10y ago
In my current job, for the OSS project I worked on -- it's got VCS history dating back to 1995. Yes, I have traced changes, design decisions, old relics, and even bugs back nearly ~20 years. Why? Because it was easy and there, and the best way to discover why something was done. It is absolutely one of the most important discovery and historic tools, for this reason. Many developers -- including myself, use it all the time. I have to second this. When I was working on the VCL component of LibreOffice I often had to backtrack through positively ancient history to try to understand decisions. Unfortunately, history before 2000 is not available, and even worse was the absolutely stupid decision someone made to merge changes but not keep commit history for branches - consequently there are poor summaries of multiple changes for a single code commit. It can be incredibly frustrating!
- luckydude 10y ago
  I'll third this. I hired a guy, a really bright guy, and put him to work. He was debugging some problem, I think a windows problem, and found the file that contained the code with the problem, looked at it, and said "this code is gross, I'm gonna rewrite it". Did I mention he was bright? He was, and he took a look at the history in bk revtool (a graphical tool, shows you the dag, you can left/right click on a pair of nodes and see diffs, double click on a node and see that version of the file). He double clicked on the very first rev and lo and behold, there was the code he was about to type in. He said it was exactly how he imagined it. Hmm, says he. I wonder why it changed. He started clicking around and went "ohh, so on this OS they have that problem". Some more clicks "oh, and windows has this problem with wrapping pids". Etc. Had he not had that history (and had it not be really fast to click around and see it), he would have rewritten the file to be back where we started, losing all the bug fixes. History matters. Being able to see it quickly and easily matters.
iainmerrick 10y ago
Git is absolutely bad at handling even moderately large repositories (unless it is 100% text based for its entire life), and its support for handling submodules completely sucks. [...] Tools like Bitkeeper are far better at this and have a way better UI, out of the box. Wait, are you saying... Git is a half-assed copy of the just the bits of BitKeeper that were needed for the Linux kernel?!? Written by people who don't care about UI? (I'm mostly joking, of course, Git is great in many ways. But it's so terrible in others. The living embodiment of "worse is better".)

iainmerrick 10y ago

I have done a (very) little bit of work in Chromium/WebKit, which at the time was split across two repos. The dance you had to do to make a change spanning both repos (make a commit in repo A that adds a hook for the thing you need but still compiles, make repo B use it, maybe a third commit to tidy up A) was incredibly tedious and I'm sure introduced extra bugs. When the Chromium team forked WebKit (renaming it Blink) they merged it into Chromium, specifically citing ease of development. To some extent I think the problems with split repositories are just down to bad design. WebKit was never a nice clean API, it was just a mess of whatever hacks Apple and Google needed for their respective browsers. The Chromium/WebKit split wasn't made on sensible engineering grounds, it just reflected the Google/Apple administrative divisions (Conway's law: "organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations") I still feel like modular code is better, and properly designed modular code can work well in modular repositories. But good modular code can work in monorepos too. Monorepos have the advantage that you can do huge codebase-wide refactorings atomically. Clang seems to use that approach -- all the internal APIs are constantly in flux.

jamesmiller5 10y ago
Gerrit, the review system Chromium uses does support the exact feature of committing across repo boundaries using "Topics" . You can verify a "Topic" is atomically submittable across repositories before submission and then commit it to those repos all at the same time. https://www.gerritcodereview.com/releases/2.12.md#New-Features https://www.gerritcodereview.com/releases/2.12.md#New-Featur...
- iainmerrick 10y ago
  Sounds good! Although it says: "This setting should be considered experimental, and is disabled by default" I assume all the repos involved have to be managed by the same Gerrit instance too. In that case it's not clear to me what you actually gain from multiple repos (besides working around Git's scaling issues).
  - 10y ago