3 ms·

>10s of thousands of commits Lol! The scale at which these companies are (ab)using git is multiple orders of magnitudes greater than that. You see, they think

>10s of thousands of commits Lol! The scale at which these companies are (ab)using git is multiple orders of magnitudes greater than that. You see, they think it’s a good idea to put every piece of code ever written in the whole company in the same repo. They call it the “monorepo”, and it’s hundreds of gigabytes with many millions of commits. Microsoft even created a virtual filesystem which they run git on top of: https://github.com/Microsoft/gvfs https://github.com/Microsoft/gvfs

edejong 8y ago
Well, Google has succesfully defended their monorepo approach on multiple occasions. So, before dissing this approach as 'non-SV-canon', read Google's [1] and Twitters [2] accounts on why they do what they do. [1] Why Google Stores Billions of Lines of Code in a Single Repository (2016) https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext https://cacm.acm.org/magazines/2016/7/204032-why-google-stor... [2] On monolithic repositories (2014) https://gregoryszorc.com/blog/2014/09/09/on-monolithic-repositories/ https://gregoryszorc.com/blog/2014/09/09/on-monolithic-repos...
- influx 8y ago
  Odd that people are including Twitter as a data point in the monorepo debate. I was there when they transitioned to a Git monorepo, and it nearly killed the companies productivity. Imagine having to torrent a starter pack of the repo, then trying to sync and failing multiple times. Then after it syncs, it could take minutes to do common operations such as change a branch, or check in a file. Not to mention many developer laptops had to have their SSD upgraded as they didn't have enough space to get the whole thing and do work. I'm sure they've improved it since then, but this was one of many things Twitter simply cargo culted from Google without any real benefit.
  - avar 8y ago
    [Source: I contribute to Git and talked to Twitter's "git guys" at the time at Git Merge, am not affiliated with Twitter in any way] My understanding is that they didn't cargo-cult anything. They made a conscious choice at Twitter to pursue the monorepo model. One reason they mentioned is that while they have a lot of services running all over the place, those services tend to heavily use the same underlying base libraries. Think the library that validates Twitter usernames, parses out the text of a tweet etc. By having a monorepo they can easily atomically change those libraries for all their consumers. This tends to be why companies go for the monorepo model in general. This is all information from 2016-ish. Around that time users with *@twopensource.com E-Mail addresses stopped contributing to git.git. I have no idea why, presumably changed internal priorities or something like that. Maybe their internal repository structure changed so they didn't need to author their own performance patches anymore, maybe not...
    - influx 8y ago
      Sure, that was the justification, but when you literally can’t switch branches without waiting for a minute, I fail to see the benefit. As I recall the developers authoring the custom git patches left the company around that time. I don’t know what happened after that. There was talk of moving to mercurial with facebooks patches, but I had already left.
      - avar 8y ago
        It would be interesting to get some background info on that branch switching. Switching branches per-se is really cheap in git, on linux.git it takes 200ms, around the time it takes to run a cold "git status". This is because just creating a new branch doesn't need to touch the tree at all. I know "status" was a bottleneck at Twitter. They had the first inotify patches to git, but it never made it in. Eventually the patches Microsoft wrote to do the same thing made it in. What can get expensive is if the tree you're switching to has drastically different content. On the latest linux.git (~60k files) switching to the 2.6.* era takes around 10 seconds or me (~10k files). I'd expect with a monorepo model like what Twitter had (has?) that most working branches are relatively up-to-date with the master branch, so switching should be cheap. Was it spending most of its time in the "Checking out files" phase, or before that? Was this on e.g. OSX with some corporate virus scanner running where each I/O syscall was wrapped? That can drastically slow things down. Or was it just that the repository truly had a ridiculous amount of files in the checkout (around 1 million?). Edit: I remember now that I have an old copy of 2015-04-03-1M-git.git which David Turner of Twitter publicly shared a while back, it was meant to emulate the size and shape of Twitter's monorepo. It has around 230k files.
- detaro 8y ago
  > So, before dissing this approach as 'non-SV-canon', what's that supposed to mean and how do you get it from the parent comment?
- tzahola 8y ago
  Wait a sec, I thought the monorepo was the SV canon. As in, "we have to keep our velocity high, to deliver customer value multiple times a day, so everyone should be able to make changes across the whole stack in a single commit, and release to production in five minutes." In my opinion it boils down to a single factor: if you don't have proper APIs between your components, then you will need to make cross-cutting changes across the whole codebase on a daily basis, thus you need everything in the same repo to work effectively. If you don't have the time to work out the proper boundaries/APIs between your components, then sure, by all means go for the monorepo! But that's like not fixing a blown tire just because you would need to stop to do it.
  - Orphis 8y ago
    At large scale, you still need to have proper API boundaries, even with a monorepo. Otherwise, even with the best tooling, you will never be able to change anything. The APIs are there to ensure the behavior won't change and break anything. In a monorepo, that is easier to check by running all the tests that are affected by your code changes. The actual programming "syntax" is a detail though, and that can be changed (atomically everywhere) anytime with proper tooling. It's only in the old traditional split model that APIs are supposed to be a guarantee for syntax + behavior (+ sometimes ABI).
  - hinkley 8y ago
    Monorepo is also an answer (though perhaps not the best) to the problem of having dependencies that are volatile. If you can build your design philosophy on getting library code to be stable fairly quickly, then making a change to the system need not start an avalanche of rebuilds and redeployments, making monorepo less of a win. But people like their kitchen sinks, their one stop shopping. And who has time to think about where the best place for a piece of code is. We are doing Agile! And that apparently means “no architecture” to way too many people.
- daveFNbuck 8y ago
  Google doesn't use Git, so their defense of a monorepo doesn't say anything about whether a monorepo is an abuse of Git. Your Twitter link defends monorepos in principle but is very clear about how Git is not designed to handle monorepos well. This seems to support the thesis that a monorepo is an abuse of Git.
shoo 8y ago
further monorepo reading: Paul Hammant also likes to write about this and related subjects. https://paulhammant.com/categories.html#Monorepos https://paulhammant.com/categories.html#Monorepos

edejong 8y ago

Well, Google has succesfully defended their monorepo approach on multiple occasions. So, before dissing this approach as 'non-SV-canon', read Google's [1] and Twitters [2] accounts on why they do what they do. [1] Why Google Stores Billions of Lines of Code in a Single Repository (2016) https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext https://cacm.acm.org/magazines/2016/7/204032-why-google-stor... [2] On monolithic repositories (2014) https://gregoryszorc.com/blog/2014/09/09/on-monolithic-repositories/ https://gregoryszorc.com/blog/2014/09/09/on-monolithic-repos...

influx 8y ago
Odd that people are including Twitter as a data point in the monorepo debate. I was there when they transitioned to a Git monorepo, and it nearly killed the companies productivity. Imagine having to torrent a starter pack of the repo, then trying to sync and failing multiple times. Then after it syncs, it could take minutes to do common operations such as change a branch, or check in a file. Not to mention many developer laptops had to have their SSD upgraded as they didn't have enough space to get the whole thing and do work. I'm sure they've improved it since then, but this was one of many things Twitter simply cargo culted from Google without any real benefit.
- avar 8y ago
  [Source: I contribute to Git and talked to Twitter's "git guys" at the time at Git Merge, am not affiliated with Twitter in any way] My understanding is that they didn't cargo-cult anything. They made a conscious choice at Twitter to pursue the monorepo model. One reason they mentioned is that while they have a lot of services running all over the place, those services tend to heavily use the same underlying base libraries. Think the library that validates Twitter usernames, parses out the text of a tweet etc. By having a monorepo they can easily atomically change those libraries for all their consumers. This tends to be why companies go for the monorepo model in general. This is all information from 2016-ish. Around that time users with *@twopensource.com E-Mail addresses stopped contributing to git.git. I have no idea why, presumably changed internal priorities or something like that. Maybe their internal repository structure changed so they didn't need to author their own performance patches anymore, maybe not...
  - influx 8y ago
    Sure, that was the justification, but when you literally can’t switch branches without waiting for a minute, I fail to see the benefit. As I recall the developers authoring the custom git patches left the company around that time. I don’t know what happened after that. There was talk of moving to mercurial with facebooks patches, but I had already left.
    - avar 8y ago
      It would be interesting to get some background info on that branch switching. Switching branches per-se is really cheap in git, on linux.git it takes 200ms, around the time it takes to run a cold "git status". This is because just creating a new branch doesn't need to touch the tree at all. I know "status" was a bottleneck at Twitter. They had the first inotify patches to git, but it never made it in. Eventually the patches Microsoft wrote to do the same thing made it in. What can get expensive is if the tree you're switching to has drastically different content. On the latest linux.git (~60k files) switching to the 2.6.* era takes around 10 seconds or me (~10k files). I'd expect with a monorepo model like what Twitter had (has?) that most working branches are relatively up-to-date with the master branch, so switching should be cheap. Was it spending most of its time in the "Checking out files" phase, or before that? Was this on e.g. OSX with some corporate virus scanner running where each I/O syscall was wrapped? That can drastically slow things down. Or was it just that the repository truly had a ridiculous amount of files in the checkout (around 1 million?). Edit: I remember now that I have an old copy of 2015-04-03-1M-git.git which David Turner of Twitter publicly shared a while back, it was meant to emulate the size and shape of Twitter's monorepo. It has around 230k files.
detaro 8y ago
> So, before dissing this approach as 'non-SV-canon', what's that supposed to mean and how do you get it from the parent comment?
tzahola 8y ago
Wait a sec, I thought the monorepo was the SV canon. As in, "we have to keep our velocity high, to deliver customer value multiple times a day, so everyone should be able to make changes across the whole stack in a single commit, and release to production in five minutes." In my opinion it boils down to a single factor: if you don't have proper APIs between your components, then you will need to make cross-cutting changes across the whole codebase on a daily basis, thus you need everything in the same repo to work effectively. If you don't have the time to work out the proper boundaries/APIs between your components, then sure, by all means go for the monorepo! But that's like not fixing a blown tire just because you would need to stop to do it.
- Orphis 8y ago
  At large scale, you still need to have proper API boundaries, even with a monorepo. Otherwise, even with the best tooling, you will never be able to change anything. The APIs are there to ensure the behavior won't change and break anything. In a monorepo, that is easier to check by running all the tests that are affected by your code changes. The actual programming "syntax" is a detail though, and that can be changed (atomically everywhere) anytime with proper tooling. It's only in the old traditional split model that APIs are supposed to be a guarantee for syntax + behavior (+ sometimes ABI).
- hinkley 8y ago
  Monorepo is also an answer (though perhaps not the best) to the problem of having dependencies that are volatile. If you can build your design philosophy on getting library code to be stable fairly quickly, then making a change to the system need not start an avalanche of rebuilds and redeployments, making monorepo less of a win. But people like their kitchen sinks, their one stop shopping. And who has time to think about where the best place for a piece of code is. We are doing Agile! And that apparently means “no architecture” to way too many people.
daveFNbuck 8y ago
Google doesn't use Git, so their defense of a monorepo doesn't say anything about whether a monorepo is an abuse of Git. Your Twitter link defends monorepos in principle but is very clear about how Git is not designed to handle monorepos well. This seems to support the thesis that a monorepo is an abuse of Git.

shoo 8y ago

further monorepo reading: Paul Hammant also likes to write about this and related subjects. https://paulhammant.com/categories.html#Monorepos https://paulhammant.com/categories.html#Monorepos