4 ms·

In my last job, I tried using a monorepo approach, merging about 50 microservices into a single repository. After several months of growth, the size of the enti

In my last job, I tried using a monorepo approach, merging about 50 microservices into a single repository. After several months of growth, the size of the entire git repository reached over 100GB. We had to create monthly checkpoints (.tar.gz) to facilitate faster time traveling or quick cloning for new team members. This experience taught me that perhaps git is not the best tool for managing a monorepo.

mananaysiempre 2y ago
> We had to create monthly checkpoints (.tar.gz) Not `git bundle`?
mbb70 2y ago
What were you checking in? I've worked in monorepos with a lot of generated code (millions of lines) and 10 years of commits and the total repo size never went over a few gb.
- ethegwo 2y ago
  Just golang codes, there are about 500 hundreds developers work on it.
  - pas 2y ago
    someone else mentioned that the the kernel is still less than 2G
  - Risord 2y ago
    TLDR: Monorepos have no issues in "real world". (= My bubble, I am quite sure I haven't ever even met person face to face who have been in project of even 100 devs.)
  - deleted 2y ago
    [deleted]
  - Capricorn2481 2y ago
    That's billions of lines of code if it's just Golang. I would imagine most of that was autogenerated code that no one was reviewing. This is not a problem with monorepos.
danpalmer 2y ago
There's some truth to this. I'd say that perhaps Git is unsuitable above certain thresholds on numbers of files, numbers of commits, and size of data, but, that the size of data (and the fact that for certain file types this gets blown out on edits) is probably the one most people hit first – it sounds like you may have. I've heard of git monorepos with >1k microservices, and it sounded like it wasn't a huge problem, but I think it likely had significantly less than 100GB. In fact I feel like you need to be committing large binary files to get towards that size.
- azangru 2y ago
  > I'd say that perhaps Git is unsuitable above certain thresholds on numbers of files Scott Chacon in a recent talk about git[0] said that Microsoft uses git to version control Windows. They have ~3.5 million files in the repo, which is about 300 gigabytes in size. There are some tweaks to improve git performance in those scenarios; but still, I'd say it's a pretty impressive threshold. [0] - https://www.youtube.com/watch?v=aolI_Rz0ZqY https://www.youtube.com/watch?v=aolI_Rz0ZqY
  - augusto-moura 2y ago
    There's a lot of "tweaks", they implemented a whole toolchain for virtualizing large repos: https://github.com/microsoft/scalar https://github.com/microsoft/scalar
    - re-thc 2y ago
      Most of those "tweaks" are no longer required. A lot of it has been merged back to git itself.
the_duke 2y ago
How did you manage to grow 50 microservices to 100GB? A fresh checkout of the whole Linux kernel currently is 1.8GB ... Were you checking in lots of vendored dependencies? Or 10s of GBs of images/videos/AI models?
- hu3 2y ago
  I'm betting autogenerated file like protobuf definitions.
  - ethegwo 2y ago
    Yes, there are lots of auto generated thrift templates, we have tried our best to keep deterministic generation.
    - acedTrex 2y ago
      With that much generation that im assuming changes regularly you almost want a seperate solution or to gitignore them and generate the files at build time...
augusto-moura 2y ago
How would that differ from having the 50 microservices cloned separately? Maybe you want to say that perhaps some of the microservices were not that useful and didn't need to be versioned as well? If that is the case I think you could benefit from Scalar [1]. Microsoft created it because of similar problems with size and binaries [1]: https://github.com/microsoft/scalar https://github.com/microsoft/scalar
- ethegwo 2y ago
  Yes, Scalar would be helpful. The problem is that not every developer needs all the code in the repository for their daily work. However, Git makes it difficult to do partial checkouts.
- slightwinder 2y ago
  > How would that differ from having the 50 microservices cloned separately? Usually, not everyone needs every single microservice, so they only checkout the handful they need, instead of the full monster. And usually one will accumulate repos over time, instead of getting them all at once. Similar, you will also only pull updates for the repo necessary for your work today, instead of getting the full update the whole company creates every day.
  - msoad 2y ago
    Sparse checkout is a new Git feature that can help with this

mananaysiempre 2y ago

> We had to create monthly checkpoints (.tar.gz) Not `git bundle`?

mbb70 2y ago

What were you checking in? I've worked in monorepos with a lot of generated code (millions of lines) and 10 years of commits and the total repo size never went over a few gb.

ethegwo 2y ago
Just golang codes, there are about 500 hundreds developers work on it.
- pas 2y ago
  someone else mentioned that the the kernel is still less than 2G
- Risord 2y ago
  TLDR: Monorepos have no issues in "real world". (= My bubble, I am quite sure I haven't ever even met person face to face who have been in project of even 100 devs.)
- deleted 2y ago
  [deleted]
- Capricorn2481 2y ago
  That's billions of lines of code if it's just Golang. I would imagine most of that was autogenerated code that no one was reviewing. This is not a problem with monorepos.

danpalmer 2y ago

There's some truth to this. I'd say that perhaps Git is unsuitable above certain thresholds on numbers of files, numbers of commits, and size of data, but, that the size of data (and the fact that for certain file types this gets blown out on edits) is probably the one most people hit first – it sounds like you may have. I've heard of git monorepos with >1k microservices, and it sounded like it wasn't a huge problem, but I think it likely had significantly less than 100GB. In fact I feel like you need to be committing large binary files to get towards that size.

azangru 2y ago
> I'd say that perhaps Git is unsuitable above certain thresholds on numbers of files Scott Chacon in a recent talk about git[0] said that Microsoft uses git to version control Windows. They have ~3.5 million files in the repo, which is about 300 gigabytes in size. There are some tweaks to improve git performance in those scenarios; but still, I'd say it's a pretty impressive threshold. [0] - https://www.youtube.com/watch?v=aolI_Rz0ZqY https://www.youtube.com/watch?v=aolI_Rz0ZqY
- augusto-moura 2y ago
  There's a lot of "tweaks", they implemented a whole toolchain for virtualizing large repos: https://github.com/microsoft/scalar https://github.com/microsoft/scalar
  - re-thc 2y ago
    Most of those "tweaks" are no longer required. A lot of it has been merged back to git itself.

the_duke 2y ago

How did you manage to grow 50 microservices to 100GB? A fresh checkout of the whole Linux kernel currently is 1.8GB ... Were you checking in lots of vendored dependencies? Or 10s of GBs of images/videos/AI models?

hu3 2y ago
I'm betting autogenerated file like protobuf definitions.
- ethegwo 2y ago
  Yes, there are lots of auto generated thrift templates, we have tried our best to keep deterministic generation.
  - acedTrex 2y ago
    With that much generation that im assuming changes regularly you almost want a seperate solution or to gitignore them and generate the files at build time...

augusto-moura 2y ago

How would that differ from having the 50 microservices cloned separately? Maybe you want to say that perhaps some of the microservices were not that useful and didn't need to be versioned as well? If that is the case I think you could benefit from Scalar [1]. Microsoft created it because of similar problems with size and binaries [1]: https://github.com/microsoft/scalar https://github.com/microsoft/scalar

ethegwo 2y ago
Yes, Scalar would be helpful. The problem is that not every developer needs all the code in the repository for their daily work. However, Git makes it difficult to do partial checkouts.
slightwinder 2y ago
> How would that differ from having the 50 microservices cloned separately? Usually, not everyone needs every single microservice, so they only checkout the handful they need, instead of the full monster. And usually one will accumulate repos over time, instead of getting them all at once. Similar, you will also only pull updates for the repo necessary for your work today, instead of getting the full update the whole company creates every day.
- msoad 2y ago
  Sparse checkout is a new Git feature that can help with this