7 ms·

It's not, why do I have to checkout terabyte of code that I don't need, even if the code is modularized?

by benmarten 8y ago

slobotron 8y ago
Chances are you will end up downloading a lot of dependencies anyways, why not have git deliver it all?
- nkozyra 8y ago
  Huh? You'd download dependencies for the repos you need, not the code and dependencies for the entire company. It could be several orders of magnitude larger and with a larger organization could be a lot of unnecessary code that any given Dev may never touch.
  - pvorb 8y ago
    But imagine the increased productivity of your devs if they only had to check out a single repo. Anyone has the same organization of projects on their machine. All tools are in one place...
    - nkozyra 8y ago
      I don't understand. Where is the argument for more productivity?
      - Tempest1981 8y ago
        We have one large-ish repo that keeps showing "This repository currently has approximately 547 loose objects.” We keep pruning and gc'ing with different flags, but pulls just seem far slower than other smaller repos.
      - Too 8y ago
        A: You avoid issues such as Readme files stating, "before compiling you have to git clone ../commonA, ../commonB". These always tend to get stale so in reality you also have to git clone ../commonC wasting you tons of hours of troubleshooting. B: Developer working on daily basis in component A finds a bug in component B. He just has to change the code and commit it for review, instead of understanding the specifics of working with component B repository.
erulabs 8y ago
If a mono-repo has a terabyte of code, or if 10 small repos have 1/10th a terabyte each, what have you really gained? In any case, git LFS solves large file storage effectively, as do a number of other artifact storage solutions, and a repo with a terabyte of code is _not_ going to be trivially split apart, since it would be by a factor of thousands, the biggest codebase ever created by humankind.
- twblalock 8y ago
  If I only need to check out one of the smaller repos then I've gained quite a lot in terms of download speed, storage size, etc. Git LFS adds a lot of complexity I'd rather avoid.
  - erulabs 8y ago
    Sure but then you only have some small portion of the total infrastructure, which adds its own layer of complexity for the people reviewing your changes :P It's all trade offs, is all I'm saying - I honestly still can't decide between the two, although for all companies sub 20 people, I'd for sure stick with a single repo.
    - tracker1 8y ago
      If I'm working on Application X, wtf do I care about infrastructure code? Or for that matter, as a specific... if someone is working on Google Maps, should they care about the codebase for Google Inbox for Android?
      - matthewmacleod 8y ago
        Does Application X rely on particular infrastructure configuration? Or does Google Inbox on Android integrate with Google Maps? There are dependencies everywhere. Monorepos are one of the tools which can be used to make dealing with them easier in some cases. They’re not an absolute solution not appropriate for all circumstances, but no tool is!
      - 8y ago
nimchimpsky 8y ago
A terabyte of code ? jesus.
jchw 8y ago
No need to checkout a terabyte of code. If your repo is scaling that high, you're going to want a VFS layer. Microsoft made a VFS layer for Git. As you might imagine, you simply grab files as needed, and your version control just deals with diffs for the most part. Google's own monorepo is proprietary but the Bazel build system is open source and would work great with a VCS hooked up with a VFS layer.
- jacques_chester 8y ago
  I want to like Bazel. I really do. But on first encounter the syntax is filled with sigils that don't seem to have obvious differences or purpose for existence. Then it turns out that I and others have spent as much time fighting it as using it. Lastly the coverage of ecosystems is sparse and there does not seem to be a lot of activity around extending them -- doing the boring, tedious, unloved work of dealing with everyone's quirks and bugs and corner cases and annoyances (been there, done that). Again: I wish it was a smooth experience. Because I like the ideas very much. But it wasn't when I tried and I don't know anyone -- outside of Google -- for whom it was a smooth experience.
  - iainmerrick 8y ago
    I can’t speak to the actual implementation, but I’m surprised at your description of the syntax as “filled with sigils”, as the syntax is basically Python -- isn’t that about as easy as you can get? I find Bazel’s syntax much easier to deal with than other build languages that use JSON (essentially the same Python syntax but with lots of extra quotes everywhere and extra fussiness about where commas are allowed).
    - jacques_chester 8y ago
      bazel build //main:hello-world I'm sure the double slashes and colon have important differences. It is not obvious what they are. cc_binary( name = "hello-world", srcs = ["hello-world.cc"], deps = [ ":hello-greet", "//lib:hello-time", ], ) It's not instantly obvious why one is :hello-greet and the other is //lib:hello-time. I could swear I've seen @ floating around as well. As I said above, I am sure these are all very sensible. But I am just tired of memorising minilanguages embedded in strings. I don't want to any more.
skj 8y ago
Sounds like a tooling problem. We shouldn't use the current state of tooling as an excuse.
- jessaustin 8y ago
  Isn't the entire argument about the current (or maybe "immediately foreseeable") state of tooling? We don't really care one way or the other, in a philosophical sense. What works?
  - skj 8y ago
    When the tools aren't good enough, we can either toss up our hands and say "I guess it's always going to be like this!", or we can get to work and make better tools.
    - jessaustin 8y ago
      This is an argument about how to use current tools. TFA doesn't argue that mono will be great once we work really hard. It argues that mono is great now. Thread parent has a specific objection to that argument. You don't reasonably counter that objection with statements about morality.
      - skj 8y ago
        A few things to note: - I was replying to a comment, not the article. - The article spoke about points that were largely independent of the current or future state of tooling. Instead, it focused on fundamental issues with mono- vs poly-repo systems. Most directly, being forced to fix migrations and incompatibilities immediately rather than letting versions skew. If you want to batter someone for not arguing for or against the points in the article, you can do it with the comment I was replying to, or with your own comment just now.
mwkaufma 8y ago
We have a perforce monorepo with ~80gb total payload for the whole thing, but everyone uses streams to filter it, so that's not a problem.
- hinkley 8y ago
  I think there's a false dichotomy here. In the post yesterday one of the arguments was that if nobody checks out all of the code then what's the value of having the code all in one place? Last monorepo I worked on, individual contributors checked out just the tree they were working on (we had a suite of applications with several shared modules). We made it simple and straightforward for them to get what they wanted and ignore people whose work didn't impact them. But the senior people, who were better with architecture and version control trivia, checked out the entire thing. They would steward any cross-cutting changes that needed to be done, and make sure any callers to shared libraries were updated in the face of breaking changes. They were also backstopped by the build plans, (some of) which also checked out the entire thing.
  - mwkaufma 8y ago
    Streams aren't modules -- they're views. If someone takes you as a dependency and wants you to have visibility on them they add themselves to your stream so you pull down their directory as well.
rhacker 8y ago
It's not for everyone, but damn, why is there a TERABYTE of code? Just curious - assets? checking in binaries?
- malkia 8y ago
  Test protos. Evaluated configs. Golden data. JAR archives, etc.
alexnewman 8y ago
Signs your build system is never going to be adopted outside of people cargo culting you? - [x] Namespaces and the like without much security benefit - [x] Giant Java dependency - [x] Strange syntax and glyphs

It's not, why do I have to checkout terabyte of code that I don't need, even if the code is modularized?

slobotron 8y ago

Chances are you will end up downloading a lot of dependencies anyways, why not have git deliver it all?

nkozyra 8y ago
Huh? You'd download dependencies for the repos you need, not the code and dependencies for the entire company. It could be several orders of magnitude larger and with a larger organization could be a lot of unnecessary code that any given Dev may never touch.
- pvorb 8y ago
  But imagine the increased productivity of your devs if they only had to check out a single repo. Anyone has the same organization of projects on their machine. All tools are in one place...
  - nkozyra 8y ago
    I don't understand. Where is the argument for more productivity?
    - Tempest1981 8y ago
      We have one large-ish repo that keeps showing "This repository currently has approximately 547 loose objects.” We keep pruning and gc'ing with different flags, but pulls just seem far slower than other smaller repos.
    - Too 8y ago
      A: You avoid issues such as Readme files stating, "before compiling you have to git clone ../commonA, ../commonB". These always tend to get stale so in reality you also have to git clone ../commonC wasting you tons of hours of troubleshooting. B: Developer working on daily basis in component A finds a bug in component B. He just has to change the code and commit it for review, instead of understanding the specifics of working with component B repository.

erulabs 8y ago

If a mono-repo has a terabyte of code, or if 10 small repos have 1/10th a terabyte each, what have you really gained? In any case, git LFS solves large file storage effectively, as do a number of other artifact storage solutions, and a repo with a terabyte of code is _not_ going to be trivially split apart, since it would be by a factor of thousands, the biggest codebase ever created by humankind.

twblalock 8y ago
If I only need to check out one of the smaller repos then I've gained quite a lot in terms of download speed, storage size, etc. Git LFS adds a lot of complexity I'd rather avoid.
- erulabs 8y ago
  Sure but then you only have some small portion of the total infrastructure, which adds its own layer of complexity for the people reviewing your changes :P It's all trade offs, is all I'm saying - I honestly still can't decide between the two, although for all companies sub 20 people, I'd for sure stick with a single repo.
  - tracker1 8y ago
    If I'm working on Application X, wtf do I care about infrastructure code? Or for that matter, as a specific... if someone is working on Google Maps, should they care about the codebase for Google Inbox for Android?
    - matthewmacleod 8y ago
      Does Application X rely on particular infrastructure configuration? Or does Google Inbox on Android integrate with Google Maps? There are dependencies everywhere. Monorepos are one of the tools which can be used to make dealing with them easier in some cases. They’re not an absolute solution not appropriate for all circumstances, but no tool is!
    - 8y ago

nimchimpsky 8y ago

A terabyte of code ? jesus.

jchw 8y ago

No need to checkout a terabyte of code. If your repo is scaling that high, you're going to want a VFS layer. Microsoft made a VFS layer for Git. As you might imagine, you simply grab files as needed, and your version control just deals with diffs for the most part. Google's own monorepo is proprietary but the Bazel build system is open source and would work great with a VCS hooked up with a VFS layer.

jacques_chester 8y ago
I want to like Bazel. I really do. But on first encounter the syntax is filled with sigils that don't seem to have obvious differences or purpose for existence. Then it turns out that I and others have spent as much time fighting it as using it. Lastly the coverage of ecosystems is sparse and there does not seem to be a lot of activity around extending them -- doing the boring, tedious, unloved work of dealing with everyone's quirks and bugs and corner cases and annoyances (been there, done that). Again: I wish it was a smooth experience. Because I like the ideas very much. But it wasn't when I tried and I don't know anyone -- outside of Google -- for whom it was a smooth experience.
- iainmerrick 8y ago
  I can’t speak to the actual implementation, but I’m surprised at your description of the syntax as “filled with sigils”, as the syntax is basically Python -- isn’t that about as easy as you can get? I find Bazel’s syntax much easier to deal with than other build languages that use JSON (essentially the same Python syntax but with lots of extra quotes everywhere and extra fussiness about where commas are allowed).
  - jacques_chester 8y ago
    bazel build //main:hello-world I'm sure the double slashes and colon have important differences. It is not obvious what they are. cc_binary( name = "hello-world", srcs = ["hello-world.cc"], deps = [ ":hello-greet", "//lib:hello-time", ], ) It's not instantly obvious why one is :hello-greet and the other is //lib:hello-time. I could swear I've seen @ floating around as well. As I said above, I am sure these are all very sensible. But I am just tired of memorising minilanguages embedded in strings. I don't want to any more.

skj 8y ago

Sounds like a tooling problem. We shouldn't use the current state of tooling as an excuse.

jessaustin 8y ago
Isn't the entire argument about the current (or maybe "immediately foreseeable") state of tooling? We don't really care one way or the other, in a philosophical sense. What works?
- skj 8y ago
  When the tools aren't good enough, we can either toss up our hands and say "I guess it's always going to be like this!", or we can get to work and make better tools.
  - jessaustin 8y ago
    This is an argument about how to use current tools. TFA doesn't argue that mono will be great once we work really hard. It argues that mono is great now. Thread parent has a specific objection to that argument. You don't reasonably counter that objection with statements about morality.
    - skj 8y ago
      A few things to note: - I was replying to a comment, not the article. - The article spoke about points that were largely independent of the current or future state of tooling. Instead, it focused on fundamental issues with mono- vs poly-repo systems. Most directly, being forced to fix migrations and incompatibilities immediately rather than letting versions skew. If you want to batter someone for not arguing for or against the points in the article, you can do it with the comment I was replying to, or with your own comment just now.

mwkaufma 8y ago

We have a perforce monorepo with ~80gb total payload for the whole thing, but everyone uses streams to filter it, so that's not a problem.

hinkley 8y ago
I think there's a false dichotomy here. In the post yesterday one of the arguments was that if nobody checks out all of the code then what's the value of having the code all in one place? Last monorepo I worked on, individual contributors checked out just the tree they were working on (we had a suite of applications with several shared modules). We made it simple and straightforward for them to get what they wanted and ignore people whose work didn't impact them. But the senior people, who were better with architecture and version control trivia, checked out the entire thing. They would steward any cross-cutting changes that needed to be done, and make sure any callers to shared libraries were updated in the face of breaking changes. They were also backstopped by the build plans, (some of) which also checked out the entire thing.
- mwkaufma 8y ago
  Streams aren't modules -- they're views. If someone takes you as a dependency and wants you to have visibility on them they add themselves to your stream so you pull down their directory as well.

rhacker 8y ago

It's not for everyone, but damn, why is there a TERABYTE of code? Just curious - assets? checking in binaries?

malkia 8y ago
Test protos. Evaluated configs. Golden data. JAR archives, etc.

alexnewman 8y ago

Signs your build system is never going to be adopted outside of people cargo culting you? - [x] Namespaces and the like without much security benefit - [x] Giant Java dependency - [x] Strange syntax and glyphs