7 ms·

Can you elaborate on “monorepos do not prevent you from checking packages into source control” and how that helps to avoid recompiling everything? Why would you

Can you elaborate on “monorepos do not prevent you from checking packages into source control” and how that helps to avoid recompiling everything? Why would you check a package into source control anyway? Surely source control is for source code? And I lean toward monorepos, btw, but there are still lots of obstacles and monorepo proponents don’t tend to acknowledge them or offer clear suggestions for how to solve or workaround them.

jayd16 7y ago
You can use something like a shared binary repo such as maven or you could just check in dependencies and not worry about an external server being available for builds. >Surely source control is for source code? This is just pedantry. Checking in binaries is a pragmatic solution that solves a lot of problems.
- weberc2 7y ago
  I wasn’t trying to be a pedant, I’ve just never heard of anyone doing this. I was wondering how it helped solve the problem of not rebuilding everything.
  - jayd16 7y ago
    In short, the binaries are already built. Usually its faster to link to a prebuilt binary than to build from scratch.
    - weberc2 7y ago
      So where do these binaries get built and how does the system know which binaries to rebuild for a given change? If developers are building binaries and committing them directly, doesn’t that open up security or even correctness issues? How does this approach satisfy compliance concerns (how can the CTO or a manager sign off on the changes that went into the binary if it’s just something a random developer committed?)? How does this scale to tens of deployments per day? These are hard monorepo problems, and they keep being handwaved away.
      - CuriousSkeptic 7y ago
        Not sure how people do this in practice. But in principle it seems rather straight forward. A compiler is just a program that takes some input and create some output. Both the compiler and the input can have a cryptographically secure hash. Putting both in a sealed box, like a docker image, with its own hash, gives you a program that takes no input and produces some output. If the box changes, run it in a trusted machine and save the output together with a signed declaration of which box version produced it
        fragmede 7y ago
        Docker makes this drastically easier (need the exact same versions of all libraries and the compiler), but there are still compile time things that are unique per-compile. Debian has been working hard to get hashes of binaries to be useful but the work is far from trivial. (See also: trusting trust)
        CuriousSkeptic 7y ago
        I see this an inherited technical debt though not a flaw in principle. It would be nice if we could solve this at the foundational level instead of forcing every dev organization to struggle with it on their own. Edit: I think we’re getting there though, with all the efforts going on with containers, webassembly blockchains, ipfs and so forth it’s getting closer
      - marcinzm 7y ago
        >So where do these binaries get built and how does the system know which binaries to rebuild for a given change? For simple things, if the code in a directory changes then the CI system does a rebuild of that directory. You can have the CI system either validate that the binary matches or commit the binary itself. More complicated things you'll have a build system such as Bazel which figures out what changed.
        weberc2 7y ago
        (Sorry for being terse—on mobile). Validate the binary matches what? If the compiler has to compile the artifact to verify the artifact provided by the developer, why bother having the developer commit the artifact? The CI system could just do it. Never mind that having a bit-for-bit reproducible build is incredibly difficult. Anyway, such simple cases where a whole app lives under a single directory are vanishingly rare.
- yjftsjthsd-h 7y ago
  I was rather under the impression that checking in binaries was discouraged because it led to performance issues and tends to blow up the repository size. I don't think it's just pedantry.
tom_ 7y ago
It's version control, not necessarily just source control! If something could benefit from being versioned, why would you not check it in? You then guarantee everybody has the same version. That's exactly what this thing is there for. Git's design can limit its usefulness in this respect - though perhaps you could solve this to some extent with git LFS? - but not all version control systems have this problem.
- fragmede 7y ago
  git annex (or git LFS, if you buy into github's NIH) is requisite if you want to use git like this, broadly. git will happily store any and all binaries you ask it to, but upon (blind) checkout, it will grab every single revision of said binary, taking up as much however much space that takes. (partial clones avoid this, but, as git isn't designed for this use case, grabbing all of history happens far too easily.)
username90 7y ago
At Google we check in the source of every library into the monorepo and compile them ourselves with cached builds from a central server, I don't think we use package managers.
- jayd16 7y ago
  How do you track dependencies of dependencies. Do you need to manually add the full dependency tree and re implement the dependency tracking through your internal system? If a project uses maven or gradle, you need to rewrite those files to point to your internal builds instead?
  - gen220 7y ago
    Not a Googler, but I think the answer is: yes. At least, it is for my monorepo company. Usually somebody else has already gone through the work of doing it for you. Sometimes there are tools that do the translation for you. For example, Go modules are quite easy to translate to a BUILD file. It’s actually not as bad as it sounds. You only have to do the hard stuff once, and every engineer in the org who uses it in the future is thankful for it.
  - weberc2 7y ago
    They use a tool called Blaze (Google around for “Bazel” which is the open source tool inspired by it). Basically you model the dependency tree such that the tool knows which targets are affected by a certain change, and then Blaze builds them in a clean room environment such that an undeclared dependency would cause the build to fail (hermetic builds). As far as I’m aware, this is the only way to sustainable operate a monorepo, but I would be happy to learn more if someone has other solutions.
  - dodobirdlord 7y ago
    I assume you mean third party dependencies that are not in the monorepo? Pretty much yes, monorepos struggle if they are expected to handle dependencies that aren't stored in the monorepo, so step 1 of using a dependency from outside of a monorepo should be to copy the source into the monorepo (and transitively copy the source of dependencies, etc).

jayd16 7y ago

You can use something like a shared binary repo such as maven or you could just check in dependencies and not worry about an external server being available for builds. >Surely source control is for source code? This is just pedantry. Checking in binaries is a pragmatic solution that solves a lot of problems.

weberc2 7y ago
I wasn’t trying to be a pedant, I’ve just never heard of anyone doing this. I was wondering how it helped solve the problem of not rebuilding everything.
- jayd16 7y ago
  In short, the binaries are already built. Usually its faster to link to a prebuilt binary than to build from scratch.
  - weberc2 7y ago
    So where do these binaries get built and how does the system know which binaries to rebuild for a given change? If developers are building binaries and committing them directly, doesn’t that open up security or even correctness issues? How does this approach satisfy compliance concerns (how can the CTO or a manager sign off on the changes that went into the binary if it’s just something a random developer committed?)? How does this scale to tens of deployments per day? These are hard monorepo problems, and they keep being handwaved away.
    - CuriousSkeptic 7y ago
      Not sure how people do this in practice. But in principle it seems rather straight forward. A compiler is just a program that takes some input and create some output. Both the compiler and the input can have a cryptographically secure hash. Putting both in a sealed box, like a docker image, with its own hash, gives you a program that takes no input and produces some output. If the box changes, run it in a trusted machine and save the output together with a signed declaration of which box version produced it
      - fragmede 7y ago
        Docker makes this drastically easier (need the exact same versions of all libraries and the compiler), but there are still compile time things that are unique per-compile. Debian has been working hard to get hashes of binaries to be useful but the work is far from trivial. (See also: trusting trust)
        CuriousSkeptic 7y ago
        I see this an inherited technical debt though not a flaw in principle. It would be nice if we could solve this at the foundational level instead of forcing every dev organization to struggle with it on their own. Edit: I think we’re getting there though, with all the efforts going on with containers, webassembly blockchains, ipfs and so forth it’s getting closer
    - marcinzm 7y ago
      >So where do these binaries get built and how does the system know which binaries to rebuild for a given change? For simple things, if the code in a directory changes then the CI system does a rebuild of that directory. You can have the CI system either validate that the binary matches or commit the binary itself. More complicated things you'll have a build system such as Bazel which figures out what changed.
      - weberc2 7y ago
        (Sorry for being terse—on mobile). Validate the binary matches what? If the compiler has to compile the artifact to verify the artifact provided by the developer, why bother having the developer commit the artifact? The CI system could just do it. Never mind that having a bit-for-bit reproducible build is incredibly difficult. Anyway, such simple cases where a whole app lives under a single directory are vanishingly rare.
yjftsjthsd-h 7y ago
I was rather under the impression that checking in binaries was discouraged because it led to performance issues and tends to blow up the repository size. I don't think it's just pedantry.

tom_ 7y ago

It's version control, not necessarily just source control! If something could benefit from being versioned, why would you not check it in? You then guarantee everybody has the same version. That's exactly what this thing is there for. Git's design can limit its usefulness in this respect - though perhaps you could solve this to some extent with git LFS? - but not all version control systems have this problem.

fragmede 7y ago
git annex (or git LFS, if you buy into github's NIH) is requisite if you want to use git like this, broadly. git will happily store any and all binaries you ask it to, but upon (blind) checkout, it will grab every single revision of said binary, taking up as much however much space that takes. (partial clones avoid this, but, as git isn't designed for this use case, grabbing all of history happens far too easily.)

username90 7y ago

At Google we check in the source of every library into the monorepo and compile them ourselves with cached builds from a central server, I don't think we use package managers.

jayd16 7y ago
How do you track dependencies of dependencies. Do you need to manually add the full dependency tree and re implement the dependency tracking through your internal system? If a project uses maven or gradle, you need to rewrite those files to point to your internal builds instead?
- gen220 7y ago
  Not a Googler, but I think the answer is: yes. At least, it is for my monorepo company. Usually somebody else has already gone through the work of doing it for you. Sometimes there are tools that do the translation for you. For example, Go modules are quite easy to translate to a BUILD file. It’s actually not as bad as it sounds. You only have to do the hard stuff once, and every engineer in the org who uses it in the future is thankful for it.
- weberc2 7y ago
  They use a tool called Blaze (Google around for “Bazel” which is the open source tool inspired by it). Basically you model the dependency tree such that the tool knows which targets are affected by a certain change, and then Blaze builds them in a clean room environment such that an undeclared dependency would cause the build to fail (hermetic builds). As far as I’m aware, this is the only way to sustainable operate a monorepo, but I would be happy to learn more if someone has other solutions.
- dodobirdlord 7y ago
  I assume you mean third party dependencies that are not in the monorepo? Pretty much yes, monorepos struggle if they are expected to handle dependencies that aren't stored in the monorepo, so step 1 of using a dependency from outside of a monorepo should be to copy the source into the monorepo (and transitively copy the source of dependencies, etc).