12 ms·

Make your monorepo feel small with Git’s sparse index

221 points by CRConrad 5y ago

speedgoose 5y ago
"One of the biggest traps for smart engineers is optimizing something that shouldn't exist." Elon Musk.
- solarmist 5y ago
  Let's make the snarky comment into a helpful comment. Why do you think it shouldn't exist?
  - speedgoose 5y ago
    Monorepos create more issues than what they solve.
    - tsimionescu 5y ago
      So the Linux devs have no idea how to properly use Git?
      - speedgoose 5y ago
        I'm not sure whether the Linux kernel git repository qualifies as a Monorepo.
        tsimionescu 5y ago
        Of course it does. It includes core kernel code, drivers, countless libraries (compression libraries, fonts, a unit testing framework, various data structure implemetations, and many many others), file systems, network stacks, a programming language runtime+compiler (ebpf), and many other subsystems. According to multi-repo advocates, it would obviously be split up into at least a dozen different repos, with dependency management tools to specify which version of zlib or ebpf to use, with a dedicated repo just for the build logic, of course.
    - solarmist 5y ago
      Such as? That's just parroting “common opinion” otherwise.
      - speedgoose 5y ago
        The second paragraph of the article we are discussing about for example. But you can find a list on Wikipedia and make your own opinion : https://en.m.wikipedia.org/wiki/Monorepo https://en.m.wikipedia.org/wiki/Monorepo
    - tambourine_man 5y ago
      Supporting evidence?
      - speedgoose 5y ago
        The Wikipedia article about monorepos has a good summary https://en.m.wikipedia.org/wiki/Monorepo https://en.m.wikipedia.org/wiki/Monorepo Then you can do your own opinion, I'm sharing mine.
        jeremyjh 5y ago
        Apart from performance issues that article offers more (and more significant) advantages than it does drawbacks, so it really does not support your statement.
    - ratww 5y ago
      That's completely false. Monorepos don't really create issues at all when done properly, and when used in situations where they make sense. In smaller scales, for example, they're fantastic for productivity, and my company is not looking back.
    - jeremyjh 5y ago
      This is no different from saying "monorepo bad". Aside from performance issues in git, why would a monorepo be bad? It seems very natural to me to have a whole system referenced with a single branch/tag that must all pass CI together. Otherwise supporting projects can introduce breaking changes downstream that are not apparent before they hit master.
    - anaganisk 5y ago
      Somehow its working for google? May be im wrong but last I heard they use mono repos.
- joconde 5y ago
  Was he talking about something specific?
  - plopz 5y ago
    I remember him saying something like that during a walkthrough of the base building starship and if I recall it was in reference to overengineering something about the grid fins.
  - junon 5y ago
    He was talking about the battery vibrator plates or something in Tesla cars.
- CRConrad 5y ago
  Wow, so he's actually said something smart for a change.
- anaganisk 5y ago
  And then he comes up with Hyperloop.
arxanas 5y ago
The index as a data structure is really starting to show its age, especially as developers adapt Git to monorepo scale. It's really fast for repositories up to a certain size, but big tech organizations grow exponentially, and start to suffer performance issues. At some point, you can't afford to use a data structure that scales with the size of the repo, and have to switch to one that scales with the size of the user's change. I spent a good chunk of time working around the lack of sparse indexes in libgit2, which produced speedups on the order of 500x for certain operations, because reading and writing the entire index is unnecessary for most users of a monorepo: https://github.com/libgit2/libgit2/issues/6036 https://github.com/libgit2/libgit2/issues/6036. I'm excited to see sparse indexes make their way into Git proper. Shameless plug: I'm working on improving monorepo-scale Git tooling at https://github.com/arxanas/git-branchless https://github.com/arxanas/git-branchless, such as with in-memory rebases: https://blog.waleedkhan.name/in-memory-rebases/ https://blog.waleedkhan.name/in-memory-rebases/. Try it out if you work in a Git monorepo.
- Buttersite 5y ago
  Maybe this is a sign that monorepos are the problem?
  - arxanas 5y ago
    Why would this indicate that monorepos are the problem?
    - chongli 5y ago
      It’s a philosophical conflict. Git was designed for distributed, decentralized version control and collaboration. Monorepos are inherently centralized. Thus git is the wrong tool for the job and all of this effort to adapt git to monorepo workflows runs counter to its philosophy.
      - status_quo69 5y ago
        I dunno, the linux kernel is pretty big and centralized by that definition, probably a monorepo by some definitions. I don't think this is a square peg round hole problem, I think that the operations that git must do coupled with the inevitable massive files that every company shoves into their repos (assets, binaries, you name it) makes git chug quite a bit.
anon9001 5y ago
This is well written and deserves my upvote, because sparse-checkout is part of git and knowing how it works is useful. That said, there's absolutely no reason to structure your code in a monorepo. Here's what I think GitHub is doing: 1) Encourage monorepo adoption 2) Build tooling for monorepos 3) Selling tooling to developers stranded in monorepos Microsoft, which owns GitHub, created the microsoft/git fork linked in the article, and they explain their justification here: https://github.com/microsoft/git#why-is-this-fork-needed https://github.com/microsoft/git#why-is-this-fork-needed > Well, because Git is a distributed version control system, each Git repository has a copy of all files in the entire history. As large repositories, aka monorepos grow, Git can struggle to manage all that data. As Git commands like status and fetch get slower, developers stop waiting and start switching context. And context switches harm developer productivity. I believe that Google's brand is so big that it led to this mass cognitive dissonance, which is being exploited by GitHub. To be clear, here are the two ideas in conflict: * Git is decentralized and fast, and Google famously doesn't use it. * Companies want to use "industry standard" tech, and Google is the standard for success. Now apply those observations to a world where your engineers only use "git". The result is market demand to misuse git for monorepos, which Microsoft is pouring huge amounts of resources into enabling via GitHub. It makes great sense that GitHub wants to lean into this. More centralization and being more reliant on GitHub's custom tooling is obviously better for GitHub. It just so happens that GitHub is building tools to enable monorepos, essentially normalizing their usage. Then GitHub can sell tools to deal with your enormous monorepo, because your traditional tools will feel slow and worse than GitHub's tools. In other words, GitHub is propping up the failed monorepo idea as a strategy to get people in the pipeline for things like CodeSpaces: https://github.com/features/codespaces https://github.com/features/codespaces Because if you have 100 projects and they're all separate, you can do development locally for each and it's fast and sensible. But if all your projects are in one repo, the tools grind to a halt, and suddenly you need to buy a solution that just works to meet your business goals.
- ratww 5y ago
  > That said, there's absolutely no reason to structure your code in a monorepo. Bullshit. There are very good reasons to use it in some situations. My company is using it and it's a tremendous productivity boon. And Git works perfectly fine for smaller scales. Obviously, "because Google does it" is a terrible reason. But it's disingenuous to say that's the only reason people are doing it. Not everyone is a moron.
nightpool 5y ago
Is there a point in having a monorepo if you're all in on the microservices approach? I'm a big microservices skeptic, but as far as I understand it, the benefit of microservices is independence of change & deployment enforced by solid API contracts—don't you give that all up when you use a monorepo? What does "Monorepo with microservices" give you that a normal monolithic backend doesn't? (Obviously e.g. an image resizer or something else completely decoupled from your business logic should be a separate service / repo anyway—my point is more along the lines of "If something shares code, shouldn't it share a deployment strategy?")
- radlad 5y ago
  Monorepo with microservices gives you the ability to scale and perform SRE-type maintenance at a granular level. Teams maintain responsibility for their service, but are more easily able to refactor, share code, pull dependencies like GraphQL schemas into the frontend, etc. across many services.
  - nightpool 5y ago
    So basically each team has to reinvent devops from the ground up, and staff their own on call rotation, instead of having a centralized devops function that provides a stable platform? That sounds horrendous. Although, that said, I can at least see the benefits of the "1 service per team" methodology, where you have a dedicated team that's independently responsible for updating their service. I'm more used to associating "microservices" with the model where a single team is managing 5 or 6 interacting services, and the benefits there seem much smaller.
    - radlad 5y ago
      > That sounds horrendous. Different teams can make their own decisions, but as a developer on a team that ran our own SRE, I found it came with many advantages. Specifically, we saw very little downtime, and when outages did occur were very prepared to fix it as we knew the exact state of our services (code, infrastructure, recent changes and deploys.) Additionally, we had very good logging and metrics because we knew what we'd want to have in the event of a problem. And I'm not sure what you mean "from the ground up." We were able to share a lot of Ansible playbooks, frameworks, and our entire observability stack across all teams. But I think you may also be missing the rest of my post. This is only one possible advantage. Even if the team doesn't perform their own SRE, these services can be scaled independently - both in terms of infrastructure and codebase - even while sharing code (including things like protocol data structures, auth schemes, etc.) A service that receives 1 SAML Response from an IdP (identity provider) per day per user may not need the same resources as a dashboard that exposes all SP (service providers) to a user many times a day. And an administration panel for this has still different needs. Yet, all of these services may communicate with each other.
aranchelk 5y ago
The more I've come to rely on techniques like canary testing and opt-in version upgrades the more firmly I believe one of the main motivations for monorepos is flawed: at any given time there may not be a fact of the matter as to which single version of an app or service is running in an environment. At places I’ve worked when we thought about canary testing we ignored the fact that there were multiple versions of the software running in parallel, we classified it as part of an orchestration process and not a reality about the code or env, but we really did have multiple versions of a service running at once, sometimes for days. Similarly if you’ve got a setup where you can upgrade users/regions/etc piecemeal (opt-in or by other selection criteria) I don’t know how you reflect this in a monorepo. I’m curious how Google actually does this as I recall they have offered user opt-in upgrades for Gmail. My suspicion is this gets solved with something like directories ./v2/ and ./v3/ — but that’s far from ideal.
- ASinclair 5y ago
  > I’m curious how Google actually does this Branches are cut for releases. Binaries are versioned.
- coryrc 5y ago
  Only binaries are released. Binaries are timestamped and linked to a changelist. The "opt-in upgrades" are all live code. I know more than a few "foo" "foo2" directories, but I wouldn't want an actively-delivered, long-running service to be living in a feature branch so I would still expect anyone to be using a similar naming scheme.
- deleted 5y ago
  [deleted]
- eximius 5y ago
  that doesnt seem like a problem with monorepos (or otherwise). youd just need to tag your releases, right?
  - aranchelk 5y ago
    I don't think it's a problem, rather I'm challenging a touted benefit. In large monorepos the supposition is you've got a class of compatible apps and services bundled together. Version dependencies are somewhat implicit: the correct version for each project to interoperate is whatever was checked-in together in your commit. I don't know how it works in practice at different orgs, but there's certainly an idea I've heard repeated that you can essentially test, build, and deploy your monorepo atomically, but the reality in my experience is you can't escape the need to think about compatibility across multiple versions of services once you use techniques like canary testing or targeted upgrades.
junon 5y ago
What I'd really like to see is Git have the ability to consolidate repeat submodules down into a single set of objects in the super repository. Currently cloning the same submodule results in a copy of the repository for each path, which is absurd. It's been something on my list to address on the mailing lists for a while, just haven't had time.
harvie 5y ago
I hope one day all of this will be as easy as in SVN. eg.: I have repository https://example.com/myrepo https://example.com/myrepo And i can simply do: svn co https://example.com/myrepo/some/directory/ https://example.com/myrepo/some/directory/ And i can work with that subdirectory as if it was actual repo. Completely transparently. This i really miss in git.
- Gigachad 5y ago
  I'm working in hell right now. The current company has the site frontend, backend, and tests in separate repos and it's basically impossible to do anything without force merging because the build is broken without a chicken and egg situation between the 3 pull requests.
  - laurent123456 5y ago
    I worked at a company that not only did that, but also decided to split the main web app into multiple repos, one per country. It was so much fun to do anything in this project.
    - xorcist 5y ago
      Now, that's a microservice if there ever was one!
    - Iwan-Zotow 5y ago
      that is some good office porno!
  - hinkley 5y ago
    You might want the front end and backend to be compatible across revision numbers, so that’s less of a problem and more of an impedance mismatch with your process. The tests are a trickier problem, in theory you could lock version numbers (the new behavior increments to the new tests) but tests have bugs, and this doesn’t solve two people making behavioral changes at the same time. Short of merging the tests into the respective projects, going full feature toggle for all breaking changes is the only other lever that comes to mind.
rq1 5y ago
What I was looking for recently is a way to make “sparse push”. And trigger a chain reaction with hooks. Didn’t find anything interesting.
rurban 5y ago
Even better looks the new OTR merge strategy, which benefits everyone. Not only the tiny monorepo userbase.
haberman 5y ago
> The index file stores a list of every file at HEAD, along with the object ID for its blob and some metadata. This list of files is stored as a flat list and Git parses the index into an array. I'm surprised that the index is not hierarchical, like tree objects in Git's object storage. With tree objects (https://git-scm.com/book/en/v2/Git-Internals-Git-Objects#_tree_objects https://git-scm.com/book/en/v2/Git-Internals-Git-Objects#_tr...), each level of the hierarchy is a separate object. So you would only need to load directories that are interesting to you. You could use a single hash compare to determine that two directories are identical without actually recursing into them. In particular, I can't understand why you would need a full list of all files to create a commit. If your commit is known not to touch certain directories, it should be able to simply refer to the existing tree object without loading or expanding it. I guess that's what this sparse-index work is doing. I'm just surprised it didn't already work that way.
- arxanas 5y ago
  It makes more sense if you think of the index as a structure meant specifically to speed up `git status` operations. (It was originally called the "dircache"! See https://github.com/git/git/commit/5adf317b31729707fad4967c1aef6cdba43d0dd3 https://github.com/git/git/commit/5adf317b31729707fad4967c1a...) We desperately want to reduce the number of file accesses we have to make, so directly using the object database and a tree object (or similar structures) would more than double file accesses. There's performance-related metadata in the index which isn't in tree objects. For example, the modified-time of a given file exists in its index entry, which can be used to avoid reading the file from disk if it seems to be unmodified. If you have to do a disk lookup to decide whether to read a file from disk, then the overhead is potentially as much as the operation itself. There's also semantic metadata, such as which stage the file is in (for merge conflict resolution). It's worth noting that you can turn on the cache tree extension (https://git-scm.com/docs/index-format#_cache_tree https://git-scm.com/docs/index-format#_cache_tree) in order to speed up commit operations. It doesn't replace objects in the index with trees, but it does keep ranges of the index cached, if they're known to correspond to a tree.

221 points by CRConrad 5y ago

speedgoose 5y ago

"One of the biggest traps for smart engineers is optimizing something that shouldn't exist." Elon Musk.

solarmist 5y ago
Let's make the snarky comment into a helpful comment. Why do you think it shouldn't exist?
- speedgoose 5y ago
  Monorepos create more issues than what they solve.
  - tsimionescu 5y ago
    So the Linux devs have no idea how to properly use Git?
    - speedgoose 5y ago
      I'm not sure whether the Linux kernel git repository qualifies as a Monorepo.
      - tsimionescu 5y ago
        Of course it does. It includes core kernel code, drivers, countless libraries (compression libraries, fonts, a unit testing framework, various data structure implemetations, and many many others), file systems, network stacks, a programming language runtime+compiler (ebpf), and many other subsystems. According to multi-repo advocates, it would obviously be split up into at least a dozen different repos, with dependency management tools to specify which version of zlib or ebpf to use, with a dedicated repo just for the build logic, of course.
  - solarmist 5y ago
    Such as? That's just parroting “common opinion” otherwise.
    - speedgoose 5y ago
      The second paragraph of the article we are discussing about for example. But you can find a list on Wikipedia and make your own opinion : https://en.m.wikipedia.org/wiki/Monorepo https://en.m.wikipedia.org/wiki/Monorepo
  - tambourine_man 5y ago
    Supporting evidence?
    - speedgoose 5y ago
      The Wikipedia article about monorepos has a good summary https://en.m.wikipedia.org/wiki/Monorepo https://en.m.wikipedia.org/wiki/Monorepo Then you can do your own opinion, I'm sharing mine.
      - jeremyjh 5y ago
        Apart from performance issues that article offers more (and more significant) advantages than it does drawbacks, so it really does not support your statement.
  - ratww 5y ago
    That's completely false. Monorepos don't really create issues at all when done properly, and when used in situations where they make sense. In smaller scales, for example, they're fantastic for productivity, and my company is not looking back.
  - jeremyjh 5y ago
    This is no different from saying "monorepo bad". Aside from performance issues in git, why would a monorepo be bad? It seems very natural to me to have a whole system referenced with a single branch/tag that must all pass CI together. Otherwise supporting projects can introduce breaking changes downstream that are not apparent before they hit master.
  - anaganisk 5y ago
    Somehow its working for google? May be im wrong but last I heard they use mono repos.
joconde 5y ago
Was he talking about something specific?
- plopz 5y ago
  I remember him saying something like that during a walkthrough of the base building starship and if I recall it was in reference to overengineering something about the grid fins.
- junon 5y ago
  He was talking about the battery vibrator plates or something in Tesla cars.
CRConrad 5y ago
Wow, so he's actually said something smart for a change.
anaganisk 5y ago
And then he comes up with Hyperloop.

arxanas 5y ago

The index as a data structure is really starting to show its age, especially as developers adapt Git to monorepo scale. It's really fast for repositories up to a certain size, but big tech organizations grow exponentially, and start to suffer performance issues. At some point, you can't afford to use a data structure that scales with the size of the repo, and have to switch to one that scales with the size of the user's change. I spent a good chunk of time working around the lack of sparse indexes in libgit2, which produced speedups on the order of 500x for certain operations, because reading and writing the entire index is unnecessary for most users of a monorepo: https://github.com/libgit2/libgit2/issues/6036 https://github.com/libgit2/libgit2/issues/6036. I'm excited to see sparse indexes make their way into Git proper. Shameless plug: I'm working on improving monorepo-scale Git tooling at https://github.com/arxanas/git-branchless https://github.com/arxanas/git-branchless, such as with in-memory rebases: https://blog.waleedkhan.name/in-memory-rebases/ https://blog.waleedkhan.name/in-memory-rebases/. Try it out if you work in a Git monorepo.

Buttersite 5y ago
Maybe this is a sign that monorepos are the problem?
- arxanas 5y ago
  Why would this indicate that monorepos are the problem?
  - chongli 5y ago
    It’s a philosophical conflict. Git was designed for distributed, decentralized version control and collaboration. Monorepos are inherently centralized. Thus git is the wrong tool for the job and all of this effort to adapt git to monorepo workflows runs counter to its philosophy.
    - status_quo69 5y ago
      I dunno, the linux kernel is pretty big and centralized by that definition, probably a monorepo by some definitions. I don't think this is a square peg round hole problem, I think that the operations that git must do coupled with the inevitable massive files that every company shoves into their repos (assets, binaries, you name it) makes git chug quite a bit.

anon9001 5y ago

This is well written and deserves my upvote, because sparse-checkout is part of git and knowing how it works is useful. That said, there's absolutely no reason to structure your code in a monorepo. Here's what I think GitHub is doing: 1) Encourage monorepo adoption 2) Build tooling for monorepos 3) Selling tooling to developers stranded in monorepos Microsoft, which owns GitHub, created the microsoft/git fork linked in the article, and they explain their justification here: https://github.com/microsoft/git#why-is-this-fork-needed https://github.com/microsoft/git#why-is-this-fork-needed > Well, because Git is a distributed version control system, each Git repository has a copy of all files in the entire history. As large repositories, aka monorepos grow, Git can struggle to manage all that data. As Git commands like status and fetch get slower, developers stop waiting and start switching context. And context switches harm developer productivity. I believe that Google's brand is so big that it led to this mass cognitive dissonance, which is being exploited by GitHub. To be clear, here are the two ideas in conflict: * Git is decentralized and fast, and Google famously doesn't use it. * Companies want to use "industry standard" tech, and Google is the standard for success. Now apply those observations to a world where your engineers only use "git". The result is market demand to misuse git for monorepos, which Microsoft is pouring huge amounts of resources into enabling via GitHub. It makes great sense that GitHub wants to lean into this. More centralization and being more reliant on GitHub's custom tooling is obviously better for GitHub. It just so happens that GitHub is building tools to enable monorepos, essentially normalizing their usage. Then GitHub can sell tools to deal with your enormous monorepo, because your traditional tools will feel slow and worse than GitHub's tools. In other words, GitHub is propping up the failed monorepo idea as a strategy to get people in the pipeline for things like CodeSpaces: https://github.com/features/codespaces https://github.com/features/codespaces Because if you have 100 projects and they're all separate, you can do development locally for each and it's fast and sensible. But if all your projects are in one repo, the tools grind to a halt, and suddenly you need to buy a solution that just works to meet your business goals.

ratww 5y ago
> That said, there's absolutely no reason to structure your code in a monorepo. Bullshit. There are very good reasons to use it in some situations. My company is using it and it's a tremendous productivity boon. And Git works perfectly fine for smaller scales. Obviously, "because Google does it" is a terrible reason. But it's disingenuous to say that's the only reason people are doing it. Not everyone is a moron.

nightpool 5y ago

Is there a point in having a monorepo if you're all in on the microservices approach? I'm a big microservices skeptic, but as far as I understand it, the benefit of microservices is independence of change & deployment enforced by solid API contracts—don't you give that all up when you use a monorepo? What does "Monorepo with microservices" give you that a normal monolithic backend doesn't? (Obviously e.g. an image resizer or something else completely decoupled from your business logic should be a separate service / repo anyway—my point is more along the lines of "If something shares code, shouldn't it share a deployment strategy?")

radlad 5y ago
Monorepo with microservices gives you the ability to scale and perform SRE-type maintenance at a granular level. Teams maintain responsibility for their service, but are more easily able to refactor, share code, pull dependencies like GraphQL schemas into the frontend, etc. across many services.
- nightpool 5y ago
  So basically each team has to reinvent devops from the ground up, and staff their own on call rotation, instead of having a centralized devops function that provides a stable platform? That sounds horrendous. Although, that said, I can at least see the benefits of the "1 service per team" methodology, where you have a dedicated team that's independently responsible for updating their service. I'm more used to associating "microservices" with the model where a single team is managing 5 or 6 interacting services, and the benefits there seem much smaller.
  - radlad 5y ago
    > That sounds horrendous. Different teams can make their own decisions, but as a developer on a team that ran our own SRE, I found it came with many advantages. Specifically, we saw very little downtime, and when outages did occur were very prepared to fix it as we knew the exact state of our services (code, infrastructure, recent changes and deploys.) Additionally, we had very good logging and metrics because we knew what we'd want to have in the event of a problem. And I'm not sure what you mean "from the ground up." We were able to share a lot of Ansible playbooks, frameworks, and our entire observability stack across all teams. But I think you may also be missing the rest of my post. This is only one possible advantage. Even if the team doesn't perform their own SRE, these services can be scaled independently - both in terms of infrastructure and codebase - even while sharing code (including things like protocol data structures, auth schemes, etc.) A service that receives 1 SAML Response from an IdP (identity provider) per day per user may not need the same resources as a dashboard that exposes all SP (service providers) to a user many times a day. And an administration panel for this has still different needs. Yet, all of these services may communicate with each other.

aranchelk 5y ago

The more I've come to rely on techniques like canary testing and opt-in version upgrades the more firmly I believe one of the main motivations for monorepos is flawed: at any given time there may not be a fact of the matter as to which single version of an app or service is running in an environment. At places I’ve worked when we thought about canary testing we ignored the fact that there were multiple versions of the software running in parallel, we classified it as part of an orchestration process and not a reality about the code or env, but we really did have multiple versions of a service running at once, sometimes for days. Similarly if you’ve got a setup where you can upgrade users/regions/etc piecemeal (opt-in or by other selection criteria) I don’t know how you reflect this in a monorepo. I’m curious how Google actually does this as I recall they have offered user opt-in upgrades for Gmail. My suspicion is this gets solved with something like directories ./v2/ and ./v3/ — but that’s far from ideal.

ASinclair 5y ago
> I’m curious how Google actually does this Branches are cut for releases. Binaries are versioned.
coryrc 5y ago
Only binaries are released. Binaries are timestamped and linked to a changelist. The "opt-in upgrades" are all live code. I know more than a few "foo" "foo2" directories, but I wouldn't want an actively-delivered, long-running service to be living in a feature branch so I would still expect anyone to be using a similar naming scheme.
deleted 5y ago
[deleted]
eximius 5y ago
that doesnt seem like a problem with monorepos (or otherwise). youd just need to tag your releases, right?
- aranchelk 5y ago
  I don't think it's a problem, rather I'm challenging a touted benefit. In large monorepos the supposition is you've got a class of compatible apps and services bundled together. Version dependencies are somewhat implicit: the correct version for each project to interoperate is whatever was checked-in together in your commit. I don't know how it works in practice at different orgs, but there's certainly an idea I've heard repeated that you can essentially test, build, and deploy your monorepo atomically, but the reality in my experience is you can't escape the need to think about compatibility across multiple versions of services once you use techniques like canary testing or targeted upgrades.

junon 5y ago

What I'd really like to see is Git have the ability to consolidate repeat submodules down into a single set of objects in the super repository. Currently cloning the same submodule results in a copy of the repository for each path, which is absurd. It's been something on my list to address on the mailing lists for a while, just haven't had time.

harvie 5y ago

I hope one day all of this will be as easy as in SVN. eg.: I have repository https://example.com/myrepo https://example.com/myrepo And i can simply do: svn co https://example.com/myrepo/some/directory/ https://example.com/myrepo/some/directory/ And i can work with that subdirectory as if it was actual repo. Completely transparently. This i really miss in git.

Gigachad 5y ago
I'm working in hell right now. The current company has the site frontend, backend, and tests in separate repos and it's basically impossible to do anything without force merging because the build is broken without a chicken and egg situation between the 3 pull requests.
- laurent123456 5y ago
  I worked at a company that not only did that, but also decided to split the main web app into multiple repos, one per country. It was so much fun to do anything in this project.
  - xorcist 5y ago
    Now, that's a microservice if there ever was one!
  - Iwan-Zotow 5y ago
    that is some good office porno!
- hinkley 5y ago
  You might want the front end and backend to be compatible across revision numbers, so that’s less of a problem and more of an impedance mismatch with your process. The tests are a trickier problem, in theory you could lock version numbers (the new behavior increments to the new tests) but tests have bugs, and this doesn’t solve two people making behavioral changes at the same time. Short of merging the tests into the respective projects, going full feature toggle for all breaking changes is the only other lever that comes to mind.

rq1 5y ago

What I was looking for recently is a way to make “sparse push”. And trigger a chain reaction with hooks. Didn’t find anything interesting.

rurban 5y ago

Even better looks the new OTR merge strategy, which benefits everyone. Not only the tiny monorepo userbase.

haberman 5y ago

> The index file stores a list of every file at HEAD, along with the object ID for its blob and some metadata. This list of files is stored as a flat list and Git parses the index into an array. I'm surprised that the index is not hierarchical, like tree objects in Git's object storage. With tree objects (https://git-scm.com/book/en/v2/Git-Internals-Git-Objects#_tree_objects https://git-scm.com/book/en/v2/Git-Internals-Git-Objects#_tr...), each level of the hierarchy is a separate object. So you would only need to load directories that are interesting to you. You could use a single hash compare to determine that two directories are identical without actually recursing into them. In particular, I can't understand why you would need a full list of all files to create a commit. If your commit is known not to touch certain directories, it should be able to simply refer to the existing tree object without loading or expanding it. I guess that's what this sparse-index work is doing. I'm just surprised it didn't already work that way.

arxanas 5y ago
It makes more sense if you think of the index as a structure meant specifically to speed up `git status` operations. (It was originally called the "dircache"! See https://github.com/git/git/commit/5adf317b31729707fad4967c1aef6cdba43d0dd3 https://github.com/git/git/commit/5adf317b31729707fad4967c1a...) We desperately want to reduce the number of file accesses we have to make, so directly using the object database and a tree object (or similar structures) would more than double file accesses. There's performance-related metadata in the index which isn't in tree objects. For example, the modified-time of a given file exists in its index entry, which can be used to avoid reading the file from disk if it seems to be unmodified. If you have to do a disk lookup to decide whether to read a file from disk, then the overhead is potentially as much as the operation itself. There's also semantic metadata, such as which stage the file is in (for merge conflict resolution). It's worth noting that you can turn on the cache tree extension (https://git-scm.com/docs/index-format#_cache_tree https://git-scm.com/docs/index-format#_cache_tree) in order to speed up commit operations. It doesn't replace objects in the index with trees, but it does keep ranges of the index cached, if they're known to correspond to a tree.