3 ms·

Most people should `git clone --depth 1` most of the time. Large businesses should do trunk-based development with something other than git that scales using di

Most people should `git clone --depth 1` most of the time. Large businesses should do trunk-based development with something other than git that scales using distributed file systems with phantom checkouts, negating the need for LFS blob stores.

vildravn 4y ago
> with something other than git that scales using distributed file systems with phantom checkouts, negating the need for LFS blob stores. What would that something be for example?
- 1letterunixname 4y ago
  NDA, so I can't talk about it. I would like to but I can't.
  - saagarjha 4y ago
    Not that you should do anything you're not comfortable with but I will note that the source control systems for most major companies are essentially known, especially around here. "At Facebook we use a Mercurial-inspired thing with some local extensions" is not really earth-shattering.
tex0 4y ago
That reminds me of Piper/CitC: https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...
iudqnolq 4y ago
I haven't been able to find a link recently, but I recall from a few years ago that that's much harder on GitHub's servers. One of the newer package managers that uses GitHub for hosting (brew maybe?) tried to be helpful and default all their users to shallow clones. GitHub people wanted them to push an update ASAP to go back to full clones, and there was an issue thread laying it all out. If I recall correctly it has to do with the way they cache git. If you request shallow clones they have to run a git process each time to make a custom download with only the data you need. Full clones get served from a cache.
- masklinn 4y ago
  It was an issue with Cocoapods: https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomment-193772935 https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomm... > If I recall correctly it has to do with the way they cache git. It was not actually the shallow clone, the shallow clone is fine. The problem was shallow fetches afterwards, as it made computing the minimum set of changes to fetch much harder for git (during the fetch operation the client and server actually try to negotiate a minimum set of changes, and the server creates an ad-hoc pack for this). Not only that, but they'd hit a git edge case which ended up causing disproportionate CPU usage and converting the shallow clones to near-full clones but very, very inefficiently. This issue was compounded by the very git-unfriendly layout of the repository: one of the directories had >16000 subdirectories, something Git's tree-processing code apparently was not much tested for, leading to significant inefficiencies.

vildravn 4y ago

> with something other than git that scales using distributed file systems with phantom checkouts, negating the need for LFS blob stores. What would that something be for example?

1letterunixname 4y ago
NDA, so I can't talk about it. I would like to but I can't.
- saagarjha 4y ago
  Not that you should do anything you're not comfortable with but I will note that the source control systems for most major companies are essentially known, especially around here. "At Facebook we use a Mercurial-inspired thing with some local extensions" is not really earth-shattering.

tex0 4y ago

That reminds me of Piper/CitC: https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...

iudqnolq 4y ago

I haven't been able to find a link recently, but I recall from a few years ago that that's much harder on GitHub's servers. One of the newer package managers that uses GitHub for hosting (brew maybe?) tried to be helpful and default all their users to shallow clones. GitHub people wanted them to push an update ASAP to go back to full clones, and there was an issue thread laying it all out. If I recall correctly it has to do with the way they cache git. If you request shallow clones they have to run a git process each time to make a custom download with only the data you need. Full clones get served from a cache.

masklinn 4y ago
It was an issue with Cocoapods: https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomment-193772935 https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomm... > If I recall correctly it has to do with the way they cache git. It was not actually the shallow clone, the shallow clone is fine. The problem was shallow fetches afterwards, as it made computing the minimum set of changes to fetch much harder for git (during the fetch operation the client and server actually try to negotiate a minimum set of changes, and the server creates an ad-hoc pack for this). Not only that, but they'd hit a git edge case which ended up causing disproportionate CPU usage and converting the shallow clones to near-full clones but very, very inefficiently. This issue was compounded by the very git-unfriendly layout of the repository: one of the directories had >16000 subdirectories, something Git's tree-processing code apparently was not much tested for, leading to significant inefficiencies.