5 ms·
Scaling Git’s garbage collection
- sc68cal 4y agoI wish they had not gone with uint32_t for storing mtimes, since they now have to deal with the 2038 problem, sometime in the future. I am surprised they didn't directly use time_t, so that they wouldn't have to deal with this (since some platforms have already gone to 64 bit time_t)
- kevingadd 4y agoWouldn't that mean if a platform changed time_t formats it would invalidate all their stored files?
- deleted 4y ago[deleted]
- cesarb 4y ago> I wish they had not gone with uint32_t for storing mtimes, since they now have to deal with the 2038 problem, sometime in the future. Since uint32_t is unsigned, wouldn't it be the Y2106 problem instead? > I am surprised they didn't directly use time_t, so that they wouldn't have to deal with this (since some platforms have already gone to 64 bit time_t) You mentioned the problem yourself without noticing: some platforms have gone to 64-bit time_t, but others haven't. This is a file format, which can be shared by multiple platforms, so it cannot use types which change size depending on the platform.
- 0x0 4y agoBut you could always a fit a uint32_t mtime in a uint64_t file format field...? Wouldn't it be better to future-proof the file format?
- cesarb 4y agoThat would double the disk space use of that particular file, for something which won't be necessary for more than 80 years (the mtime of a git object or pack file will always be in the past, unless the computer clock is wrong); and when that time arrives, a new file format can be defined (and the old format kept as read-only, with new data being written to the new file).
- grogers 4y agoWell if they use unsigned 32 bit they at least extended it to Y2106 :-) But for this use case it's not really an issue though. FTA it sounded like they always write the mtime as now. It's unlikely they wouldn't GC the repo in 68 years to make wraparound an issue.
- est31 4y agoFor on-disk formats, time_t would probably not be a good choice, but indeed, they have a time_t to uint32_t conversion going on, that is not even saturating, just cutting bits off: https://github.com/git/git/blob/e188ec3a735ae52a0d0d3c22f9df6b29fa613b1e/builtin/pack-objects.c#L3497-L3503 https://github.com/git/git/blob/e188ec3a735ae52a0d0d3c22f9df... https://github.com/git/git/blob/e188ec3a735ae52a0d0d3c22f9df6b29fa613b1e/builtin/pack-objects.c#L3423-L3425 https://github.com/git/git/blob/e188ec3a735ae52a0d0d3c22f9df...
- forrestthewoods 4y ago> At GitHub, we store a lot of Git data: more than 18.6 petabytes of it, to be precise. That actually seems kinda small. Git’s lack of good support for large files means there’s probably an exabyte of data that, imho, should be source control but isn’t.
- kortex 4y agoIt would be amazing if Github/lab provided a backing store for www.dvc.org . I've been using to great effect, but I have to rely on separate AWS integration for storing the large objects in s3.
- arjvik 4y agoTake a look at DagsHub - https://dagshub.com https://dagshub.com They do exactly what you're looking for! They provide both a DVC remote to push and pull DVC-tracked objects, as well as a UI showing them integrated alongside git-tracked files in the repository view. (Disclaimer: I work at DagsHub)
- kccqzy 4y agoThat's indeed small. I'd guess that Google probably stores 4 orders of magnitude more data than GitHub. (I was in fact asked a long time ago in an interview to estimate how much disk was needed to store Google's search index.)
- sulam 4y agoGlad it was a long time ago. Those kinds of questions are awful.
- isatty 4y agoAgreed that it isn’t ideal, but about “awful” specifically - I’m not too sure. I would never ask such a question but I would assume the intent is just to find out how you think and not to get you to spit out a number. Would it be fun if the interviewer worked together with you to approximate it?
- cutierust 4y agoJust use pijul. Backend might be good but I'm tired of gits ui and am impressed with pijul so far (very unfurnished yet,)
- ironmagma 4y agoI would, but PIJUL IS BROKEN* and nobody has fixed it since it was reported. [1] [1] https://discourse.pijul.org/t/signup-on-pijul-nest-gives-forbidden-and-thats-it/949 https://discourse.pijul.org/t/signup-on-pijul-nest-gives-for... * forgive the caps, I know I have sinned but I do wanna be able to use this VCS.
- pmeunier 4y agoThis is just the Nest, not Pijul. I'm responsible for that, and sorry about it. I said I'd fix it soon, at the moment I'm working on an app to fix (a small part of) France's electricity crisis. It's mostly done, but it took me a while to write something that would be able to scale to loads of users before the winter.
- ironmagma 4y agoThank you for the update, and the electricity crisis is (probably... :P) more important. No biggie.
- pmeunier 4y agoI'm just following Rustacean traditions of fixing things from the root. When you have to write a web browser, you do have to fix C++ before you can actually start. I'm digging a bit deeper by avoiding blackouts so my French servers keep running (the Nest has two other locations), then I'll fix the server code.
- cutierust 4y agoThat's nest not pijul.
- 1letterunixname 4y agoMost people should `git clone --depth 1` most of the time. Large businesses should do trunk-based development with something other than git that scales using distributed file systems with phantom checkouts, negating the need for LFS blob stores.
- vildravn 4y ago> with something other than git that scales using distributed file systems with phantom checkouts, negating the need for LFS blob stores. What would that something be for example?
- 1letterunixname 4y agoNDA, so I can't talk about it. I would like to but I can't.
- saagarjha 4y agoNot that you should do anything you're not comfortable with but I will note that the source control systems for most major companies are essentially known, especially around here. "At Facebook we use a Mercurial-inspired thing with some local extensions" is not really earth-shattering.
- tex0 4y agoThat reminds me of Piper/CitC: https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext https://cacm.acm.org/magazines/2016/7/204032-why-google-stor...
- iudqnolq 4y agoI haven't been able to find a link recently, but I recall from a few years ago that that's much harder on GitHub's servers. One of the newer package managers that uses GitHub for hosting (brew maybe?) tried to be helpful and default all their users to shallow clones. GitHub people wanted them to push an update ASAP to go back to full clones, and there was an issue thread laying it all out. If I recall correctly it has to do with the way they cache git. If you request shallow clones they have to run a git process each time to make a custom download with only the data you need. Full clones get served from a cache.
- Denisos 4y agoam surprised they didnt use time