4 ms·
I have written a small python library to read info directly from the .git folder (without using the git executable): https://github.com/spapas/python-git-info/
by spapas82 4y ago
I have written a small python library to read info directly from the .git folder (without using the git executable): https://github.com/spapas/python-git-info/ https://github.com/spapas/python-git-info/
Decoding the pack files was notoriously difficult, probably there are still cases that I don't handle properly. The packfile is a really complex/messy format and in the top of that it lacks any proper documentation!
This article explains a lot of the concepts very good but doesn't go into implementation details (where the big snakes are). Another great resource for unpacking packfiles is this post: https://codewords.recurse.com/issues/three/unpacking-git-packfiles https://codewords.recurse.com/issues/three/unpacking-git-pac... and of course reading the source code of other git clients.
Unfortunately it seems that the official packfile format documentation is the git source code :|
- masklinn 4y ago> The packfile is a really complex/messy format and in the top of that it lacks any proper documentation! Maybe it’s changed since you'd checked it, but I found the pack files to be documented, through nowhere near as well as index files (now that was a pleasure, the index file format is straightforward, pretty logical, and very well documented). The problem I had with the pack files documentation is that it’s non-linear, so if you read it linearly as you implement it you hit areas which turn out to be documented a few screens later. Furthermore it doesn’t necessarily define or spell out what it’s talking about, so it can take a while to realise that it has 3 (.5?) different formats of varints, or that the size it provides is for decompressed content, and that it relies on zlib to discover the end of the compressed stream (and good luck to you if your implementation doesn’t expose that). But in my experience, it’s really nothing compared to the documentation of the wire format. That leaves even more details out, some of the explanations are outright misleading (I spent hours convinced the v2 protocol was using an HTTP connection as a half-duplex socket and wondering how I would manage), and with TODOs covering half the protocol.
- spapas82 4y agoCan you please share the documentation you used to understand the pack file implemention? It will be very useful to me!
- masklinn 4y agoGit’s own: https://github.com/git/git/blob/master/Documentation/gitformat-pack.txt https://github.com/git/git/blob/master/Documentation/gitform... I also used gitoxide’s source when I got unsure / lost about the nomenclature / varints. I’d suggest giving it a pass unless you have to or are really interested though.
- spapas82 4y agoI stand corrected these are much better that what I was able to find! Thank you
- ori_b 4y agoYes. The wire format docs, especially for HTTP, are atrocious. The pack format is well documented enough. (https://github.com/git/git/tree/maint/Documentation/technical https://github.com/git/git/tree/maint/Documentation/technica...) I wrote the 9front implementation (https://git.9front.org/plan9front/plan9front/HEAD/sys/src/cmd/git/f.html https://git.9front.org/plan9front/plan9front/HEAD/sys/src/cm...) more or less entirely from the official documentation in the git repo. For debugging, GIT_TRACE_PACKET=1 is very useful for getting valid packet traces. Repacking efficiently for serving is the most annoying part, and I still haven't implemented delta reuse.
- masklinn 4y agoFWIW git has recently reorganised their docs a lot, and moved some things to man pages. The pack format docs were one of the documents affected, it now lives at https://github.com/git/git/blob/master/Documentation/gitformat-pack.txt https://github.com/git/git/blob/master/Documentation/gitform....