7 ms·

Mounting Git commits as folders with NFS

211 points by chmaynard 3y ago

reverius42 3y ago
Very nice! I really like the design choices here. Though I (edit: am biased and) would personally have used Rust and https://github.com/xetdata/nfsserve https://github.com/xetdata/nfsserve.
- eru 3y ago
  I've built something like this as well, but in FUSE for Linux, not via NFS. It really brings home how close git's design already is to being a file system.
  - bmacho 3y ago
    A virtual file system? How usable are those? I want something that can store slightly different data (modifies data on the fly), and one, that I can use as a tagged file system (generates folder structure on the fly). Is that viable with FUSE/NFS? Do programs play nice with them (no caching, no recursive lookahead and such)? Is it hard to write one? Which language did you used?
    - eru 3y ago
      I wrote version in Python and Rust. (And ended up contributing to libfuse and its Rust equivalent along the way.) > Do programs play nice with them (no caching, no recursive lookahead and such)? Other programs play nice with my peculiar file systems as far as I can tell: they just look like ordinary files and directories and symlinks etc to them. I don't know how well that would work for your usecase. Most of the caching etc that I am aware of happens at the layer of the Linux kernel, and your FUSE daemon can configure what caching it wants or doesn't want. I don't know much about NFS. I only used FUSE directly. For me, presenting git as a filesystem was relatively easy to write. I think that comes down to two main factors: * Git's internal structure is very close to how Linux (and especially fuse) do filesystems. That's probably not a big surprise, given that git and the Linux kernel are both projects started by the same guy. * I exposed the git data as a read-only filesystem. In general, if you can avoid mutation, you can avoid a lot of complexity. (The main source of complexity in the Python version I wrote first was that we allowed wanted to reflect changes in the contents of git branches to be reflected in the filesystem side. That's mutability flowing in only one direction (from git to the filesystem), but it was already annoying enough to deal with.) On a more general note: I see filesystems mostly as one peculiar interface you can put in front of your data. So one some level git has an underlying database, and we just expose it. The same could be done for eg postgres or mongodb with a suitable fuse daemon. But you can also think of ext4 being such a database with a weird UI layer on top. The main difference in practice is that they run their 'fuse daemon' in the kernel. From what I've read the new bcachefs explicitly embraces this view of 'filesystems are just databases', and uses techniques borrowed from relational databases for its internal datastructures.
- rapnie 3y ago
  Functional but not complete. The project is looking for maintainers. Overall it is a pity how so many potentially great Rust projects are inactive and left in incomplete state.
  - reverius42 3y ago
    It's not complete in its current state with respect to the NFS protocol, but is very usable as a base for things that work (for instance, https://github.com/xetdata/xet-core https://github.com/xetdata/xet-core uses it to mount git(-xet) repos as directories, similarly to the linked post but with some different design choices). It is maintained for this use case and contributors are welcome for other use cases that aren't covered yet.
trollied 3y ago
Reminds me of Rational ClearCase, which probably inspired the idea. You could specify a "view" using tags, and it'd present it as a filesystem to remote machines. I think IBM now own them.
- bigstrat2003 3y ago
  It also reminds me of ClearCase, which is why I kind of hate this, lol. ClearCase is fucking awful to work with, the last thing I want is to turn git into it.
  - codewiz 3y ago
    Several years ago, one of my customers deployed Rational ClearCase across the company. Every employee and consultant, me included, was signed up for a week-long course on how to use the "Rational Unified Process" and other nonsense like that. By day 2, it was clear that they were selling us snake oil, but my customer was paying for my time, so I sit through it to the end. A few engineers in the room were genuinely hooked up by the promise of generating 90% of the code from UML, integration streams, automatic "un-branching" and all that. I wonder what they spent overall in licensing fees, training and lost productivity? And that's probably a fraction of the long-term damage dealt by this absurd process to a large C++ codebase. Somewhere, I should still have a "degree" issued by Rational University :-)
    - donaldihunter 3y ago
      Yeah, that's mostly nothing to do with ClearCase tho. Raw ClearCase was just a VCS. It was Unified Change Management (UCM) that brought in the Rational Unified Process garbage.
    - jerf 3y ago
      Wow. Circa 1999 my software engineering class, the college class that has the distinction of being the only class I've ever taken that I'm pretty sure I now disagree with literally everything that was taught in it, had us using that software suite to try to do that. I recall it made a very uncompelling case, on the grounds that it was putatively the future of software engineering, and you could hardly right-click on anything without the stupid thing hard crashing. Pro-tip: If your software is the future of software engineering, an engineering student using your software to do exactly what it was designed to do should not be able to crash it in under two minutes. And then keep crashing it. But even when it was working, it was literally virtually impossible to so much as contort a student assignment into that model. I can't imagine working somewhere that insisted on building production software that way, and I'm shocked they're still finding enough chumps to stay in business with that complete and utter trip pipe dream. (In a nutshell, the primary problem with such systems is that they do not account for the fact that every entity in a diagram is a cost. Every entity in a diagram needs to carry a value in excess of that cost, preferably comfortably so. Any methodology that insists on a totalizing view of the world in which everything must be in a diagram will produce diagrams so cognitively expensive that they are just a blurred mass of diagram entities no easier to understand than the underlying code. The people pushing these systems build nice little demo diagrams with 10-15 entities in them that are easy to understand, and then incorrectly attribute the ease of understanding to the fact that it is a diagram, rather than the fact the diagram only contains a handful of entities! Then they build systems based on the utterly incorrect belief that all diagrams are simple. The results are entirely predictable when you see it from this point of view. The part that's mindblowing is just how hard some people need to be beaten over their skull with the fact that diagrams aren't necessarily simple once they've ingested this idea, no matter how many hundreds of insanely complicated diagrams they stare at over the years....)
masklinn 3y ago
> Git repositories sometimes have submodules. I don’t understand anything about submodules so right now I’m just ignoring them. Submodules are interesting, because they’re next to unusable from a user perspective (they’re a pain to maintain and interact with unless you never ever update them) but they’re ridiculously simple technically which I assume is what made them attractive. A submodule is an entry in “.gitmodules” mapping a path to a repository URL (and branch), then at the specified path in the repository is a tree entry of mode 160000 (S_IFDIR + S_IFLNK), whose oid is the commit to check out (in the submodule-linked repository).
- AceJohnny2 3y ago
  To add to this, Submodules are a hack on Git's data model. Git's data model, put simply, is this: * Branches are pointers to Commit objects * Commit objects are a composite of {Commit_Comment, Tree, Parent Commit(s)}, referenced by the hash of that set * a Tree (like a directory) is a list of Blobs and/or Trees (associating filenames with them) referenced by the hash of its contents. * a Blob is a file, referenced by the hash of its contents. So the set of types of objects in Git are: Commits, Trees, and Blobs. Note that I said that a Tree can contain other Trees or Blobs... but what if... you put a Commit in it!? That's a submodule! Now if the Commit you reference doesn't exist in the current repo, Git can't do anything with it. That's where the .gitmodules file comes in, to associate a given path with a repo, so that Git can look up the Commit object in that repo.
  - masklinn 3y ago
    > Branches are pointers to Commit objects Nit: the data model has refs, which are pointer to objects. Branches are the subset of refs in the special-cased heads/ namespace which should be pointing to commit objects. And there’s also tags, which are the subset of refs in the special-cased tags/ namespace, which should be pointing to commit (“lightweight”) or tag (“annotated”) objects.
    - AceJohnny2 3y ago
      Absolutely. Going further, the little-known git-notes [1] feature also uses its own reference namespace, `refs/notes/` Going even further, Gerrit [2] leverages the wide-open reference namespace/directories to create its own. For example, pushing under the `refs/for/` namespace creates a new review, and specific reviews can be looked up under the `refs/changes/` namespace. Even even further, Gerrit's special repos All-Users.git and All-Projects.git are "databases" for project configuration and user configuration, where for example external IDs (like usernames) are stored under the special `refs/meta/external-ids` ref/branch. This has the notable benefit that all configuration changes are tracked and auditable. I believe git-appraise [3] also leverages special reference namespaces in Git for performing its review duties (but I don't know details) Edit: actually no, it "just" leverages git-notes. [1] https://git-scm.com/docs/git-notes https://git-scm.com/docs/git-notes [2] https://gerrit-review.googlesource.com/Documentation/note-db.html https://gerrit-review.googlesource.com/Documentation/note-db... [3] https://github.com/google/git-appraise https://github.com/google/git-appraise
mbakke 3y ago
For those who resonate with "why might this be useful", here are "plain git" alternatives to this tool: > searching for a function I deleted git log -G someFunc > quickly looking at a file on another branch to copy a line from it I use `git worktree` to "mount" long-running branches much to the same effect as Julias tool. To quickly look at a file from a non-mounted branch/commit, I use: git show $REF:$FILENAME > searching every branch for a function git log --all -G someFunc Note that -G can be replaced with -F for a speedup if the pattern you are searching for is a fixed string.
- levidos 3y ago
  Extremely handy, saving these. Thank you.
- globular-toast 3y ago
  Magit has a really easy to use way to "step" through previous versions of files. It's usually bound to something like "C-f p". You get a read only buffer of the previous version open in the best text editor (emacs). You can then press n and p to step through next and previous versions of that file. Can be pretty useful! It's kind of funny, I think, how most git users don't seem to know how to access any version other than the current one. So many people think of it simply as the annoying tool you have to use to make code changes but don't really know what version control is.
  - divbzero 3y ago
    That’s a pretty cool feature of Magit. I was inspired to look for something similar for the next best text editor (vim) and came across this: https://salferrarello.com/using-vim-view-git-commits/ https://salferrarello.com/using-vim-view-git-commits/ git log | vim -R - Placing your cursor over a commit hash and entering K displays git show for that commit.
    - cnity 3y ago
      If you're not using neovim you're really missing out right now IMO. It's a renaissance for a hackable text editor because it uses a sensible modern programming language (Lua) rather than vimscript (yikes) or elisp (eh).
okl 3y ago
Git already has a similar feature. It's called worktree https://git-scm.com/docs/git-worktree https://git-scm.com/docs/git-worktree For example: `git worktree add <folder> <commit-hash>` checks out that commit in that folder.
- vnorilo 3y ago
  TFA is for the actual data in the commit, not a parallel worktree.
- Vohlenzer 3y ago
  I find it helpful to keep a few worktrees around for common tasks. This saves me from throwing away my working copy when I switch to a different task; /source/repos/AcmeCorp 2fddd74f9a [bug/CurrentWork] /source/repos/AcmeCorp-hotfix 27175cf6c5 [hotfix/2023-11-27.1] /source/repos/AcmeCorp-master 016ca20b75 [master] /source/repos/AcmeCorp-reference 454be5348d [feature/RecentReviewedWork] /source/repos/AcmeCorp-release 95027177d7 [release9.12.0]
- masklinn 3y ago
  Worktrees require checking out each branch individually. And they’re something of a pain in the ass to manage as you can’t have two worktrees to the same branch. So for temporary querying across a few branches doing clones is a lot easier (git will hardlink when cloning on the same FS so performances are not an issue, and you can just delete the scratch clones afterwards without the need to prune the worktrees), and if you regularly need to query things across a lot of branches worktrees are unusable.
kristopolous 3y ago
This reminds me of the SVN days.
CodeCompost 3y ago
IMO one of the major flaws of Subversion is that branches are folders.
- masklinn 3y ago
  OTOH it’s also a major advantage of subversion, because if you have long-running branches and need to fix something in all of them all the fixes can be in the same commit across every branch instead of needing a commit in each branch informally linked to the previous one (or formally so via a merge commit if every fix in an older branch has to get ported forwards, as an empty commit if not applicable).
assimpleaspossi 3y ago
I just hate these things where people use the terms "folder" and "directory" as if they were the same thing. They are not and I wish some people would wise up to that. Yes. This is a pet peeve of mine.
- ongy 3y ago
  Please don't just rant about it, but explain, or link to an explanation, the difference so others can see if there's actually a substantial difference.
  - arbitrandomuser 3y ago
    > Please don't just rant about it, but explain, or link to an explanation, the difference so others can see if there's actually a substantial difference. Fairly easy to look up and I just did , https://stackoverflow.com/questions/5078676/what-is-the-difference-between-a-directory-and-a-folder https://stackoverflow.com/questions/5078676/what-is-the-diff... but I agree , comments like this come across as smug and unhelpful. A few short lines explaining and a link would have been much helpful and greatly improves the quality of discussion on this site.
    - avgcorrection 3y ago
      The top answer > > There is a difference between a directory, which is a file system concept, and the graphical user interface metaphor that is used to represent it (a folder). So there really is no difference in a technical (programmers talking) context. :)
      - boffinAudio 3y ago
        There is no difference between a directory and a folder except the term used to refer to the special kind of file that can hold links to other files. :) Bonus points if you can find any kind of modern user that knows the difference between a FILE and a FOLDER (directory) ..
      - 3y ago
franky47 3y ago
Is there a project that does the reverse of this, namely mounting a filesystem where every write to a file in the repository becomes a Git commit?
- extraduder_ire 3y ago
  You mean like when nothing has the file open any more, or doing a commit with every flush to disk? I think the latter is a bit excessive. One of the FUSE implementations mentioned in this post may do something like that.
  - franky47 3y ago
    File handle refcounting would be nice to avoid spamming Git with commits, indeed.
  - dekhn 3y ago
    I tried this with mod_dav_svn back in the day and never really managed to convince myself that commit-on-write is a good idea.
- eterps 3y ago
  I have been looking for that as well. Some sort of 'implicit versioning', a bit like how version history works for Google Docs [1], but instead for the filesystem. [1] https://support.google.com/docs/answer/190843 https://support.google.com/docs/answer/190843
- joh6nn 3y ago
  doesn't seem like the project is in active development anymore, but there's gitFS: https://github.com/presslabs/gitfs https://github.com/presslabs/gitfs
- pacifika 3y ago
  https://filehamster.com/ https://filehamster.com/ Maybe?
tambourine_man 3y ago
I don’t understand why Apple doesn’t provide a rock solid first party FUSE implementation. This and a container story are the main things holding macOS back.
- callalex 3y ago
  If it doesn’t increase revenue that they can tax at 30% they won’t bother to touch it.
- mdaniel 3y ago
  I believe both of your complaints have the same underlying cause: they are Linux-centric (that goes extra for the "container story" part since I doubt very seriously that you have a use case for running containerized Darwin executables) I believe Apple's version of FUSE is https://developer.apple.com/documentation/fileprovider/nonreplicated_file_provider_extension https://developer.apple.com/documentation/fileprovider/nonre... and, while I am absolutely certain someone will point out things that FUSE has that the macOS version doesn't, I'm only saying that, just like Windows, macOS has its own way of doing things and thus very little incentive to implement someone else's vision
  - tambourine_man 3y ago
    Apple goes to great lengths to increase security while annoying its users. Containers could help in that regard. Having an app (and GUI too) run in a container of sorts could avoid the countless dialog boxes we’ve grown accustomed to. Regarding FUSE, there have been many implementations for macOS, just not very good ones. I don’t see a fundamental limitation on XNU, Darwin, Mach. I haven’t though of File Provider in this way, but maybe it’s a good solution. Apple seems reasonably committed to it so that’s encouraging.
evgpbfhnr 3y ago
> stale file handles... This is still a problem and I’m not sure how to fix it. I don’t understand how real NFS servers do this, maybe they just have a really big cache? While some implementations probably cache things, most filesystems have an underlying file handle which allow opening the file directly (see name_to_handle_at(2)/open_by_handle_at(2) on linux, but other systems also have something similar it's just not necessarily exposed to userspace) NFS servers can then just expose this along with some internal stuff (like export id for internal permissions checks etc) This can lead to some interesting misfeature if a user can guess the handles -- on most filesystems it'll just be something like the inode number and some extra stuff that depending on the filesystem can be contiguously allocated, so +1 can work -- in that open by handle at won't check that the user has access to the full path: so a malign user could access some/subdir/file if file itself is open even if subdir isn't accessible. Thanksfully most recent filesystems allocate inodes pseudo-randomly, so this won't work well for most people. (In this particular case, you could have commit id + a big hash table with paths inside the repo? Most recently large repos would fit in memory for one single revision and layout doesn't change all that much between commits, so that could work fairly well a bit past linux kernel size. For humongous repos some more tricks might be needed though)
- EdSchouten 3y ago
  Worth mentioning that if you use NFSv4, file handles can be up to 128 bytes in size. That’s big enough that you can sometimes get away with dumping the entire file state in it. For Buildbarn (a distributed build cluster for Bazel) I also wrote an NFSv4 server in Go to act as a lazy-loading file system for input files. As the remote execution protocol uses Content Addressable Storage, I ended up dumping the entire SHA-256, size and permission bits of files in the NFSv4 file handle. This allows me to reconstruct files as needed. Some notes I made at the time (which are written with Buildbarn knowledge in mind): https://github.com/buildbarn/bb-adrs/blob/master/0009-nfsv4.md https://github.com/buildbarn/bb-adrs/blob/master/0009-nfsv4....
- cesarb 3y ago
  > most filesystems have an underlying file handle which allow opening the file directly (see name_to_handle_at(2)/open_by_handle_at(2) on linux, but other systems also have something similar it's just not necessarily exposed to userspace) On Windows, access to this is controlled by the confusingly named "Bypass traverse checking" (aka SeChangeNotifyPrivilege) permission (see for instance https://techcommunity.microsoft.com/t5/windows-blog-archive/the-bypass-traverse-checking-or-is-it-the-change-notify/ba-p/723428 https://techcommunity.microsoft.com/t5/windows-blog-archive/... for some information about it; I recall once reading an article from either Raymond Chen or Larry Osterman explaining the naming of that permission, but can't find it at the moment).
skadamat 3y ago
Big fans of NFS, we built a Rust library that implemented NFS for a similar same use case (mounting Git repos): https://about.xethub.com/blog/nfs-fuse-why-we-built-nfs-server-rust https://about.xethub.com/blog/nfs-fuse-why-we-built-nfs-serv...
orf 3y ago
It’s a shame this is NFSv3. It’s my understanding that NFSv4 is significantly faster to access due to pipelining requests?
avgcorrection 3y ago
> The main reason I wanted to make this was to give folks some intuition for how git works under the hood. After all, git commits really are very similar to folders – every Git commit contains a directory listing of the files in it, and that directory can have subdirectories, etc. > It’s just that git commits aren’t actually implemented as folders to save disk space. Okay. So what. People are used to archive formats. > git worktree also lets you have multiple branches checked out at the same time, but to me it feels weird to set up an entire worktree just to look at 1 file. But one directory per commit is less weird.

211 points by chmaynard 3y ago

Very nice! I really like the design choices here. Though I (edit: am biased and) would personally have used Rust and https://github.com/xetdata/nfsserve https://github.com/xetdata/nfsserve.

eru 3y ago
I've built something like this as well, but in FUSE for Linux, not via NFS. It really brings home how close git's design already is to being a file system.
- bmacho 3y ago
  A virtual file system? How usable are those? I want something that can store slightly different data (modifies data on the fly), and one, that I can use as a tagged file system (generates folder structure on the fly). Is that viable with FUSE/NFS? Do programs play nice with them (no caching, no recursive lookahead and such)? Is it hard to write one? Which language did you used?
  - eru 3y ago
    I wrote version in Python and Rust. (And ended up contributing to libfuse and its Rust equivalent along the way.) > Do programs play nice with them (no caching, no recursive lookahead and such)? Other programs play nice with my peculiar file systems as far as I can tell: they just look like ordinary files and directories and symlinks etc to them. I don't know how well that would work for your usecase. Most of the caching etc that I am aware of happens at the layer of the Linux kernel, and your FUSE daemon can configure what caching it wants or doesn't want. I don't know much about NFS. I only used FUSE directly. For me, presenting git as a filesystem was relatively easy to write. I think that comes down to two main factors: * Git's internal structure is very close to how Linux (and especially fuse) do filesystems. That's probably not a big surprise, given that git and the Linux kernel are both projects started by the same guy. * I exposed the git data as a read-only filesystem. In general, if you can avoid mutation, you can avoid a lot of complexity. (The main source of complexity in the Python version I wrote first was that we allowed wanted to reflect changes in the contents of git branches to be reflected in the filesystem side. That's mutability flowing in only one direction (from git to the filesystem), but it was already annoying enough to deal with.) On a more general note: I see filesystems mostly as one peculiar interface you can put in front of your data. So one some level git has an underlying database, and we just expose it. The same could be done for eg postgres or mongodb with a suitable fuse daemon. But you can also think of ext4 being such a database with a weird UI layer on top. The main difference in practice is that they run their 'fuse daemon' in the kernel. From what I've read the new bcachefs explicitly embraces this view of 'filesystems are just databases', and uses techniques borrowed from relational databases for its internal datastructures.
rapnie 3y ago
Functional but not complete. The project is looking for maintainers. Overall it is a pity how so many potentially great Rust projects are inactive and left in incomplete state.
- reverius42 3y ago
  It's not complete in its current state with respect to the NFS protocol, but is very usable as a base for things that work (for instance, https://github.com/xetdata/xet-core https://github.com/xetdata/xet-core uses it to mount git(-xet) repos as directories, similarly to the linked post but with some different design choices). It is maintained for this use case and contributors are welcome for other use cases that aren't covered yet.

trollied 3y ago

Reminds me of Rational ClearCase, which probably inspired the idea. You could specify a "view" using tags, and it'd present it as a filesystem to remote machines. I think IBM now own them.

bigstrat2003 3y ago
It also reminds me of ClearCase, which is why I kind of hate this, lol. ClearCase is fucking awful to work with, the last thing I want is to turn git into it.
- codewiz 3y ago
  Several years ago, one of my customers deployed Rational ClearCase across the company. Every employee and consultant, me included, was signed up for a week-long course on how to use the "Rational Unified Process" and other nonsense like that. By day 2, it was clear that they were selling us snake oil, but my customer was paying for my time, so I sit through it to the end. A few engineers in the room were genuinely hooked up by the promise of generating 90% of the code from UML, integration streams, automatic "un-branching" and all that. I wonder what they spent overall in licensing fees, training and lost productivity? And that's probably a fraction of the long-term damage dealt by this absurd process to a large C++ codebase. Somewhere, I should still have a "degree" issued by Rational University :-)
  - donaldihunter 3y ago
    Yeah, that's mostly nothing to do with ClearCase tho. Raw ClearCase was just a VCS. It was Unified Change Management (UCM) that brought in the Rational Unified Process garbage.
  - jerf 3y ago
    Wow. Circa 1999 my software engineering class, the college class that has the distinction of being the only class I've ever taken that I'm pretty sure I now disagree with literally everything that was taught in it, had us using that software suite to try to do that. I recall it made a very uncompelling case, on the grounds that it was putatively the future of software engineering, and you could hardly right-click on anything without the stupid thing hard crashing. Pro-tip: If your software is the future of software engineering, an engineering student using your software to do exactly what it was designed to do should not be able to crash it in under two minutes. And then keep crashing it. But even when it was working, it was literally virtually impossible to so much as contort a student assignment into that model. I can't imagine working somewhere that insisted on building production software that way, and I'm shocked they're still finding enough chumps to stay in business with that complete and utter trip pipe dream. (In a nutshell, the primary problem with such systems is that they do not account for the fact that every entity in a diagram is a cost. Every entity in a diagram needs to carry a value in excess of that cost, preferably comfortably so. Any methodology that insists on a totalizing view of the world in which everything must be in a diagram will produce diagrams so cognitively expensive that they are just a blurred mass of diagram entities no easier to understand than the underlying code. The people pushing these systems build nice little demo diagrams with 10-15 entities in them that are easy to understand, and then incorrectly attribute the ease of understanding to the fact that it is a diagram, rather than the fact the diagram only contains a handful of entities! Then they build systems based on the utterly incorrect belief that all diagrams are simple. The results are entirely predictable when you see it from this point of view. The part that's mindblowing is just how hard some people need to be beaten over their skull with the fact that diagrams aren't necessarily simple once they've ingested this idea, no matter how many hundreds of insanely complicated diagrams they stare at over the years....)

masklinn 3y ago

> Git repositories sometimes have submodules. I don’t understand anything about submodules so right now I’m just ignoring them. Submodules are interesting, because they’re next to unusable from a user perspective (they’re a pain to maintain and interact with unless you never ever update them) but they’re ridiculously simple technically which I assume is what made them attractive. A submodule is an entry in “.gitmodules” mapping a path to a repository URL (and branch), then at the specified path in the repository is a tree entry of mode 160000 (S_IFDIR + S_IFLNK), whose oid is the commit to check out (in the submodule-linked repository).

AceJohnny2 3y ago
To add to this, Submodules are a hack on Git's data model. Git's data model, put simply, is this: * Branches are pointers to Commit objects * Commit objects are a composite of {Commit_Comment, Tree, Parent Commit(s)}, referenced by the hash of that set * a Tree (like a directory) is a list of Blobs and/or Trees (associating filenames with them) referenced by the hash of its contents. * a Blob is a file, referenced by the hash of its contents. So the set of types of objects in Git are: Commits, Trees, and Blobs. Note that I said that a Tree can contain other Trees or Blobs... but what if... you put a Commit in it!? That's a submodule! Now if the Commit you reference doesn't exist in the current repo, Git can't do anything with it. That's where the .gitmodules file comes in, to associate a given path with a repo, so that Git can look up the Commit object in that repo.
- masklinn 3y ago
  > Branches are pointers to Commit objects Nit: the data model has refs, which are pointer to objects. Branches are the subset of refs in the special-cased heads/ namespace which should be pointing to commit objects. And there’s also tags, which are the subset of refs in the special-cased tags/ namespace, which should be pointing to commit (“lightweight”) or tag (“annotated”) objects.
  - AceJohnny2 3y ago
    Absolutely. Going further, the little-known git-notes [1] feature also uses its own reference namespace, `refs/notes/` Going even further, Gerrit [2] leverages the wide-open reference namespace/directories to create its own. For example, pushing under the `refs/for/` namespace creates a new review, and specific reviews can be looked up under the `refs/changes/` namespace. Even even further, Gerrit's special repos All-Users.git and All-Projects.git are "databases" for project configuration and user configuration, where for example external IDs (like usernames) are stored under the special `refs/meta/external-ids` ref/branch. This has the notable benefit that all configuration changes are tracked and auditable. I believe git-appraise [3] also leverages special reference namespaces in Git for performing its review duties (but I don't know details) Edit: actually no, it "just" leverages git-notes. [1] https://git-scm.com/docs/git-notes https://git-scm.com/docs/git-notes [2] https://gerrit-review.googlesource.com/Documentation/note-db.html https://gerrit-review.googlesource.com/Documentation/note-db... [3] https://github.com/google/git-appraise https://github.com/google/git-appraise

mbakke 3y ago

For those who resonate with "why might this be useful", here are "plain git" alternatives to this tool: > searching for a function I deleted git log -G someFunc > quickly looking at a file on another branch to copy a line from it I use `git worktree` to "mount" long-running branches much to the same effect as Julias tool. To quickly look at a file from a non-mounted branch/commit, I use: git show $REF:$FILENAME > searching every branch for a function git log --all -G someFunc Note that -G can be replaced with -F for a speedup if the pattern you are searching for is a fixed string.

levidos 3y ago
Extremely handy, saving these. Thank you.
globular-toast 3y ago
Magit has a really easy to use way to "step" through previous versions of files. It's usually bound to something like "C-f p". You get a read only buffer of the previous version open in the best text editor (emacs). You can then press n and p to step through next and previous versions of that file. Can be pretty useful! It's kind of funny, I think, how most git users don't seem to know how to access any version other than the current one. So many people think of it simply as the annoying tool you have to use to make code changes but don't really know what version control is.
- divbzero 3y ago
  That’s a pretty cool feature of Magit. I was inspired to look for something similar for the next best text editor (vim) and came across this: https://salferrarello.com/using-vim-view-git-commits/ https://salferrarello.com/using-vim-view-git-commits/ git log | vim -R - Placing your cursor over a commit hash and entering K displays git show for that commit.
  - cnity 3y ago
    If you're not using neovim you're really missing out right now IMO. It's a renaissance for a hackable text editor because it uses a sensible modern programming language (Lua) rather than vimscript (yikes) or elisp (eh).

okl 3y ago

Git already has a similar feature. It's called worktree https://git-scm.com/docs/git-worktree https://git-scm.com/docs/git-worktree For example: `git worktree add <folder> <commit-hash>` checks out that commit in that folder.

vnorilo 3y ago
TFA is for the actual data in the commit, not a parallel worktree.
Vohlenzer 3y ago
I find it helpful to keep a few worktrees around for common tasks. This saves me from throwing away my working copy when I switch to a different task; /source/repos/AcmeCorp 2fddd74f9a [bug/CurrentWork] /source/repos/AcmeCorp-hotfix 27175cf6c5 [hotfix/2023-11-27.1] /source/repos/AcmeCorp-master 016ca20b75 [master] /source/repos/AcmeCorp-reference 454be5348d [feature/RecentReviewedWork] /source/repos/AcmeCorp-release 95027177d7 [release9.12.0]
masklinn 3y ago
Worktrees require checking out each branch individually. And they’re something of a pain in the ass to manage as you can’t have two worktrees to the same branch. So for temporary querying across a few branches doing clones is a lot easier (git will hardlink when cloning on the same FS so performances are not an issue, and you can just delete the scratch clones afterwards without the need to prune the worktrees), and if you regularly need to query things across a lot of branches worktrees are unusable.

kristopolous 3y ago

This reminds me of the SVN days.

CodeCompost 3y ago

IMO one of the major flaws of Subversion is that branches are folders.

masklinn 3y ago
OTOH it’s also a major advantage of subversion, because if you have long-running branches and need to fix something in all of them all the fixes can be in the same commit across every branch instead of needing a commit in each branch informally linked to the previous one (or formally so via a merge commit if every fix in an older branch has to get ported forwards, as an empty commit if not applicable).

assimpleaspossi 3y ago

I just hate these things where people use the terms "folder" and "directory" as if they were the same thing. They are not and I wish some people would wise up to that. Yes. This is a pet peeve of mine.

ongy 3y ago
Please don't just rant about it, but explain, or link to an explanation, the difference so others can see if there's actually a substantial difference.
- arbitrandomuser 3y ago
  > Please don't just rant about it, but explain, or link to an explanation, the difference so others can see if there's actually a substantial difference. Fairly easy to look up and I just did , https://stackoverflow.com/questions/5078676/what-is-the-difference-between-a-directory-and-a-folder https://stackoverflow.com/questions/5078676/what-is-the-diff... but I agree , comments like this come across as smug and unhelpful. A few short lines explaining and a link would have been much helpful and greatly improves the quality of discussion on this site.
  - avgcorrection 3y ago
    The top answer > > There is a difference between a directory, which is a file system concept, and the graphical user interface metaphor that is used to represent it (a folder). So there really is no difference in a technical (programmers talking) context. :)
    - boffinAudio 3y ago
      There is no difference between a directory and a folder except the term used to refer to the special kind of file that can hold links to other files. :) Bonus points if you can find any kind of modern user that knows the difference between a FILE and a FOLDER (directory) ..
    - 3y ago

franky47 3y ago

Is there a project that does the reverse of this, namely mounting a filesystem where every write to a file in the repository becomes a Git commit?

extraduder_ire 3y ago
You mean like when nothing has the file open any more, or doing a commit with every flush to disk? I think the latter is a bit excessive. One of the FUSE implementations mentioned in this post may do something like that.
- franky47 3y ago
  File handle refcounting would be nice to avoid spamming Git with commits, indeed.
- dekhn 3y ago
  I tried this with mod_dav_svn back in the day and never really managed to convince myself that commit-on-write is a good idea.
eterps 3y ago
I have been looking for that as well. Some sort of 'implicit versioning', a bit like how version history works for Google Docs [1], but instead for the filesystem. [1] https://support.google.com/docs/answer/190843 https://support.google.com/docs/answer/190843
joh6nn 3y ago
doesn't seem like the project is in active development anymore, but there's gitFS: https://github.com/presslabs/gitfs https://github.com/presslabs/gitfs
pacifika 3y ago
https://filehamster.com/ https://filehamster.com/ Maybe?

tambourine_man 3y ago

I don’t understand why Apple doesn’t provide a rock solid first party FUSE implementation. This and a container story are the main things holding macOS back.

callalex 3y ago
If it doesn’t increase revenue that they can tax at 30% they won’t bother to touch it.
mdaniel 3y ago
I believe both of your complaints have the same underlying cause: they are Linux-centric (that goes extra for the "container story" part since I doubt very seriously that you have a use case for running containerized Darwin executables) I believe Apple's version of FUSE is https://developer.apple.com/documentation/fileprovider/nonreplicated_file_provider_extension https://developer.apple.com/documentation/fileprovider/nonre... and, while I am absolutely certain someone will point out things that FUSE has that the macOS version doesn't, I'm only saying that, just like Windows, macOS has its own way of doing things and thus very little incentive to implement someone else's vision
- tambourine_man 3y ago
  Apple goes to great lengths to increase security while annoying its users. Containers could help in that regard. Having an app (and GUI too) run in a container of sorts could avoid the countless dialog boxes we’ve grown accustomed to. Regarding FUSE, there have been many implementations for macOS, just not very good ones. I don’t see a fundamental limitation on XNU, Darwin, Mach. I haven’t though of File Provider in this way, but maybe it’s a good solution. Apple seems reasonably committed to it so that’s encouraging.

evgpbfhnr 3y ago

> stale file handles... This is still a problem and I’m not sure how to fix it. I don’t understand how real NFS servers do this, maybe they just have a really big cache? While some implementations probably cache things, most filesystems have an underlying file handle which allow opening the file directly (see name_to_handle_at(2)/open_by_handle_at(2) on linux, but other systems also have something similar it's just not necessarily exposed to userspace) NFS servers can then just expose this along with some internal stuff (like export id for internal permissions checks etc) This can lead to some interesting misfeature if a user can guess the handles -- on most filesystems it'll just be something like the inode number and some extra stuff that depending on the filesystem can be contiguously allocated, so +1 can work -- in that open by handle at won't check that the user has access to the full path: so a malign user could access some/subdir/file if file itself is open even if subdir isn't accessible. Thanksfully most recent filesystems allocate inodes pseudo-randomly, so this won't work well for most people. (In this particular case, you could have commit id + a big hash table with paths inside the repo? Most recently large repos would fit in memory for one single revision and layout doesn't change all that much between commits, so that could work fairly well a bit past linux kernel size. For humongous repos some more tricks might be needed though)

EdSchouten 3y ago
Worth mentioning that if you use NFSv4, file handles can be up to 128 bytes in size. That’s big enough that you can sometimes get away with dumping the entire file state in it. For Buildbarn (a distributed build cluster for Bazel) I also wrote an NFSv4 server in Go to act as a lazy-loading file system for input files. As the remote execution protocol uses Content Addressable Storage, I ended up dumping the entire SHA-256, size and permission bits of files in the NFSv4 file handle. This allows me to reconstruct files as needed. Some notes I made at the time (which are written with Buildbarn knowledge in mind): https://github.com/buildbarn/bb-adrs/blob/master/0009-nfsv4.md https://github.com/buildbarn/bb-adrs/blob/master/0009-nfsv4....
cesarb 3y ago
> most filesystems have an underlying file handle which allow opening the file directly (see name_to_handle_at(2)/open_by_handle_at(2) on linux, but other systems also have something similar it's just not necessarily exposed to userspace) On Windows, access to this is controlled by the confusingly named "Bypass traverse checking" (aka SeChangeNotifyPrivilege) permission (see for instance https://techcommunity.microsoft.com/t5/windows-blog-archive/the-bypass-traverse-checking-or-is-it-the-change-notify/ba-p/723428 https://techcommunity.microsoft.com/t5/windows-blog-archive/... for some information about it; I recall once reading an article from either Raymond Chen or Larry Osterman explaining the naming of that permission, but can't find it at the moment).

skadamat 3y ago

Big fans of NFS, we built a Rust library that implemented NFS for a similar same use case (mounting Git repos): https://about.xethub.com/blog/nfs-fuse-why-we-built-nfs-server-rust https://about.xethub.com/blog/nfs-fuse-why-we-built-nfs-serv...

orf 3y ago

It’s a shame this is NFSv3. It’s my understanding that NFSv4 is significantly faster to access due to pipelining requests?

avgcorrection 3y ago

> The main reason I wanted to make this was to give folks some intuition for how git works under the hood. After all, git commits really are very similar to folders – every Git commit contains a directory listing of the files in it, and that directory can have subdirectories, etc. > It’s just that git commits aren’t actually implemented as folders to save disk space. Okay. So what. People are used to archive formats. > git worktree also lets you have multiple branches checked out at the same time, but to me it feels weird to set up an entire worktree just to look at 1 file. But one directory per commit is less weird.