6 ms·

Show HN: A versioned filesystem inspired by Git

115 points by artagnon 14y ago

jblz 14y ago
This is tremendously cool! Any plans for MacOS (or other *nix) support?
- artagnon 14y ago
  There's a FUSE for OS X, so a port should be possible (should be possible for other *nix too, as long as they have FUSE). I currently don't have access to an OS X machine, so no immediate plans.
mosburger 14y ago
Nice! Reminds me a bit of a better version of VMS's versioned filesystem. Also: I'm old enough to have used VMS's filesystem. :(
- ScottBurson 14y ago
  Chill out. I'm old enough to have used ITS, which introduced versioning(†). († Apparently: http://en.wikipedia.org/wiki/Versioning_file_system http://en.wikipedia.org/wiki/Versioning_file_system)
- josephscott 14y ago
  Agreed, VMS filesystem was the first thing that came to mind for me as well. Seems rather sad that in 2012 we don't have that as a standard feature like we did back then. For the record, you don't have to be all that old to have used VMS's filesystem :-)
- Hominem 14y ago
  My company still uses VMS specifically for the versioned filesystem.
- ylem 14y ago
  We just retired our old vax a couple of years ago :>
al3xbio 14y ago
Nice. Why not use libgit2 (https://github.com/libgit2/libgit2 https://github.com/libgit2/libgit2)?
- artagnon 14y ago
  It's a bit of an overkill for my purposes- I've only copied out (and modified) a little of the packfile/ packfile index generation code from git.git. Plus, dealing with the actual code is a great learning exercise :)
nicholassmith 14y ago
This looks pretty good. I had a look at doing something similar (but wayyyyy smaller scale than full FS), and gave up as there was many dragons. Nice to see someone pull it off.
akg 14y ago
Congrats on a tremendous technical feat! Any advantage of using a full-scale FS over something like git-annex (http://git-annex.branchable.com/ http://git-annex.branchable.com/)?
- artagnon 14y ago
  The seamless workflow that only a full-blown filesystem can offer- new revisions are created automatically when files are edited and saved.
  - fuzzix 14y ago
    I like it. It reminds me of VMS's versioning. I implemented something like VMS style versioning with Fuse and Perl for a talk/demo a couple of years ago - it simply created a new version on open() if any writeable flags were passed - not robust, but functional enough to demo. edit Possessed by VMS, not multiple VMSes.
  - p9idf 14y ago
    Does your file system create revisions of the entire tree, or only revisions of individual files within the tree as VMS and RCS do?† Because there are some problems with the latter approach. One problem described by Geoff Collyer (the C News guy) is that "version numbers of related files are not kept in synch, so you can't just cp *';6' /tmp/compile".†† † I tried answering that question myself, but your program kept crashing. †† http://9fans.net/archive/2001/11/759 http://9fans.net/archive/2001/11/759
    - artagnon 14y ago
      Revisions of individual files only. Yeah, you can't do that kind of sorcery with phoenixfs. † Fixed. Sorry about that.
  - spartango 14y ago
    You should be able to achieve this with Filesystem events (inotify/kqueue/filesystemwatcher). Just another way of doing things. I'm not the biggest fan of FUSE filesystems(until highly-tuned, ala NTFS-3G), but they certainly do work.
icebraining 14y ago
Extremely interesting. I wonder: when does it "commit"/create a new rev?
- artagnon 14y ago
  When you modify a file and save it.
  - rix0r 14y ago
    So on close() ?
    - artagnon 14y ago
      Yes. The event is called `release`: see phoenixfs_release() in fuse.c:374.
      - ferrouswheel 14y ago
        What about when a file is fsynced?
prezjordan 14y ago
Wow, awesome project!
adam-a 14y ago
Sounds a bit like ZFS snapshots[1]. I always wished that would catch on a bit more, seems like it would be very useful. [1]: http://en.wikipedia.org/wiki/ZFS#Snapshots_and_clones http://en.wikipedia.org/wiki/ZFS#Snapshots_and_clones
- justinmk 14y ago
  And btrfs[1] of course. There's also git bup[2] and git-annex[3]. [1] http://www.dedoimedo.com/computers/btrfs-snapshots.html http://www.dedoimedo.com/computers/btrfs-snapshots.html [2] https://github.com/apenwarr/bup https://github.com/apenwarr/bup [3] http://git-annex.branchable.com/ http://git-annex.branchable.com/
jeremyjh 14y ago
Very cool. I would like to have some automatic revisions for example of my /etc tree.
- RobotCaleb 14y ago
  There's a tool that will commit changes to /etc before every apt-get install. This might be what I'm thinking of: http://joeyh.name/code/etckeeper/ http://joeyh.name/code/etckeeper/
tolmasky 14y ago
OK, first off -- awesome! Secondly, I have a number of questions that probably have to do with the fact that I'm unfamiliar with a lot of this stuff. 1. Would it be possible to build this fully on top of git (perhaps using libgit2)? I ask, because the "holy" grail for me is finding some magic way for my designers to use git without knowing it. Right now my designers (and I think most non-programmer people), love to use Dropbox. Dropbox has a number of issues as a shared project tool in my perspective (not the least of which being that you are only allowed ONE dropbox account per computer, so you start having a shared folder mess and your dropbox begins to balloon in size on every computer with that account). It appears that something like this could look like Dropbox to everyone disinterested in vcs and act like Git to everyone else who cares. 2. The other reason for libgit2 is to ideally also git-push after every edit/save, to something like github. 3. Max OS X support please :)
- martythemaniak 14y ago
  Check out Pixelapse, which is kind of like github for designers: http://www.pixelapse.com/ http://www.pixelapse.com/
  - tolmasky 14y ago
    Yeah I asked for an invite ages ago, haven't heard back. if they ever let me in it might do the trick. Separately, actual github for designers would be cooler because I could: 1. Have an "artwork" git repo which my designers interact with in this way, but which I include as a submodule to a larger git repo for my programmers. 2. Use the same bug tracking for art as I do programming 3. Backup the git repo because its an open protocol vs. some proprietary thing. Again though, I'm lazy and don't have time to write project management tools for myself so if these guys ever got around to sending me an invite I'd probably just settle.
    - debacle 14y ago
      The windows github client is relatively idiot proof. I don't know if they offer something similar for Macs.
gcr 14y ago
Can this handle large binary files? That's been my main problem with storing backups of my servers in git, and it's one of the strong-ish points of git-annex. There seems to be nothing inherently limiting in Git's repository design. Interested parties can look into bup, for example, which stores large binary files in git's packfile format. https://github.com/apenwarr/bup https://github.com/apenwarr/bup
- artagnon 14y ago
  Yes; I don't see why not. I haven't stress-tested this enough to tell for certain though.
tensor 14y ago
This is relevant: http://www.hpl.hp.com/personal/alistair_veitch/papers/elephant-sosp/efs.pdf http://www.hpl.hp.com/personal/alistair_veitch/papers/elepha... If only half the stuff done by the likes of IBM and HP were to make it into common usage the world would be a very different place.
halayli 14y ago
https://github.com/artagnon/phoenixfs/blob/master/delta.c#L489 https://github.com/artagnon/phoenixfs/blob/master/delta.c#L4... https://github.com/artagnon/phoenixfs/blob/master/fuse.c#L594 https://github.com/artagnon/phoenixfs/blob/master/fuse.c#L59... Missed malloc check. https://github.com/artagnon/phoenixfs/blob/master/fuse.c#L82 https://github.com/artagnon/phoenixfs/blob/master/fuse.c#L82 system calls like mkdir can fail. Better to check return values of such calls.
- voltagex_ 14y ago
  Might want to post that to the issue tracker
jarvuschris 14y ago
I'm working on a similar concept which provide copy-on-write over HTTP as well: http://emr.ge http://emr.ge

115 points by artagnon 14y ago

jblz 14y ago

This is tremendously cool! Any plans for MacOS (or other *nix) support?

artagnon 14y ago
There's a FUSE for OS X, so a port should be possible (should be possible for other *nix too, as long as they have FUSE). I currently don't have access to an OS X machine, so no immediate plans.

mosburger 14y ago

Nice! Reminds me a bit of a better version of VMS's versioned filesystem. Also: I'm old enough to have used VMS's filesystem. :(

ScottBurson 14y ago
Chill out. I'm old enough to have used ITS, which introduced versioning(†). († Apparently: http://en.wikipedia.org/wiki/Versioning_file_system http://en.wikipedia.org/wiki/Versioning_file_system)
josephscott 14y ago
Agreed, VMS filesystem was the first thing that came to mind for me as well. Seems rather sad that in 2012 we don't have that as a standard feature like we did back then. For the record, you don't have to be all that old to have used VMS's filesystem :-)
Hominem 14y ago
My company still uses VMS specifically for the versioned filesystem.
ylem 14y ago
We just retired our old vax a couple of years ago :>

al3xbio 14y ago

Nice. Why not use libgit2 (https://github.com/libgit2/libgit2 https://github.com/libgit2/libgit2)?

artagnon 14y ago
It's a bit of an overkill for my purposes- I've only copied out (and modified) a little of the packfile/ packfile index generation code from git.git. Plus, dealing with the actual code is a great learning exercise :)

nicholassmith 14y ago

This looks pretty good. I had a look at doing something similar (but wayyyyy smaller scale than full FS), and gave up as there was many dragons. Nice to see someone pull it off.

akg 14y ago

Congrats on a tremendous technical feat! Any advantage of using a full-scale FS over something like git-annex (http://git-annex.branchable.com/ http://git-annex.branchable.com/)?

artagnon 14y ago
The seamless workflow that only a full-blown filesystem can offer- new revisions are created automatically when files are edited and saved.
- fuzzix 14y ago
  I like it. It reminds me of VMS's versioning. I implemented something like VMS style versioning with Fuse and Perl for a talk/demo a couple of years ago - it simply created a new version on open() if any writeable flags were passed - not robust, but functional enough to demo. edit Possessed by VMS, not multiple VMSes.
- p9idf 14y ago
  Does your file system create revisions of the entire tree, or only revisions of individual files within the tree as VMS and RCS do?† Because there are some problems with the latter approach. One problem described by Geoff Collyer (the C News guy) is that "version numbers of related files are not kept in synch, so you can't just cp *';6' /tmp/compile".†† † I tried answering that question myself, but your program kept crashing. †† http://9fans.net/archive/2001/11/759 http://9fans.net/archive/2001/11/759
  - artagnon 14y ago
    Revisions of individual files only. Yeah, you can't do that kind of sorcery with phoenixfs. † Fixed. Sorry about that.
- spartango 14y ago
  You should be able to achieve this with Filesystem events (inotify/kqueue/filesystemwatcher). Just another way of doing things. I'm not the biggest fan of FUSE filesystems(until highly-tuned, ala NTFS-3G), but they certainly do work.

icebraining 14y ago

Extremely interesting. I wonder: when does it "commit"/create a new rev?

artagnon 14y ago
When you modify a file and save it.
- rix0r 14y ago
  So on close() ?
  - artagnon 14y ago
    Yes. The event is called `release`: see phoenixfs_release() in fuse.c:374.
    - ferrouswheel 14y ago
      What about when a file is fsynced?

prezjordan 14y ago

Wow, awesome project!

adam-a 14y ago

Sounds a bit like ZFS snapshots[1]. I always wished that would catch on a bit more, seems like it would be very useful. [1]: http://en.wikipedia.org/wiki/ZFS#Snapshots_and_clones http://en.wikipedia.org/wiki/ZFS#Snapshots_and_clones

justinmk 14y ago
And btrfs[1] of course. There's also git bup[2] and git-annex[3]. [1] http://www.dedoimedo.com/computers/btrfs-snapshots.html http://www.dedoimedo.com/computers/btrfs-snapshots.html [2] https://github.com/apenwarr/bup https://github.com/apenwarr/bup [3] http://git-annex.branchable.com/ http://git-annex.branchable.com/

jeremyjh 14y ago

Very cool. I would like to have some automatic revisions for example of my /etc tree.

RobotCaleb 14y ago
There's a tool that will commit changes to /etc before every apt-get install. This might be what I'm thinking of: http://joeyh.name/code/etckeeper/ http://joeyh.name/code/etckeeper/

tolmasky 14y ago

OK, first off -- awesome! Secondly, I have a number of questions that probably have to do with the fact that I'm unfamiliar with a lot of this stuff. 1. Would it be possible to build this fully on top of git (perhaps using libgit2)? I ask, because the "holy" grail for me is finding some magic way for my designers to use git without knowing it. Right now my designers (and I think most non-programmer people), love to use Dropbox. Dropbox has a number of issues as a shared project tool in my perspective (not the least of which being that you are only allowed ONE dropbox account per computer, so you start having a shared folder mess and your dropbox begins to balloon in size on every computer with that account). It appears that something like this could look like Dropbox to everyone disinterested in vcs and act like Git to everyone else who cares. 2. The other reason for libgit2 is to ideally also git-push after every edit/save, to something like github. 3. Max OS X support please :)

martythemaniak 14y ago
Check out Pixelapse, which is kind of like github for designers: http://www.pixelapse.com/ http://www.pixelapse.com/
- tolmasky 14y ago
  Yeah I asked for an invite ages ago, haven't heard back. if they ever let me in it might do the trick. Separately, actual github for designers would be cooler because I could: 1. Have an "artwork" git repo which my designers interact with in this way, but which I include as a submodule to a larger git repo for my programmers. 2. Use the same bug tracking for art as I do programming 3. Backup the git repo because its an open protocol vs. some proprietary thing. Again though, I'm lazy and don't have time to write project management tools for myself so if these guys ever got around to sending me an invite I'd probably just settle.
  - debacle 14y ago
    The windows github client is relatively idiot proof. I don't know if they offer something similar for Macs.

gcr 14y ago

Can this handle large binary files? That's been my main problem with storing backups of my servers in git, and it's one of the strong-ish points of git-annex. There seems to be nothing inherently limiting in Git's repository design. Interested parties can look into bup, for example, which stores large binary files in git's packfile format. https://github.com/apenwarr/bup https://github.com/apenwarr/bup

artagnon 14y ago
Yes; I don't see why not. I haven't stress-tested this enough to tell for certain though.

tensor 14y ago

This is relevant: http://www.hpl.hp.com/personal/alistair_veitch/papers/elephant-sosp/efs.pdf http://www.hpl.hp.com/personal/alistair_veitch/papers/elepha... If only half the stuff done by the likes of IBM and HP were to make it into common usage the world would be a very different place.

halayli 14y ago

https://github.com/artagnon/phoenixfs/blob/master/delta.c#L489 https://github.com/artagnon/phoenixfs/blob/master/delta.c#L4... https://github.com/artagnon/phoenixfs/blob/master/fuse.c#L594 https://github.com/artagnon/phoenixfs/blob/master/fuse.c#L59... Missed malloc check. https://github.com/artagnon/phoenixfs/blob/master/fuse.c#L82 https://github.com/artagnon/phoenixfs/blob/master/fuse.c#L82 system calls like mkdir can fail. Better to check return values of such calls.

voltagex_ 14y ago
Might want to post that to the issue tracker

jarvuschris 14y ago

I'm working on a similar concept which provide copy-on-write over HTTP as well: http://emr.ge http://emr.ge