4 ms·

I think this kind of file system would do well to power platforms that AREN'T for programmers. While this may not seem useful for programmers (because they are

I think this kind of file system would do well to power platforms that AREN'T for programmers. While this may not seem useful for programmers (because they are so involved in git usage in the first place), I think others might benefit I've had the idea to make an interative design/architecture plugin for programs like photoshop/3dmax/etc rolling around in my head for a while now. Git is a perfect way to store/save the progress people make in those programs (as files change over time, or as they save), and gitfs seems like it would be the fs to run on the backing server. Instant branching (like when an artist decides to riff on a new idea), undo/rollback, progress tweens/reports.

TranquilMarmot 12y ago
It's my understanding that Git is pretty bad for storing binary files that change a lot since it has to keep a copy of every version of that file (whereas with plaintext it can just store the diffs). Now, maybe that's what you would want, but that repo is going to get very big very fast. Pulls won't be so bad if your machine already has all the commits, but cloning would be a nightmare. And it'll be taking up a lot of space on your git server.
- vtemian 12y ago
  You are right about large binary files and gitfs currently has an option limiting the maximum file size.
- jimktrains2 12y ago
  > (whereas with plaintext it can just store the diffs) git stores full blobs, not deltas.
  - rakoo 12y ago
    I guess it depends how deep you dive... It is true that on the surface, git gives you the user access to full blobs only and calculates the difference every time you access blobs. But when you go in packfiles the content is actually diff'ed because it compresses so well. In the context of the discussion, since we're interested in the on-disk format, it's more accurate to say that git will try to diff binary blobs, fail at that and so store the full content of blobs.
    - jimktrains2 12y ago
      Not everything is in packfiles right away.
- peff 12y ago
  It's not exactly plaintext versus binary. Git "stores" a copy of every file, no matter the content. But it also does "delta compression" between objects when you run "git gc"; it tries to find binary diffs between objects that are similar. So the poor performance comes from files that don't "delta" well. These tend to be things which are compressed or encrypted, where a small semantic change can cascade into a lot of bytes changing. Of course, binary formats often do both of those things. And the larger the files, the more painful it is when it happens.
  - e12e 12y ago
    Incidentally, the new office formats (both the "open" and the open one) are zip-files with (among other things) xml-documents. I saw someone a while back recommending storing them uncompressed when working with VCS -- gives pretty useful diffs with no extra work (I forget if they recommended extracting, or just storing without compression).
- hardwaresofton 12y ago
  Very good point - however I think this is just an implementation detail of Git as it stands today. Also, if the person is working with SVG (for example), then it's less of a problem. Also, given the cheapness of disk, I don't think it would be a limiting cost. And since git is open source, if I were to actually make this thing, it would definitely incentivize me (or others) to make git less bad for storing binary files :)
- agumonkey 12y ago
  I remember trying candidely git and svn (around 2008 so..) on a webdev project involving many images including some xMB PSD files. Surprisingly the git repo grew sublinearly, while svn was absuperlinear. With git packing, the full repo with a bit of history (mostly adding some php and a few images) ended up smaller than the original windows native folder. Version control and compression; nice :)
csmattryder 12y ago
> I've had the idea to make an interative design/architecture plugin for programs like photoshop/3dmax/etc Hah, same idea here. I used to live with 3d artists, and I asked "How do you live without Git?!", they said they version their files into milestones (e.g. environment-milestone1.3ds) and upload them all to Dropbox. I think Git wouldn't gain as much traction as you'd expect, there's just easier (for non-programmers) alternatives with a large userbase.
- hardwaresofton 12y ago
  Yeah, was watching some designer friends of mine, and they have all this back and forth with clients, and keep immaculate notes with every stage of the transformation (from concept to finished design), I was wondering why in the world it wasn't automated. Yeah, I'm not really planning on introducing them to git, more using git behind the scenes to get all those good qualities (undos, infinite history, instant backup) that someone might think was hard (that didn't know git)
  - e12e 12y ago
    There exist various solutions -- the technology is called Digital Asset Managements (DAM). There are some open source/Free ones as well. But none that are really good AFAIK. You generally have the option of either using lots of data and bandwidth (think: storing rendered 4/8K jpeg2000 frames of animations, for each new version/cut/render you want to keep) -- or tight integration with the various software involved (ie: only keep latest "render" -- and a track of changes/a "script/log" that can be used to render/generate all the missing versions). I'd think in general 3d meshes shouldn't be too hard -- but good luck finding someone doing media that don't use some big bitmaps somewhere (textures, backgrounds, etc).
jawngee 12y ago
Git is awful for photoshop. This is also a pretty crowded market place with pixelapse, layervault, etc.
- hardwaresofton 12y ago
  A little more detail on the first statement would be great (outside of what people have posted already). And yeah, I know it's been done before -- but that only proves that the market is there, and more importantly investors (notably YC) think it's worth it to pursue. I doubt I could do as great a job as some of these products and their engineering teams, but then again, I don't know that I have to. I think it'd be a fun open source project.
emodendroket 12y ago
I don't know, judging by all the gnashing of the teeth I get when I answer questions with something like "you should actually learn how to use git instead of banging your head against the wall with this wrapper for it" developers could use it too.

TranquilMarmot 12y ago

It's my understanding that Git is pretty bad for storing binary files that change a lot since it has to keep a copy of every version of that file (whereas with plaintext it can just store the diffs). Now, maybe that's what you would want, but that repo is going to get very big very fast. Pulls won't be so bad if your machine already has all the commits, but cloning would be a nightmare. And it'll be taking up a lot of space on your git server.

vtemian 12y ago
You are right about large binary files and gitfs currently has an option limiting the maximum file size.
jimktrains2 12y ago
> (whereas with plaintext it can just store the diffs) git stores full blobs, not deltas.
- rakoo 12y ago
  I guess it depends how deep you dive... It is true that on the surface, git gives you the user access to full blobs only and calculates the difference every time you access blobs. But when you go in packfiles the content is actually diff'ed because it compresses so well. In the context of the discussion, since we're interested in the on-disk format, it's more accurate to say that git will try to diff binary blobs, fail at that and so store the full content of blobs.
  - jimktrains2 12y ago
    Not everything is in packfiles right away.
peff 12y ago
It's not exactly plaintext versus binary. Git "stores" a copy of every file, no matter the content. But it also does "delta compression" between objects when you run "git gc"; it tries to find binary diffs between objects that are similar. So the poor performance comes from files that don't "delta" well. These tend to be things which are compressed or encrypted, where a small semantic change can cascade into a lot of bytes changing. Of course, binary formats often do both of those things. And the larger the files, the more painful it is when it happens.
- e12e 12y ago
  Incidentally, the new office formats (both the "open" and the open one) are zip-files with (among other things) xml-documents. I saw someone a while back recommending storing them uncompressed when working with VCS -- gives pretty useful diffs with no extra work (I forget if they recommended extracting, or just storing without compression).
hardwaresofton 12y ago
Very good point - however I think this is just an implementation detail of Git as it stands today. Also, if the person is working with SVG (for example), then it's less of a problem. Also, given the cheapness of disk, I don't think it would be a limiting cost. And since git is open source, if I were to actually make this thing, it would definitely incentivize me (or others) to make git less bad for storing binary files :)
agumonkey 12y ago
I remember trying candidely git and svn (around 2008 so..) on a webdev project involving many images including some xMB PSD files. Surprisingly the git repo grew sublinearly, while svn was absuperlinear. With git packing, the full repo with a bit of history (mostly adding some php and a few images) ended up smaller than the original windows native folder. Version control and compression; nice :)

csmattryder 12y ago

> I've had the idea to make an interative design/architecture plugin for programs like photoshop/3dmax/etc Hah, same idea here. I used to live with 3d artists, and I asked "How do you live without Git?!", they said they version their files into milestones (e.g. environment-milestone1.3ds) and upload them all to Dropbox. I think Git wouldn't gain as much traction as you'd expect, there's just easier (for non-programmers) alternatives with a large userbase.

hardwaresofton 12y ago
Yeah, was watching some designer friends of mine, and they have all this back and forth with clients, and keep immaculate notes with every stage of the transformation (from concept to finished design), I was wondering why in the world it wasn't automated. Yeah, I'm not really planning on introducing them to git, more using git behind the scenes to get all those good qualities (undos, infinite history, instant backup) that someone might think was hard (that didn't know git)
- e12e 12y ago
  There exist various solutions -- the technology is called Digital Asset Managements (DAM). There are some open source/Free ones as well. But none that are really good AFAIK. You generally have the option of either using lots of data and bandwidth (think: storing rendered 4/8K jpeg2000 frames of animations, for each new version/cut/render you want to keep) -- or tight integration with the various software involved (ie: only keep latest "render" -- and a track of changes/a "script/log" that can be used to render/generate all the missing versions). I'd think in general 3d meshes shouldn't be too hard -- but good luck finding someone doing media that don't use some big bitmaps somewhere (textures, backgrounds, etc).

jawngee 12y ago

Git is awful for photoshop. This is also a pretty crowded market place with pixelapse, layervault, etc.

hardwaresofton 12y ago
A little more detail on the first statement would be great (outside of what people have posted already). And yeah, I know it's been done before -- but that only proves that the market is there, and more importantly investors (notably YC) think it's worth it to pursue. I doubt I could do as great a job as some of these products and their engineering teams, but then again, I don't know that I have to. I think it'd be a fun open source project.

emodendroket 12y ago

I don't know, judging by all the gnashing of the teeth I get when I answer questions with something like "you should actually learn how to use git instead of banging your head against the wall with this wrapper for it" developers could use it too.