4 ms·

> They're also not diffable. I think that really is a problem with the diff tools, not the format itself. This is why both Mercurial and git allow you to pick

by jordigh 12y ago

> They're also not diffable. I think that really is a problem with the diff tools, not the format itself. This is why both Mercurial and git allow you to pick special diff and merge tools per filename extension.

WallWextra 12y ago
A bit-exact differ for, say, jpegs, probably wouldn't work all that well. Even for losslessly compressed formats, it's very complex and I'm not sure how small the diffs would be.
- Thrymr 12y ago
  > A bit-exact differ for, say, jpegs, probably wouldn't work all that well. Depends. For some uses (e.g. a change in one corner of the file, encoded by the same tool with the same parameters), the diff of JPEG would be fine, differences will be restricted to the 8x8 pixel blocks that were touched by the change. Other changes (e.g., changing encoding quality, trimming a row of pixels off the edge of an image) would lead to more complex diffs.
  - WallWextra 12y ago
    What about compressing the same image with two different encoders? Or with different zlib settings? Even if the diff after lossless decompression is small, you have to somehow reconstruct the decisions that the compressor made on the target file.
  - deleted 12y ago
    [deleted]
- derefr 12y ago
  I think the problem here is that people are jumping from "git should be able to store binary assets" to "git should be able to store binary compilation objects." You would never check a .o file into your SCM; an SCM, as the name implies, is for managing source files, not object files. A PSD or a DOCX is also a source file, despite being binary: they're representations of the work-in-progress itself, containing enough data to let you resume editing the document. A JPEG, on the other hand, is a compiled object—something you export from your image editor, not something you open and edit and save again. (Unless you're intentionally going for that recompressed-shitpost look, I suppose.) When you pull down a source repo—a thing you edit—you should expect to get source. That applies to both your text/code assets, and your image/binary assets. If, on the other hand, you need some assets to just sit there and be consumed by your project, then those aren't source, and so don't belong in your source repo. Those are likely dependencies, which can be resolved to (a triggered compilation of) the relevant source repo for those assets, or which can be resolved to a linear(!)-versioned binary package containing the compiled objects for those assets. Which is all to say: PSDs, given an appropriate diff tool, could go in git. Final, "product" JPEGs, on the other hand? Those should be sitting in a gem (or equivalent), which was the tagged continuous-integration result of pointing a buildbot (exportbot?) at the relevant source repo full of PSDs. When you build your project, that gem gets pulled down (hopefully from your own private CDN) and suddenly you have some JPEGs, just like suddenly you have some native module .so files.
  - leni536 12y ago
    Someone could write an image editor where the project format is a directory with the source jpeg or png assets and a makefile which generates the resulting image with imagemagick (or other) calls. The makefile could be easily diffable and mergeable and people wouldn't touch the binary assets (or at least wouldn't expect to diff or merge them). Brushes wouldn't be easy though. This could be handled by trivial svg or ps paths which would get rendered in the make process. This way even brush strokes could be diffable. I wounder if it would be possible to convert existing PSD files or Gimp project files into such a format.
infogulch 12y ago
Most binary assets are compressed. Diff tools can't work with compressed assets; you'll have to decompress before diffing. Sometimes they also include checksum information which would invalidate any attempt to merge. How far do you take the decompression? For raster images, you'll probably have to decompress all the way to bitmap because the same image could have multiple completely different binary representations in a format like png. How do you diff changes? If we have a raster image and one person changes one thing by a small amount, say increases brightness 1%, this could alter every pixel of the image! How would you detect that change and interleave it with something like a contrast adjustment of 1% that could also change every pixel? Sure, the merger would still have to choose which adjustment goes first if the changes aren't independent, but how would they know that's what changed? I.e. how would the diff tool know that the changes are "brightness +1%" and "contrast +1%" and not some other arbitrary number of adjustments?
- jordigh 12y ago
  I don't think the problem is trivial, but I don't think it's hopeless either. If we can measure a person's pulse and mood with a camera pointed at their face, I'm sure we can come up with a tool that can approximate semantically meaningful diffs of artwork. For image formats like xcf that store parts of the image or the editing history independently, this problem becomes even more tractable.
  - WallWextra 12y ago
    It's very tractable if you don't insist on the diff reconstructing the target file byte-for-byte. But this would require pretty invasive changes in the version control system, no? If you change the top-left pixel of a PNG, for example, between the intra prediction and the DEFLATE compression, the new file can be totally different, and to reconstruct it you either hope the destination is using the exact same libpng with the exact same settings, or you have to find a space-efficient way to write down all the arbitrary encoding decisions the format allows.
- rakoo 12y ago
  I don't think those are your scm's business. git is the stupid content tracker, it tracks whatever you push into it. If I were to make a contrived analogy, how do you know how to diff random arrays of bytes ? Where do you start, where do you stop ? How do you know that "\n" or "\r\n" is some kind of delimiter ? You put that knowledge in "diff" and in your editor, and git stores the raw array of bytes. It's the same with binary content: git doesn't care that you don't deal with UTF-8 characters, it doesn't care that it isn't bounded by newline characters. If you take things this way, you start to understand that the "diff" tool you use must be appropriate to the content you have, and it's not the scm's business. Now, how exactly would a diff work for images, I have absolutely no idea.

by jordigh 12y ago

WallWextra 12y ago

A bit-exact differ for, say, jpegs, probably wouldn't work all that well. Even for losslessly compressed formats, it's very complex and I'm not sure how small the diffs would be.

Thrymr 12y ago
> A bit-exact differ for, say, jpegs, probably wouldn't work all that well. Depends. For some uses (e.g. a change in one corner of the file, encoded by the same tool with the same parameters), the diff of JPEG would be fine, differences will be restricted to the 8x8 pixel blocks that were touched by the change. Other changes (e.g., changing encoding quality, trimming a row of pixels off the edge of an image) would lead to more complex diffs.
- WallWextra 12y ago
  What about compressing the same image with two different encoders? Or with different zlib settings? Even if the diff after lossless decompression is small, you have to somehow reconstruct the decisions that the compressor made on the target file.
- deleted 12y ago
  [deleted]
derefr 12y ago
I think the problem here is that people are jumping from "git should be able to store binary assets" to "git should be able to store binary compilation objects." You would never check a .o file into your SCM; an SCM, as the name implies, is for managing source files, not object files. A PSD or a DOCX is also a source file, despite being binary: they're representations of the work-in-progress itself, containing enough data to let you resume editing the document. A JPEG, on the other hand, is a compiled object—something you export from your image editor, not something you open and edit and save again. (Unless you're intentionally going for that recompressed-shitpost look, I suppose.) When you pull down a source repo—a thing you edit—you should expect to get source. That applies to both your text/code assets, and your image/binary assets. If, on the other hand, you need some assets to just sit there and be consumed by your project, then those aren't source, and so don't belong in your source repo. Those are likely dependencies, which can be resolved to (a triggered compilation of) the relevant source repo for those assets, or which can be resolved to a linear(!)-versioned binary package containing the compiled objects for those assets. Which is all to say: PSDs, given an appropriate diff tool, could go in git. Final, "product" JPEGs, on the other hand? Those should be sitting in a gem (or equivalent), which was the tagged continuous-integration result of pointing a buildbot (exportbot?) at the relevant source repo full of PSDs. When you build your project, that gem gets pulled down (hopefully from your own private CDN) and suddenly you have some JPEGs, just like suddenly you have some native module .so files.
- leni536 12y ago
  Someone could write an image editor where the project format is a directory with the source jpeg or png assets and a makefile which generates the resulting image with imagemagick (or other) calls. The makefile could be easily diffable and mergeable and people wouldn't touch the binary assets (or at least wouldn't expect to diff or merge them). Brushes wouldn't be easy though. This could be handled by trivial svg or ps paths which would get rendered in the make process. This way even brush strokes could be diffable. I wounder if it would be possible to convert existing PSD files or Gimp project files into such a format.

infogulch 12y ago

Most binary assets are compressed. Diff tools can't work with compressed assets; you'll have to decompress before diffing. Sometimes they also include checksum information which would invalidate any attempt to merge. How far do you take the decompression? For raster images, you'll probably have to decompress all the way to bitmap because the same image could have multiple completely different binary representations in a format like png. How do you diff changes? If we have a raster image and one person changes one thing by a small amount, say increases brightness 1%, this could alter every pixel of the image! How would you detect that change and interleave it with something like a contrast adjustment of 1% that could also change every pixel? Sure, the merger would still have to choose which adjustment goes first if the changes aren't independent, but how would they know that's what changed? I.e. how would the diff tool know that the changes are "brightness +1%" and "contrast +1%" and not some other arbitrary number of adjustments?

jordigh 12y ago
I don't think the problem is trivial, but I don't think it's hopeless either. If we can measure a person's pulse and mood with a camera pointed at their face, I'm sure we can come up with a tool that can approximate semantically meaningful diffs of artwork. For image formats like xcf that store parts of the image or the editing history independently, this problem becomes even more tractable.
- WallWextra 12y ago
  It's very tractable if you don't insist on the diff reconstructing the target file byte-for-byte. But this would require pretty invasive changes in the version control system, no? If you change the top-left pixel of a PNG, for example, between the intra prediction and the DEFLATE compression, the new file can be totally different, and to reconstruct it you either hope the destination is using the exact same libpng with the exact same settings, or you have to find a space-efficient way to write down all the arbitrary encoding decisions the format allows.
rakoo 12y ago
I don't think those are your scm's business. git is the stupid content tracker, it tracks whatever you push into it. If I were to make a contrived analogy, how do you know how to diff random arrays of bytes ? Where do you start, where do you stop ? How do you know that "\n" or "\r\n" is some kind of delimiter ? You put that knowledge in "diff" and in your editor, and git stores the raw array of bytes. It's the same with binary content: git doesn't care that you don't deal with UTF-8 characters, it doesn't care that it isn't bounded by newline characters. If you take things this way, you start to understand that the "diff" tool you use must be appropriate to the content you have, and it's not the scm's business. Now, how exactly would a diff work for images, I have absolutely no idea.