6 ms·

This looks like it misses the mark a bit. As anyone who's worked on project with large binary files(the docs assume PSDs) you need to be able to lock unmergeab

This looks like it misses the mark a bit. As anyone who's worked on project with large binary files(the docs assume PSDs) you need to be able to lock unmergeable binary assets. Otherwise you get two people touching the same file and someone has to destroy their changes. That never makes anyone happy. It's also unseen how good the disk performance is. These two areas are the reason why Perforce is still my go-to solution for large binary files.

jordigh 12y ago
Mercurial has largefiles and locking too: http://mercurial.selenic.com/wiki/LargefilesExtension http://mercurial.selenic.com/wiki/LargefilesExtension http://mercurial.selenic.com/wiki/LockExtension http://mercurial.selenic.com/wiki/LockExtension Like other people have noticed, you can have the good parts of a DVCS and the good parts of a CVCS. It doesn't have to be either-or: https://blogs.janestreet.com/centralizing-distributed-version-control-revisited/ https://blogs.janestreet.com/centralizing-distributed-versio... http://bitquabit.com/post/unorthodocs-abandon-your-dvcs-and-return-to-sanity/ http://bitquabit.com/post/unorthodocs-abandon-your-dvcs-and-...
- neandrake 12y ago
  We turned on largefiles extension almost a year ago and have come to regret that decision. The major pain point is with integrations, from Eclipse plugin to most any repository management/hosting solution tend to either have bugs or full-on don't support those repos. With plain mercurial workflow it works in most situations, but still rears its ugly head. The maintainers of mercurial classify it as a "feature of last resort" (Read http://mercurial.selenic.com/wiki/FeaturesOfLastResort http://mercurial.selenic.com/wiki/FeaturesOfLastResort). Switching off largefiles requires rebuilding the repository which rebuilds the entire repository. Orchestrating the migration to a new repository for engineering department is also painful which is why we're stuck for the near future (for example ongoing support for a version that's built from largefiles repo with ongoing feature work in non-largefiles repo). The tooling for mercurial tends to lack behind git's, likely due to git's enormous popularity - so I personally would recommend avoiding largefiles extension.
  - durin42 12y ago
    OOC, what kinds of problems did you see with editor integrations? I work on hg, and I'm wondering if there's anything we can do to make the situation less painful.
    - neandrake 12y ago
      We primarily use Eclipse as our editor developing Java. I think I'm using the correct plugin (MercurialEclipse) but there's another one (HgEclipse, or there used to be) which has often caused confusion. Some of the issues are likely due to our project which is >93k commits at around 1GB for the repo size - I think we have a commit in the history that has 40kb commit message - I believe the developer mixed up some streams/pipes for some reason or other while committing - I'm not sure why we got stuck with it in our history though I suspect this could be the cause of some issues. I can list a few of the issues I still see regularly, but since about a year or so back I've abandoned using the plugin in favor of doing everything manually on the command line, except for viewing history and resolving merge conflicts. My setup: - Eclipse 4.4.2 (configured to run with 2GB memory) - though most all of these issues I've seen since 3.something - MercurialEclipse: http://mercurialeclipse.eclipselabs.org.codespot.com/hg.wiki/update_site/stable http://mercurialeclipse.eclipselabs.org.codespot.com/hg.wiki... - OS X (10.10 and 10.9), mercurial installed through homebrew. 1. During some actions (I believe to be cloning/sharing projects, maybe elsewhere) the the config for mercurial in eclipse pops up thinking it can't find the hg binary, giving an error about bad location. Nothing has moved or changed on the system, simply focusing in the location field of the hg location and back out causes it to re-validate. I've seen this one recently, maybe a few days ago. 2. Some files can't retrieve history or show annotations. I suspect because the files have been through numerous edits over the years and the plugin runs out of memory or hits a timeout when trying to load everything. 3. Occasionally refreshing status throws error and pops up dialog informing me. When this happens the plugin becomes unusable until eclipse is restarted, but popups continue to show up when trying to do any sort of task (such as refreshing project) 4. Using mq patches cause the mercurial status of project to not update regularly (in Navigator pane). I've since moved away from using mq patches in favor of managing multiple local heads. 5. The merge view/window no longer automatically shows up when performing a merge in eclipse. It has to be manually opened, but used to automatically show up - I liken it to performing a search but the search results not automatically showing up unless it's part of your workspace layout already. These are what I can recall off the top of my head, but I've also shied away from using the plugin since quite a while ago. In general the speed is sometimes frustrating to deal with, when wanting to view history of a file. It feels like it's gotten slower after some time that we've had largefiles extension turned on, though it's unfair to assume so because I'm biased.
infogulch 12y ago
That's an interesting point I hadn't considered before. Git, as a distributed vcs, takes the position that everyone can edit all files, and rare conflicts can be managed easily because changes are relatively small and diffable. But these assumptions break down with binary assets. Changes are not small, they typically change entire files at once. They're also not diffable. As a result, conflicts are not rare, they're common, and impossible to resolve for both parties. That's why locking or "checking out" certain files is a needed feature, so changes are ordered strictly linearly -- a graph structure doesn't work.
- jordigh 12y ago
  > They're also not diffable. I think that really is a problem with the diff tools, not the format itself. This is why both Mercurial and git allow you to pick special diff and merge tools per filename extension.
  - WallWextra 12y ago
    A bit-exact differ for, say, jpegs, probably wouldn't work all that well. Even for losslessly compressed formats, it's very complex and I'm not sure how small the diffs would be.
    - Thrymr 12y ago
      > A bit-exact differ for, say, jpegs, probably wouldn't work all that well. Depends. For some uses (e.g. a change in one corner of the file, encoded by the same tool with the same parameters), the diff of JPEG would be fine, differences will be restricted to the 8x8 pixel blocks that were touched by the change. Other changes (e.g., changing encoding quality, trimming a row of pixels off the edge of an image) would lead to more complex diffs.
      - WallWextra 12y ago
        What about compressing the same image with two different encoders? Or with different zlib settings? Even if the diff after lossless decompression is small, you have to somehow reconstruct the decisions that the compressor made on the target file.
cortesoft 12y ago
While the use case you describe isn't solved by this, other use cases are. For example, wanting to easily access versioned, compiled binaries for a revision.
res0nat0r 12y ago
This looks to be competing with AWS CodeCommit too which will allow any size of files to be committed since it is all just backed by S3. http://aws.amazon.com/codecommit/ http://aws.amazon.com/codecommit/
cbhl 12y ago
Anyone who currently stores large files in git already has this problem; git-lfs is strictly better than plain old git in this case. Format-specific diffing and merging tools that are aware of git-lfs would probably help, and those can come later.

jordigh 12y ago

Mercurial has largefiles and locking too: http://mercurial.selenic.com/wiki/LargefilesExtension http://mercurial.selenic.com/wiki/LargefilesExtension http://mercurial.selenic.com/wiki/LockExtension http://mercurial.selenic.com/wiki/LockExtension Like other people have noticed, you can have the good parts of a DVCS and the good parts of a CVCS. It doesn't have to be either-or: https://blogs.janestreet.com/centralizing-distributed-version-control-revisited/ https://blogs.janestreet.com/centralizing-distributed-versio... http://bitquabit.com/post/unorthodocs-abandon-your-dvcs-and-return-to-sanity/ http://bitquabit.com/post/unorthodocs-abandon-your-dvcs-and-...

neandrake 12y ago
We turned on largefiles extension almost a year ago and have come to regret that decision. The major pain point is with integrations, from Eclipse plugin to most any repository management/hosting solution tend to either have bugs or full-on don't support those repos. With plain mercurial workflow it works in most situations, but still rears its ugly head. The maintainers of mercurial classify it as a "feature of last resort" (Read http://mercurial.selenic.com/wiki/FeaturesOfLastResort http://mercurial.selenic.com/wiki/FeaturesOfLastResort). Switching off largefiles requires rebuilding the repository which rebuilds the entire repository. Orchestrating the migration to a new repository for engineering department is also painful which is why we're stuck for the near future (for example ongoing support for a version that's built from largefiles repo with ongoing feature work in non-largefiles repo). The tooling for mercurial tends to lack behind git's, likely due to git's enormous popularity - so I personally would recommend avoiding largefiles extension.
- durin42 12y ago
  OOC, what kinds of problems did you see with editor integrations? I work on hg, and I'm wondering if there's anything we can do to make the situation less painful.
  - neandrake 12y ago
    We primarily use Eclipse as our editor developing Java. I think I'm using the correct plugin (MercurialEclipse) but there's another one (HgEclipse, or there used to be) which has often caused confusion. Some of the issues are likely due to our project which is >93k commits at around 1GB for the repo size - I think we have a commit in the history that has 40kb commit message - I believe the developer mixed up some streams/pipes for some reason or other while committing - I'm not sure why we got stuck with it in our history though I suspect this could be the cause of some issues. I can list a few of the issues I still see regularly, but since about a year or so back I've abandoned using the plugin in favor of doing everything manually on the command line, except for viewing history and resolving merge conflicts. My setup: - Eclipse 4.4.2 (configured to run with 2GB memory) - though most all of these issues I've seen since 3.something - MercurialEclipse: http://mercurialeclipse.eclipselabs.org.codespot.com/hg.wiki/update_site/stable http://mercurialeclipse.eclipselabs.org.codespot.com/hg.wiki... - OS X (10.10 and 10.9), mercurial installed through homebrew. 1. During some actions (I believe to be cloning/sharing projects, maybe elsewhere) the the config for mercurial in eclipse pops up thinking it can't find the hg binary, giving an error about bad location. Nothing has moved or changed on the system, simply focusing in the location field of the hg location and back out causes it to re-validate. I've seen this one recently, maybe a few days ago. 2. Some files can't retrieve history or show annotations. I suspect because the files have been through numerous edits over the years and the plugin runs out of memory or hits a timeout when trying to load everything. 3. Occasionally refreshing status throws error and pops up dialog informing me. When this happens the plugin becomes unusable until eclipse is restarted, but popups continue to show up when trying to do any sort of task (such as refreshing project) 4. Using mq patches cause the mercurial status of project to not update regularly (in Navigator pane). I've since moved away from using mq patches in favor of managing multiple local heads. 5. The merge view/window no longer automatically shows up when performing a merge in eclipse. It has to be manually opened, but used to automatically show up - I liken it to performing a search but the search results not automatically showing up unless it's part of your workspace layout already. These are what I can recall off the top of my head, but I've also shied away from using the plugin since quite a while ago. In general the speed is sometimes frustrating to deal with, when wanting to view history of a file. It feels like it's gotten slower after some time that we've had largefiles extension turned on, though it's unfair to assume so because I'm biased.

infogulch 12y ago

That's an interesting point I hadn't considered before. Git, as a distributed vcs, takes the position that everyone can edit all files, and rare conflicts can be managed easily because changes are relatively small and diffable. But these assumptions break down with binary assets. Changes are not small, they typically change entire files at once. They're also not diffable. As a result, conflicts are not rare, they're common, and impossible to resolve for both parties. That's why locking or "checking out" certain files is a needed feature, so changes are ordered strictly linearly -- a graph structure doesn't work.

jordigh 12y ago
> They're also not diffable. I think that really is a problem with the diff tools, not the format itself. This is why both Mercurial and git allow you to pick special diff and merge tools per filename extension.
- WallWextra 12y ago
  A bit-exact differ for, say, jpegs, probably wouldn't work all that well. Even for losslessly compressed formats, it's very complex and I'm not sure how small the diffs would be.
  - Thrymr 12y ago
    > A bit-exact differ for, say, jpegs, probably wouldn't work all that well. Depends. For some uses (e.g. a change in one corner of the file, encoded by the same tool with the same parameters), the diff of JPEG would be fine, differences will be restricted to the 8x8 pixel blocks that were touched by the change. Other changes (e.g., changing encoding quality, trimming a row of pixels off the edge of an image) would lead to more complex diffs.
    - WallWextra 12y ago
      What about compressing the same image with two different encoders? Or with different zlib settings? Even if the diff after lossless decompression is small, you have to somehow reconstruct the decisions that the compressor made on the target file.

cortesoft 12y ago

While the use case you describe isn't solved by this, other use cases are. For example, wanting to easily access versioned, compiled binaries for a revision.

res0nat0r 12y ago

This looks to be competing with AWS CodeCommit too which will allow any size of files to be committed since it is all just backed by S3. http://aws.amazon.com/codecommit/ http://aws.amazon.com/codecommit/

cbhl 12y ago

Anyone who currently stores large files in git already has this problem; git-lfs is strictly better than plain old git in this case. Format-specific diffing and merging tools that are aware of git-lfs would probably help, and those can come later.