7 ms·

I feel like Mercurial still has a chance to become a real force if it can provably solve some of the real issues with git. It won't do to just have a nicer CLI

by tomlu 12y ago

I feel like Mercurial still has a chance to become a real force if it can provably solve some of the real issues with git. It won't do to just have a nicer CLI since most people are used to git by now. Here are some of git's real problems: * Performance issues with multi-GB git repos * Handling of large binary files * Submodules - Mercurial has subrepos, but I don't know how they compare

colechristensen 12y ago
There is opportunity for a powerful tool to handle the very common bad practice of including giant binary blobs which don't belong in version control. I'd like to see a second class of files which are only checksummed (or timestamped) on 'status', 'diff', 'add' and then binary-diffed on commit for possible compression (or perhaps deduped with checksums of blocks) with features like 'git/hg binary-add somefile.jar' to differentiate.
- mintplant 12y ago
  Check out git-annex: https://git-annex.branchable.com/how_it_works/ https://git-annex.branchable.com/how_it_works/ It operates under a similar principle (committing hashes rather than file contents) but stores the actual file contents elsewhere rather than using a compression/deduplication within the repo itself.
  - colechristensen 12y ago
    I have. It's neat, but it's more about replacing something like Dropbox than it is replacing git (despite being based on git). It also isn't particularly stable, though I have nothing but respect for the developer.
    - AceJohnny2 12y ago
      As I understand it, it became something to replace Dropbox after Joey realized that git-annex's task of distributed storage of large files intersected with the Dropbox use-case, but it didn't start that way, and IMHO doesn't exclude the parent's suggestion. I backed Joey's kickstarter, but to my shame I have yet to give git-annex-assistant a spin.
- tomlu 12y ago
  > There is opportunity for a powerful tool to handle the very common bad practice of including giant binary blobs which don't belong in version control Why do you say it's bad practice? Game assets come to mind, these aren't derived artifacts and you can't build the game without them.
  - wtallis 12y ago
    Usually, you can build the game without them, up to the final packaging stage. Almost any game that is built on an engine can have all of the asset files swapped out without having to recompile anything, and as long as the file format and structure for assets is relatively stable there doesn't have to be a tight coupling between engine versions and asset versions except when doing QA and benchmarking. This is one of the cases where git's inability to handle rewriting a public history is a pain. The art and programming departments should be able to develop along separate branches, and the programmers should be able to pull a tree from art that has everything between tags squashed (but in a reversible fashion in case there's a need to bisect something later).
    - tomlu 12y ago
      There usually are dependencies between the game and its assets, but sure, these dependencies can be managed. Historically I've dealt with the problem by keeping code and assets in separate repositories (using different version control systems). While it's worthwhile to do so in order to use git for code it really is a pain.
      - colechristensen 12y ago
        Exactly my initial point. It is really worthwhile to treat code and binary data differently, and the tools don't exist to make it not-so-painful. To answer a couple of levels up, it's not bad practice because it isn't useful... there are many circumstances where having big binary blobs versioned and associated with code. It is bad practice because the tools are all written _for_ code and can get nasty when filled with binaries. So there's a problem and a solution, but nobody seems to have put together the right magic yet.
- Locke1689 12y ago
  There is opportunity for a powerful tool to handle the very common bad practice of including giant binary blobs which don't belong in version control. It's only a bad practice because our tools don't support it properly. There seems no reasonable argument that there is some predefined size limit that all assets in version control must, by natural law, fall under. The criteria I have for version controlled assets is they must 1) be versionable, 2) have a comparison tool, and 3) be strongly connected to the other assets under control.
  - colechristensen 12y ago
    >It's only a bad practice because our tools don't support it properly. Yes, exactly.
kyrra 12y ago
Mercurial has the LargeFiles extension[0]. This ships with Mercurial by default, just disabled. This does help some. [0] http://mercurial.selenic.com/wiki/LargefilesExtension http://mercurial.selenic.com/wiki/LargefilesExtension
- novaleaf 12y ago
  while i like LargeFiles, it's really not enough. Support for binary content (and indeed, history truncation) needs to be native. As it stands, LargeFiles can only be used on intranet networks, and if you've ever used it for any serious purpose, you'll run into corruption. It's really disappointing to see 3.0 announced with no solution to this.
  - kyrra 12y ago
    Why intranet only? The data still comes from the HG server through standard means. And what kind of data corruption do you see? It tracks files by their SHA1 value and downloads them as needed. Did you report any corruption issues you hit? Or have links to anything specific?
  - mrtngslr 12y ago
    Largefiles are being used for production in game companies and we have not gotten reports about corruption.
    - novaleaf 12y ago
      i used large files with game assets via tortoise hg. if there's a network timeout, corruption of the largefiles occurs .. what client do you use?
  - tonfa 12y ago
    FYI Mercurial has time-based releases. So 3.0 doesn't mean anything special (it's just the next feature release after 2.9). http://mercurial.selenic.com/wiki/TimeBasedReleasePlan http://mercurial.selenic.com/wiki/TimeBasedReleasePlan
sherlockmao 12y ago
Facebook's hgwatchman and remotefilelog extensions; largefiles
- tomlu 12y ago
  Those are interesting, and git has some similar large file extensions too. However during some years of experimentation I have become convinced that these features have to be part of core or they will always be second-rate citizens and not work well in practice.
  - ygra 12y ago
    Hg is modular and lots of more advanced features are shipped as extensions. Just because they are disabled by default doesn't mean that they are unsupported or are second-rate citizens.

by tomlu 12y ago

colechristensen 12y ago

There is opportunity for a powerful tool to handle the very common bad practice of including giant binary blobs which don't belong in version control. I'd like to see a second class of files which are only checksummed (or timestamped) on 'status', 'diff', 'add' and then binary-diffed on commit for possible compression (or perhaps deduped with checksums of blocks) with features like 'git/hg binary-add somefile.jar' to differentiate.

mintplant 12y ago
Check out git-annex: https://git-annex.branchable.com/how_it_works/ https://git-annex.branchable.com/how_it_works/ It operates under a similar principle (committing hashes rather than file contents) but stores the actual file contents elsewhere rather than using a compression/deduplication within the repo itself.
- colechristensen 12y ago
  I have. It's neat, but it's more about replacing something like Dropbox than it is replacing git (despite being based on git). It also isn't particularly stable, though I have nothing but respect for the developer.
  - AceJohnny2 12y ago
    As I understand it, it became something to replace Dropbox after Joey realized that git-annex's task of distributed storage of large files intersected with the Dropbox use-case, but it didn't start that way, and IMHO doesn't exclude the parent's suggestion. I backed Joey's kickstarter, but to my shame I have yet to give git-annex-assistant a spin.
tomlu 12y ago
> There is opportunity for a powerful tool to handle the very common bad practice of including giant binary blobs which don't belong in version control Why do you say it's bad practice? Game assets come to mind, these aren't derived artifacts and you can't build the game without them.
- wtallis 12y ago
  Usually, you can build the game without them, up to the final packaging stage. Almost any game that is built on an engine can have all of the asset files swapped out without having to recompile anything, and as long as the file format and structure for assets is relatively stable there doesn't have to be a tight coupling between engine versions and asset versions except when doing QA and benchmarking. This is one of the cases where git's inability to handle rewriting a public history is a pain. The art and programming departments should be able to develop along separate branches, and the programmers should be able to pull a tree from art that has everything between tags squashed (but in a reversible fashion in case there's a need to bisect something later).
  - tomlu 12y ago
    There usually are dependencies between the game and its assets, but sure, these dependencies can be managed. Historically I've dealt with the problem by keeping code and assets in separate repositories (using different version control systems). While it's worthwhile to do so in order to use git for code it really is a pain.
    - colechristensen 12y ago
      Exactly my initial point. It is really worthwhile to treat code and binary data differently, and the tools don't exist to make it not-so-painful. To answer a couple of levels up, it's not bad practice because it isn't useful... there are many circumstances where having big binary blobs versioned and associated with code. It is bad practice because the tools are all written _for_ code and can get nasty when filled with binaries. So there's a problem and a solution, but nobody seems to have put together the right magic yet.
Locke1689 12y ago
There is opportunity for a powerful tool to handle the very common bad practice of including giant binary blobs which don't belong in version control. It's only a bad practice because our tools don't support it properly. There seems no reasonable argument that there is some predefined size limit that all assets in version control must, by natural law, fall under. The criteria I have for version controlled assets is they must 1) be versionable, 2) have a comparison tool, and 3) be strongly connected to the other assets under control.
- colechristensen 12y ago
  >It's only a bad practice because our tools don't support it properly. Yes, exactly.

kyrra 12y ago

Mercurial has the LargeFiles extension[0]. This ships with Mercurial by default, just disabled. This does help some. [0] http://mercurial.selenic.com/wiki/LargefilesExtension http://mercurial.selenic.com/wiki/LargefilesExtension

novaleaf 12y ago
while i like LargeFiles, it's really not enough. Support for binary content (and indeed, history truncation) needs to be native. As it stands, LargeFiles can only be used on intranet networks, and if you've ever used it for any serious purpose, you'll run into corruption. It's really disappointing to see 3.0 announced with no solution to this.
- kyrra 12y ago
  Why intranet only? The data still comes from the HG server through standard means. And what kind of data corruption do you see? It tracks files by their SHA1 value and downloads them as needed. Did you report any corruption issues you hit? Or have links to anything specific?
- mrtngslr 12y ago
  Largefiles are being used for production in game companies and we have not gotten reports about corruption.
  - novaleaf 12y ago
    i used large files with game assets via tortoise hg. if there's a network timeout, corruption of the largefiles occurs .. what client do you use?
- tonfa 12y ago
  FYI Mercurial has time-based releases. So 3.0 doesn't mean anything special (it's just the next feature release after 2.9). http://mercurial.selenic.com/wiki/TimeBasedReleasePlan http://mercurial.selenic.com/wiki/TimeBasedReleasePlan

sherlockmao 12y ago

Facebook's hgwatchman and remotefilelog extensions; largefiles

tomlu 12y ago
Those are interesting, and git has some similar large file extensions too. However during some years of experimentation I have become convinced that these features have to be part of core or they will always be second-rate citizens and not work well in practice.
- ygra 12y ago
  Hg is modular and lots of more advanced features are shipped as extensions. Just because they are disabled by default doesn't mean that they are unsupported or are second-rate citizens.