4 ms·

Even ancient svn works much better out of the box for large binary files than git (e.g. a 150 GB working directory with many big Photoshop files and other binar

Even ancient svn works much better out of the box for large binary files than git (e.g. a 150 GB working directory with many big Photoshop files and other binary assets is no problem in SVN). What does SVN do differently than git when it comes to large binary files, and why can't git use the same approach? I also don't quite understand tbh how offloading large files to somewhere else would be fundamentally different than storing all files in one place except complicating everything? Storage is storage, how would a different storage location fix any of the current performance and robustness problems? Offloading just sounds like a solution for public git forges which don't want to deal with big files because it's too costly for them, but increased hosting cost is not the 'large binary file problem' of git. (edit: apparently git supports proper locking(?) so I removed that section - ps: nvm it looks like the file locking feature is only in git-lfs)

pjc50 1y ago
Completely different design. Git is intended to be fully distributed, so (a) every repo is supposed to have the full history of every file, and (b) locking is meaningless. People should use the VCS that's appropriate for their project rather than insist on git everywhere.
- cyphar 1y ago
  To be clear, "fully distributed" also means "can use all of the features offline, and without needing commit access to the central repository". I can't imagine living without that feature, but I also do a lot of OSS work so I'm probably biased.
  - flohofwoe 1y ago
    How often are you fully offline though? A centralized version system could be updated to work in 'offline mode' by queueing pushed changes locally until internet is available again (and in SVN that would be quite trivial because it keeps the last known state of the repository in a local shadow copy anyway).
    - jayd16 1y ago
      All the time. If you use p4 and git you'll notice the difference just from trying to move clone git repos locally vs trying to make a new workspace.
    - themafia 1y ago
      > by queueing pushed changes locally And if you and another developer make conflicting changes while offline? What should happen when you return online?
      - flohofwoe 1y ago
        The same thing as in git: you're not allowed to push your changes to the server until you have resolved the conflicts locally. E.g. with current svn you get the latest changes from the server, open a diff editor, fix the conflicts and then commit. The only difference here between svn and git is that svn merges the 'commit' and 'push' operations into one, e.g. instead of not being allowed to push, you're not allowed to commit in svn if there are pending conflicts. This would be the part that would need to change if svn would get a proper 'offline mode', e.g. commits would need to go into some sort of 'local staging queue' until you get internet access back, and conflict resolutions would need to happen on the commits in that staging queue. But I really doubt if that's worth the hassle because how often are you actually without internet while coding?
    - cyphar 1y ago
      It's not terribly common, but more critically it means I can work even when the source forge is down. (And for corporate stuff it means I can work on stuff without connecting to the VPN.) Also, designing around distribution meant that merges have to be fast and work well -- this is a problem that most centralised systems struggle with because it's not a key part of the workflow. Branching and merging are indispensable tools in version control and I'm shocked how long CVS and SVN survived despite having completely broken systems for one or both. Being able to do both (and do stuff like blames over the entire history) without needing to communicate with a server is something that changes the way you work. My actual hot take (as a kernel developer) is that email patches are good, actually. I really wish more people were aware of how incredibly powerful they are -- I have yet to see a source forge that can get close to the resiliency and flexibility of email (though SourceHut gets very close). git-send-email has its quirks, but b4 honestly makes it incredibly easy. (There's also the owning your own data benefits that I think often go overlooked -- reflog and local clones mean that you cannot practically lose data even if the server goes away or you get blocked. I suspect Russian or Iranian developers losing access to their full repo history because of US sanctions wouldn't share your views on centralised services.)
- flohofwoe 1y ago
  > Git is intended to be fully distributed Which is kinda funny because most people use git through Github or Gitlab, e.g. forcing git back into a centralized model ;) > People should use the VCS that's appropriate for their project rather than insist on git everywhere. Indeed, but I think that train has left long ago :) I had to look it up after I wrote that paragraph about locking, but it looks like file locking is supported in Git (just weird that I need to press a button in the Gitlab UI to lock a file: https://docs.gitlab.com/user/project/file_lock/ https://docs.gitlab.com/user/project/file_lock/ ...and here it says it's only supported with git lfs (so still weird af): https://docs.gitlab.com/topics/git/file_management/#file-locks https://docs.gitlab.com/topics/git/file_management/#file-loc... ...why not simply 'git lock [file]' and 'git push --locks' like it works with tags?
  - nomel 1y ago
    I very much dislike git (mostly because software hasn't been "just source" for many decades before git was hobbled together, and git's binary handling is a huge pain for me), but what does a lockfile achieve that a branch -> merge doesn't, practically, and even conceptually?
    - flohofwoe 1y ago
      Binary files can't be meaningfully merged. E.g. if two people create separate branches and work on the same Photoshop file, good luck trying to merge those changes, you can at best pick one version and the other user just wasted a lot of time. A lock communicates to other users "hey I'm currently working on that file, don't touch it" - this also means that lockable files must be write-only all the time unless explicitly locked - so that people don't "accidentially" start working on the file without acquiring a lock first. Apparently (as far as I have gathered so far from googling) git-lfs supports file locking but not vanilla git.
      - xg15 1y ago
        > Binary files can't be meaningfully merged. I think we really need more development of format-specific diff and merge tools. A lot of binary formats could absolutely be diffed or merged, but you'd need algorithms and maybe UIs specific to that format - there is no "generic" algorithm like for text-based files. (And even there, generic line-wise diffing if often more "good enough" than really good) I think it would be awesome if we could get "diff/merge servers" analogous to the "language servers" for IDEs some day.
- xg15 1y ago
  > People should use the VCS that's appropriate for their project rather than insist on git everywhere. Disagree. I really like the "de-facto standard" that git has become for open source. It means if I want to understand some new project's source code, there is one less hassle for me to deal with: I don't need to learn any new concepts just to access the source code and all the tooling is already right there. The situation we have with package managers, dependency managers and package managers for package managers is worse enough. I really don't want a world in which every language or every project also comes with its own version control system and remote repo infrastructure.
  - flohofwoe 1y ago
    A "proper" versioning system doesn't need to be learned since you literally only need a handful of straightforward operations (how do I get the latest version? how do I add a file? how do I commit changes?) - in svn that's 'svn update', 'svn add' and 'svn commit', that's all what's needed to get you through the day, no 'push', no 'staging area', no 'fetch' vs 'pull' and the inevitable merge-vs-rebase discussion... etc etc etc...) It's only git which has this fractal feature set which requires expert knowledge to untangle.
    - xg15 1y ago
      But if all systems are so similar anyway, why would you need "the right tool for the job"? If nothing else, you have to install it. There will also be subtle differences between concepts, e.g. git and svn both have versions and branches, but the concepts behave differently. I don't know about Mercurial, but I'm sure they have their own quirks as well. Also, tooling: I have a VSCode plugin that visualizes the entire graph structure of a git repo really nicely. Right now, I can use that on 99% of all repos to get an overview of the branches, last commits, activity, etc. If version systems were fragmented, I'd have to look for equivalent tools for every versioning system separately - if they exist at all. More likely, I'd be restricted just to the on-board tools of every system.
      - trinix912 1y ago
- linhns 1y ago
  > People should use the VCS that's appropriate for their project rather than insist on git everywhere. But git is likely to be appropriate almost everywhere. You won’t just use svn just for big file purposes while git is better for everything else in the same project
  - flohofwoe 1y ago
    The thing is, in a game project, easily 99% of all version controlled data is in large binary files, and text files are by far the minority (at least by size). Yet still people try to use git for version control in game projects just because it is the popular option elsewhere and git is all they know.
    - IshKebab 1y ago
      > text files are by far the minority (at least by size) Well yeah because text files are small. Thinking text files are insignificant to games because they are small is a really dumb perspective. > Yet still people try to use git for version control in game projects just because it is the popular option elsewhere and git is all they know. Or perhaps it's because it works really well for text files, which are a significant part of most games, and because the tooling is much better than for other VCS's.
      - flohofwoe 1y ago
        > Thinking text files are insignificant to games because they are small is a really dumb perspective. Fact is that code is only one aspect of a game project, and arguably not the most important. Forcing a programmer-centric workflow on artists and designers is an even dumber perspective ;) > and because the tooling is much better than for other VCS's ...only for text files. For assets like images, 3d models or audio data it's pretty much a wasteland.
        IshKebab 1y ago
        Sure, which is why many studios don't use Git. I was just saying that your argument that code is unimportant because they are text files is super dumb.
- dazzawazza 1y ago
  > People should use the VCS that's appropriate for their project rather than insist on git everywhere. A lot of people don't seem to realise this. I work in game dev and SVN or Perforce are far far better than Git for source control in this space. In AA game dev a checkout (not the complete history, not the source art files) can easily get to 300GB of binary data. This is really pushing Subversion to it's limits. In AAA gamedev you are looking at a full checkout of the latest assets (not the complete history, not the source art files) of at least 1TB and 2TB is becoming more and more common. The whole repo can easily come in at 100 TB. At this scale Perforce is really the only game in town (and they know this and charge through the nose for it). In the movie industry you can multiply AAA gamedev by ~10. Git has no hope of working at this scale as much as I'd like it to.
  - gmokki 1y ago
    I've been thinking of using git filter to split the huge asset files (that are just internally a collection of assets bundler to 200M-1GB files) into smaller ones. That way when artist modifies one sub-asset in a huge file only the small change is recorded in history. There is an example filter for doing this with zip files. The above should work. But does git support multiple filters for a file? For example first the above asset split filter and then store the files in LFS which is another filter.
    - dazzawazza 1y ago
      I mean it might work but you'll still get pull time-outs constantly with LFS. It's expensive to wait two or three days before you can start working on a project. Go way for two weeks, it will be a day before you can "pulled" up to date. I hope this "new" system works but I think Perforce is safe for now.
  - jayd16 1y ago
    Perforce gets the job done but it's a major reason why build tooling is worse in games. Github/gitlab is miles ahead of anything you can get with Perforce. People are not just pushing for git because they ux of it, they're pushing git so they can use the ecosystem.
jayd16 1y ago
You should look into how git is architected to support a lot of features SVN doesn't (distributed repos is a big one). When you clone a git repo you clone the full file history. This is trivial for text but can be extremely large for binary files. Storage is not storage as you can store things as copies or diffs (and a million other ways). For code, diffs are efficient but for binaries, diffs approach double the size of the original files, simply sending/storing the full file is better. These differences have big effects on how git operates and many design choices assumed diffable text assets only. If you do a little research, there's plenty of information on the topic.

pjc50 1y ago

Completely different design. Git is intended to be fully distributed, so (a) every repo is supposed to have the full history of every file, and (b) locking is meaningless. People should use the VCS that's appropriate for their project rather than insist on git everywhere.

cyphar 1y ago
To be clear, "fully distributed" also means "can use all of the features offline, and without needing commit access to the central repository". I can't imagine living without that feature, but I also do a lot of OSS work so I'm probably biased.
- flohofwoe 1y ago
  How often are you fully offline though? A centralized version system could be updated to work in 'offline mode' by queueing pushed changes locally until internet is available again (and in SVN that would be quite trivial because it keeps the last known state of the repository in a local shadow copy anyway).
  - jayd16 1y ago
    All the time. If you use p4 and git you'll notice the difference just from trying to move clone git repos locally vs trying to make a new workspace.
  - themafia 1y ago
    > by queueing pushed changes locally And if you and another developer make conflicting changes while offline? What should happen when you return online?
    - flohofwoe 1y ago
      The same thing as in git: you're not allowed to push your changes to the server until you have resolved the conflicts locally. E.g. with current svn you get the latest changes from the server, open a diff editor, fix the conflicts and then commit. The only difference here between svn and git is that svn merges the 'commit' and 'push' operations into one, e.g. instead of not being allowed to push, you're not allowed to commit in svn if there are pending conflicts. This would be the part that would need to change if svn would get a proper 'offline mode', e.g. commits would need to go into some sort of 'local staging queue' until you get internet access back, and conflict resolutions would need to happen on the commits in that staging queue. But I really doubt if that's worth the hassle because how often are you actually without internet while coding?
  - cyphar 1y ago
    It's not terribly common, but more critically it means I can work even when the source forge is down. (And for corporate stuff it means I can work on stuff without connecting to the VPN.) Also, designing around distribution meant that merges have to be fast and work well -- this is a problem that most centralised systems struggle with because it's not a key part of the workflow. Branching and merging are indispensable tools in version control and I'm shocked how long CVS and SVN survived despite having completely broken systems for one or both. Being able to do both (and do stuff like blames over the entire history) without needing to communicate with a server is something that changes the way you work. My actual hot take (as a kernel developer) is that email patches are good, actually. I really wish more people were aware of how incredibly powerful they are -- I have yet to see a source forge that can get close to the resiliency and flexibility of email (though SourceHut gets very close). git-send-email has its quirks, but b4 honestly makes it incredibly easy. (There's also the owning your own data benefits that I think often go overlooked -- reflog and local clones mean that you cannot practically lose data even if the server goes away or you get blocked. I suspect Russian or Iranian developers losing access to their full repo history because of US sanctions wouldn't share your views on centralised services.)
flohofwoe 1y ago
> Git is intended to be fully distributed Which is kinda funny because most people use git through Github or Gitlab, e.g. forcing git back into a centralized model ;) > People should use the VCS that's appropriate for their project rather than insist on git everywhere. Indeed, but I think that train has left long ago :) I had to look it up after I wrote that paragraph about locking, but it looks like file locking is supported in Git (just weird that I need to press a button in the Gitlab UI to lock a file: https://docs.gitlab.com/user/project/file_lock/ https://docs.gitlab.com/user/project/file_lock/ ...and here it says it's only supported with git lfs (so still weird af): https://docs.gitlab.com/topics/git/file_management/#file-locks https://docs.gitlab.com/topics/git/file_management/#file-loc... ...why not simply 'git lock [file]' and 'git push --locks' like it works with tags?
- nomel 1y ago
  I very much dislike git (mostly because software hasn't been "just source" for many decades before git was hobbled together, and git's binary handling is a huge pain for me), but what does a lockfile achieve that a branch -> merge doesn't, practically, and even conceptually?
  - flohofwoe 1y ago
    Binary files can't be meaningfully merged. E.g. if two people create separate branches and work on the same Photoshop file, good luck trying to merge those changes, you can at best pick one version and the other user just wasted a lot of time. A lock communicates to other users "hey I'm currently working on that file, don't touch it" - this also means that lockable files must be write-only all the time unless explicitly locked - so that people don't "accidentially" start working on the file without acquiring a lock first. Apparently (as far as I have gathered so far from googling) git-lfs supports file locking but not vanilla git.
    - xg15 1y ago
      > Binary files can't be meaningfully merged. I think we really need more development of format-specific diff and merge tools. A lot of binary formats could absolutely be diffed or merged, but you'd need algorithms and maybe UIs specific to that format - there is no "generic" algorithm like for text-based files. (And even there, generic line-wise diffing if often more "good enough" than really good) I think it would be awesome if we could get "diff/merge servers" analogous to the "language servers" for IDEs some day.
xg15 1y ago
> People should use the VCS that's appropriate for their project rather than insist on git everywhere. Disagree. I really like the "de-facto standard" that git has become for open source. It means if I want to understand some new project's source code, there is one less hassle for me to deal with: I don't need to learn any new concepts just to access the source code and all the tooling is already right there. The situation we have with package managers, dependency managers and package managers for package managers is worse enough. I really don't want a world in which every language or every project also comes with its own version control system and remote repo infrastructure.
- flohofwoe 1y ago
  A "proper" versioning system doesn't need to be learned since you literally only need a handful of straightforward operations (how do I get the latest version? how do I add a file? how do I commit changes?) - in svn that's 'svn update', 'svn add' and 'svn commit', that's all what's needed to get you through the day, no 'push', no 'staging area', no 'fetch' vs 'pull' and the inevitable merge-vs-rebase discussion... etc etc etc...) It's only git which has this fractal feature set which requires expert knowledge to untangle.
  - xg15 1y ago
    But if all systems are so similar anyway, why would you need "the right tool for the job"? If nothing else, you have to install it. There will also be subtle differences between concepts, e.g. git and svn both have versions and branches, but the concepts behave differently. I don't know about Mercurial, but I'm sure they have their own quirks as well. Also, tooling: I have a VSCode plugin that visualizes the entire graph structure of a git repo really nicely. Right now, I can use that on 99% of all repos to get an overview of the branches, last commits, activity, etc. If version systems were fragmented, I'd have to look for equivalent tools for every versioning system separately - if they exist at all. More likely, I'd be restricted just to the on-board tools of every system.
    - trinix912 1y ago
linhns 1y ago
> People should use the VCS that's appropriate for their project rather than insist on git everywhere. But git is likely to be appropriate almost everywhere. You won’t just use svn just for big file purposes while git is better for everything else in the same project
- flohofwoe 1y ago
  The thing is, in a game project, easily 99% of all version controlled data is in large binary files, and text files are by far the minority (at least by size). Yet still people try to use git for version control in game projects just because it is the popular option elsewhere and git is all they know.
  - IshKebab 1y ago
    > text files are by far the minority (at least by size) Well yeah because text files are small. Thinking text files are insignificant to games because they are small is a really dumb perspective. > Yet still people try to use git for version control in game projects just because it is the popular option elsewhere and git is all they know. Or perhaps it's because it works really well for text files, which are a significant part of most games, and because the tooling is much better than for other VCS's.
    - flohofwoe 1y ago
      > Thinking text files are insignificant to games because they are small is a really dumb perspective. Fact is that code is only one aspect of a game project, and arguably not the most important. Forcing a programmer-centric workflow on artists and designers is an even dumber perspective ;) > and because the tooling is much better than for other VCS's ...only for text files. For assets like images, 3d models or audio data it's pretty much a wasteland.
      - IshKebab 1y ago
        Sure, which is why many studios don't use Git. I was just saying that your argument that code is unimportant because they are text files is super dumb.
dazzawazza 1y ago
> People should use the VCS that's appropriate for their project rather than insist on git everywhere. A lot of people don't seem to realise this. I work in game dev and SVN or Perforce are far far better than Git for source control in this space. In AA game dev a checkout (not the complete history, not the source art files) can easily get to 300GB of binary data. This is really pushing Subversion to it's limits. In AAA gamedev you are looking at a full checkout of the latest assets (not the complete history, not the source art files) of at least 1TB and 2TB is becoming more and more common. The whole repo can easily come in at 100 TB. At this scale Perforce is really the only game in town (and they know this and charge through the nose for it). In the movie industry you can multiply AAA gamedev by ~10. Git has no hope of working at this scale as much as I'd like it to.
- gmokki 1y ago
  I've been thinking of using git filter to split the huge asset files (that are just internally a collection of assets bundler to 200M-1GB files) into smaller ones. That way when artist modifies one sub-asset in a huge file only the small change is recorded in history. There is an example filter for doing this with zip files. The above should work. But does git support multiple filters for a file? For example first the above asset split filter and then store the files in LFS which is another filter.
  - dazzawazza 1y ago
    I mean it might work but you'll still get pull time-outs constantly with LFS. It's expensive to wait two or three days before you can start working on a project. Go way for two weeks, it will be a day before you can "pulled" up to date. I hope this "new" system works but I think Perforce is safe for now.
- jayd16 1y ago
  Perforce gets the job done but it's a major reason why build tooling is worse in games. Github/gitlab is miles ahead of anything you can get with Perforce. People are not just pushing for git because they ux of it, they're pushing git so they can use the ecosystem.

jayd16 1y ago

You should look into how git is architected to support a lot of features SVN doesn't (distributed repos is a big one). When you clone a git repo you clone the full file history. This is trivial for text but can be extremely large for binary files. Storage is not storage as you can store things as copies or diffs (and a million other ways). For code, diffs are efficient but for binaries, diffs approach double the size of the original files, simply sending/storing the full file is better. These differences have big effects on how git operates and many design choices assumed diffable text assets only. If you do a little research, there's plenty of information on the topic.