6 ms·

What makes it complicated? I don't do games development so I've no idea how the project structure looks.

JohnBooty 6y ago
I'd be interested to hear the answer as well! I am neither an Unreal nor games developer, but in case nobody provides a better answer... From what I've heard, what makes source control tricky for game development is all of the non-text files. Git's distributed "everybody has a local copy of the entire repo" approach is not well suited for binary assets, especially large and frequently changing ones. Nearly any "modern" game development project will have gigabytes if not terabytes of these. Imagine you're a game coder and every `get fetch` grabs another 400GB of textures from the level designers. Last I heard many game dev shops still used Subversion instead of git for precisely this reason. There are also workarounds/extensions for git; not sure of their maturity/adoption.
- illvm 6y ago
  Why not just split the assets and the code, have the code reference a version of the assets, which are on a separate server and only fetched when needed?
  - thu2111 6y ago
    That's the same thing as not using version control, given that the point of using UE4 is that you don't have to write much code. Note the repeated references to "blueprint spaghetti". Blueprint is a visual DSL for game designers. The assets end up encoding much of the game logic.
  - pjc50 6y ago
    That's basically what git-annex is, but I guarantee that companies with this problem will be doing something disastrously incoherent with file shares and document_final_2 names instead.
    - mrec 6y ago
      I thought Git LFS had largely replaced git-annex these days? (I haven't used either, just going off [1].) Apparently git-annex never really supported Windows, which is a bit of a dealbreaker for gamedev. [1] https://stackoverflow.com/questions/39337586/how-do-git-lfs-and-git-annex-differ https://stackoverflow.com/questions/39337586/how-do-git-lfs-...
  - JohnBooty 6y ago
    Such a separation makes sense to me (a non game dev) but the assets need version control too, though, so that doesn't really solve the problem.
    - teamonkey 6y ago
      Raw assets do, built versions of assets and binaries don't. Most workflows I've seen cache built data outside of source control. It needs to be in sync with the source but not versioned. Downloading a 10GB+ built map data file over SMB is a sensible option - much faster than via Perforce. I've even seen bittorrent solutions for this before.
- khalladay 6y ago
  I've been in games for about 9 years. In my experience, virtually everyone uses Perforce. Studios that use Subversion seem to be the outlier. You're dead on with the rest though. I'll add that UE4 is a large, hulking behemoth that generates a metric crap ton of intermediate files during builds and general development, which makes managing what to check in a bit of bear. Additionally, a lot of teams try to avoid having artists need to re-compile their editor when they pull changes from P4, so some amount of compiled binaries get checked in as well (usually from an automated build system like Jenkins or Team City that runs after every source file commit). There are also some parts of the engine that need to exist in order for other parts to run properly, but that don't get compiled when you're making changes to the engine itself (like helper programs, or platform specific DLLs, which you need to specifically compile when you want them). Sometimes, these binaries are checked into p4 for one reason or another, which leads to fun things like needing to remember that if you're checking in the DotNet binaries folder, to explicitly NOT check in a couple iOS related DLLs, since the engine's build tool will try to recompile them every time you build a game for any platform, and will fail if those files aren't writeable. The engine is so massive at this point that there several other things like this that need to be kept in mind when setting up version control on a project. [edit: I originally said there were "at least 20" things like this and realized I couldn't think of that many, so I've revised] The engine is very capable (there's a reason that AAA studios without an in-house engine default to it), but it's also got 20 years of legacy systems in it and a lot of pain points to deal with for projects of any real size.
  - echelon 6y ago
    Is there any room for a new company to make a clean break and offer something completely new, or would that take decades of engineering work?
    - Danieru 6y ago
      More likely Epic themselves would offer source control. I run my indie studio off UE4 on git lfs, which I can do since we are small. Perforce and Perfoce centric workflows tend to be broken not because of UE4 but because of bad workflow design. Take our parent poster's example: using source control for binary build distribution. It does not matter what engine you use if you abuse the source control to distribute full builds your workflow will have pain points. Same deal with the example for iOS binaries. This is related to Perforce locking those files: "Doctor, it hurts when I stab myself, but my workflow requires it". Don't checkin build files is the "no duh" answer. Such "no duh" answers from indies usually result in "but you do not understand our unique insane required workflow" responses from big team engineers. In the future when you hear a weird non-sense workflow from AAA teams keep in mind part of the pain is self-inflicted. Big teams try optimizing for different goals, like not requiring artists to compile code. Yet do so in weird half-measures.
learc83 6y ago
In addition to large binary asset files, in Unity you have tons of yaml files that you don’t edit by hand. If you make a small change to a scene with the editor, it can change dozens of lines in the corresponding yaml file. Now imagine 2 people working on the same scene. When you attempt to merge, you have to read and understand an auto generated file that was never intended for human consumption. To a lesser degree, it’s analogous to using line based source control on a jpeg to handle merging edits made in photoshop.
- ryandrake 6y ago
  Sounds like the yaml shouldn't be in source control. I've always followed the guideline: If it's generated by the computer from some other thing, then put the other thing in source control.
  - thu2111 6y ago
    It's not generated from another thing. It's the editor's save format, so it's the direct output of human work. It's just not in a 'genuinely' mergeable format. The fact is, that we take VCS for granted as programmers, but the vast majority of people don't have access to workflows that allow for true multi-user collaboration or branching/merging. Google Docs was a revolution because even though real-time joint editing is not that great and no substitute for the "work independent, review, merge" workflow developers use, it's still better than emailing Word docs to each other. Which is BTW still extremely common even in firms where they have Office 365 - most people never learned about the sharing and collab editing features. On a game most people aren't devs. So VCS is less relevant to them, and git especially so, for the reasons someone else discusses below. Note the mention of Perforce in the article. Why are they using expensive proprietary VCS? Well, Perforce is better at handling fully centralised workflows with large binary assets.
    - deleted 6y ago
      [deleted]
- deleted 6y ago
  [deleted]

I'd be interested to hear the answer as well! I am neither an Unreal nor games developer, but in case nobody provides a better answer... From what I've heard, what makes source control tricky for game development is all of the non-text files. Git's distributed "everybody has a local copy of the entire repo" approach is not well suited for binary assets, especially large and frequently changing ones. Nearly any "modern" game development project will have gigabytes if not terabytes of these. Imagine you're a game coder and every `get fetch` grabs another 400GB of textures from the level designers. Last I heard many game dev shops still used Subversion instead of git for precisely this reason. There are also workarounds/extensions for git; not sure of their maturity/adoption.

illvm 6y ago
Why not just split the assets and the code, have the code reference a version of the assets, which are on a separate server and only fetched when needed?
- thu2111 6y ago
  That's the same thing as not using version control, given that the point of using UE4 is that you don't have to write much code. Note the repeated references to "blueprint spaghetti". Blueprint is a visual DSL for game designers. The assets end up encoding much of the game logic.
- pjc50 6y ago
  That's basically what git-annex is, but I guarantee that companies with this problem will be doing something disastrously incoherent with file shares and document_final_2 names instead.
  - mrec 6y ago
    I thought Git LFS had largely replaced git-annex these days? (I haven't used either, just going off [1].) Apparently git-annex never really supported Windows, which is a bit of a dealbreaker for gamedev. [1] https://stackoverflow.com/questions/39337586/how-do-git-lfs-and-git-annex-differ https://stackoverflow.com/questions/39337586/how-do-git-lfs-...
- JohnBooty 6y ago
  Such a separation makes sense to me (a non game dev) but the assets need version control too, though, so that doesn't really solve the problem.
  - teamonkey 6y ago
    Raw assets do, built versions of assets and binaries don't. Most workflows I've seen cache built data outside of source control. It needs to be in sync with the source but not versioned. Downloading a 10GB+ built map data file over SMB is a sensible option - much faster than via Perforce. I've even seen bittorrent solutions for this before.
khalladay 6y ago
I've been in games for about 9 years. In my experience, virtually everyone uses Perforce. Studios that use Subversion seem to be the outlier. You're dead on with the rest though. I'll add that UE4 is a large, hulking behemoth that generates a metric crap ton of intermediate files during builds and general development, which makes managing what to check in a bit of bear. Additionally, a lot of teams try to avoid having artists need to re-compile their editor when they pull changes from P4, so some amount of compiled binaries get checked in as well (usually from an automated build system like Jenkins or Team City that runs after every source file commit). There are also some parts of the engine that need to exist in order for other parts to run properly, but that don't get compiled when you're making changes to the engine itself (like helper programs, or platform specific DLLs, which you need to specifically compile when you want them). Sometimes, these binaries are checked into p4 for one reason or another, which leads to fun things like needing to remember that if you're checking in the DotNet binaries folder, to explicitly NOT check in a couple iOS related DLLs, since the engine's build tool will try to recompile them every time you build a game for any platform, and will fail if those files aren't writeable. The engine is so massive at this point that there several other things like this that need to be kept in mind when setting up version control on a project. [edit: I originally said there were "at least 20" things like this and realized I couldn't think of that many, so I've revised] The engine is very capable (there's a reason that AAA studios without an in-house engine default to it), but it's also got 20 years of legacy systems in it and a lot of pain points to deal with for projects of any real size.
- echelon 6y ago
  Is there any room for a new company to make a clean break and offer something completely new, or would that take decades of engineering work?
  - Danieru 6y ago
    More likely Epic themselves would offer source control. I run my indie studio off UE4 on git lfs, which I can do since we are small. Perforce and Perfoce centric workflows tend to be broken not because of UE4 but because of bad workflow design. Take our parent poster's example: using source control for binary build distribution. It does not matter what engine you use if you abuse the source control to distribute full builds your workflow will have pain points. Same deal with the example for iOS binaries. This is related to Perforce locking those files: "Doctor, it hurts when I stab myself, but my workflow requires it". Don't checkin build files is the "no duh" answer. Such "no duh" answers from indies usually result in "but you do not understand our unique insane required workflow" responses from big team engineers. In the future when you hear a weird non-sense workflow from AAA teams keep in mind part of the pain is self-inflicted. Big teams try optimizing for different goals, like not requiring artists to compile code. Yet do so in weird half-measures.

learc83 6y ago

In addition to large binary asset files, in Unity you have tons of yaml files that you don’t edit by hand. If you make a small change to a scene with the editor, it can change dozens of lines in the corresponding yaml file. Now imagine 2 people working on the same scene. When you attempt to merge, you have to read and understand an auto generated file that was never intended for human consumption. To a lesser degree, it’s analogous to using line based source control on a jpeg to handle merging edits made in photoshop.

ryandrake 6y ago
Sounds like the yaml shouldn't be in source control. I've always followed the guideline: If it's generated by the computer from some other thing, then put the other thing in source control.
- thu2111 6y ago
  It's not generated from another thing. It's the editor's save format, so it's the direct output of human work. It's just not in a 'genuinely' mergeable format. The fact is, that we take VCS for granted as programmers, but the vast majority of people don't have access to workflows that allow for true multi-user collaboration or branching/merging. Google Docs was a revolution because even though real-time joint editing is not that great and no substitute for the "work independent, review, merge" workflow developers use, it's still better than emailing Word docs to each other. Which is BTW still extremely common even in firms where they have Office 365 - most people never learned about the sharing and collab editing features. On a game most people aren't devs. So VCS is less relevant to them, and git especially so, for the reasons someone else discusses below. Note the mention of Perforce in the article. Why are they using expensive proprietary VCS? Well, Perforce is better at handling fully centralised workflows with large binary assets.
  - deleted 6y ago
    [deleted]
deleted 6y ago
[deleted]