5 ms·

A solution for game assets building. Ideally it would be possible to build all assets (maps, models, textures, binaries) in a game from source files to the for

by thexa4 10y ago

A solution for game assets building. Ideally it would be possible to build all assets (maps, models, textures, binaries) in a game from source files to the format that's distributed to people. Unfortunately due to the build times for certain things like maps it's not feasible to rebuild everything before every release. Combined with the fact that the files are relatively large means that we can't store everything in one git repository. I'd be very grateful if anyone knows a solution to this.

macca321 10y ago
Could you set this up using gulp?
- thexa4 10y ago
  Yes, the problem is that due to the size most of it lives in different repositories so it's hard to know what version you're supposed to build and when something you depend on changes.
lj3 10y ago
I assume you're talking about a large 3d game? Are you using Unity3D? Because if you are, then building game assets involves scripting the Unity3D client itself, which is a huge pain in the ass. As for the git problem, why not use svn or perforce for the game assets? They handle binaries much better than git does.
- thexa4 10y ago
  Yes, it's a large 3d game made in the Source Engine. Luckily all tools are scriptable using bash. Using svn for the game assets would work but to build them we would still need the shaders for example, which would live in the git repository alongside the rest of the code of the game. The resulting problem is that the asset git/svn should be triggered to rebuild when the code git changes. I'm looking for a tool that could help with that. Thanks for the suggestion of perforce, I'll look at that.
  - lj3 10y ago
    > Luckily all tools are scriptable using bash. Lucky you. :) This isn't a simple problem and I don't know of any one size fits all solution for this. From what I could tell, every company comes up with its own asset pipeline that fits with their art, code, test and release processes. I came up with a half assed solution myself back when I was still working in games. We had a separate build server just for assets. It ran a script every hour that pulled down both the code and art repos and watched for changes in certain folders. The shaders, 3d models and textures were linked via naming conventions. Through those naming conventions, the system always knew where to find all of the relevant pieces of any given individual 3d asset (in theory, anyway). It would then create the assets and check them into a separate SVN repo. This system worked ok, but it had a number of problems. One problem was the naming conventions. The artists didn't much pay attention to file names, so any new asset they created would inevitably not get built because it didn't adhere to the conventions. We thought about replacing the naming conventions with a json file describing every 3d asset, where their shaders were, where their 3d models were, any special settings, where they should be saved, etc. but that would have required either a GUI application to help manage it all or a coder whose job was to manage the data files. The former would have taken time and the latter wasn't much better than what we were doing already. We still needed a coder whose job it was to fix asset build issues, which would pop up on a regular basis. Another issue we had was build revisions. Game builds are complicated enough and we added the additional complexity of having to specify a branch name and version number for the assets used. It doesn't seem like it would be an issue, except when you have parallel art work going on in parallel branches along with parallel dev work going on in parallel branches things break. In retrospect, we should have put our foot down and said there is only one official asset branch and all dev builds will use the latest version of that branch, but we didn't. Sorry for the essay, but I hope it was of some use to you. If you ever come up with a universal asset pipeline, sell that sucker! :)
    - thexa4 10y ago
      Good to know other people also had the same problem. If I build it I'll certainly try selling it :)
darkblackcorner 10y ago
Have you looked at Git LFS? https://git-lfs.github.com/ https://git-lfs.github.com/ GitLab have it in the Community and Commercial Editions: https://about.gitlab.com/2015/11/23/announcing-git-lfs-support-in-gitlab/ https://about.gitlab.com/2015/11/23/announcing-git-lfs-suppo...
- thexa4 10y ago
  I haven't yet, thanks for the suggestion, I'll look into it.
deepaksurti 10y ago
Please look at https://www.plasticscm.com/version-control-for-games.html https://www.plasticscm.com/version-control-for-games.html We are. 2 man team and have our own hackeneyed way, some in house bunch of Python scripts to build our game assets. As we do this seriously as a hobby it works :). But now we are at a stage where the 2 games are coming along well and to go full hog, I would choose plastic. It has integration with Unity and gutsync as well so migration is easy and with 7$ pm cloud plans for 5gb assets works for us. Last when I had checked plastic it was out of reach out hobbyist / indie game devs. Hope this helps.
stephenr 10y ago
For unity, there was Unity Asset Server. Not sure if it's still available/supported. More generically, I'd suggest you check out Mercurial with the Largefiles extension.

by thexa4 10y ago

macca321 10y ago

Could you set this up using gulp?

thexa4 10y ago
Yes, the problem is that due to the size most of it lives in different repositories so it's hard to know what version you're supposed to build and when something you depend on changes.

lj3 10y ago

I assume you're talking about a large 3d game? Are you using Unity3D? Because if you are, then building game assets involves scripting the Unity3D client itself, which is a huge pain in the ass. As for the git problem, why not use svn or perforce for the game assets? They handle binaries much better than git does.

thexa4 10y ago
Yes, it's a large 3d game made in the Source Engine. Luckily all tools are scriptable using bash. Using svn for the game assets would work but to build them we would still need the shaders for example, which would live in the git repository alongside the rest of the code of the game. The resulting problem is that the asset git/svn should be triggered to rebuild when the code git changes. I'm looking for a tool that could help with that. Thanks for the suggestion of perforce, I'll look at that.
- lj3 10y ago
  > Luckily all tools are scriptable using bash. Lucky you. :) This isn't a simple problem and I don't know of any one size fits all solution for this. From what I could tell, every company comes up with its own asset pipeline that fits with their art, code, test and release processes. I came up with a half assed solution myself back when I was still working in games. We had a separate build server just for assets. It ran a script every hour that pulled down both the code and art repos and watched for changes in certain folders. The shaders, 3d models and textures were linked via naming conventions. Through those naming conventions, the system always knew where to find all of the relevant pieces of any given individual 3d asset (in theory, anyway). It would then create the assets and check them into a separate SVN repo. This system worked ok, but it had a number of problems. One problem was the naming conventions. The artists didn't much pay attention to file names, so any new asset they created would inevitably not get built because it didn't adhere to the conventions. We thought about replacing the naming conventions with a json file describing every 3d asset, where their shaders were, where their 3d models were, any special settings, where they should be saved, etc. but that would have required either a GUI application to help manage it all or a coder whose job was to manage the data files. The former would have taken time and the latter wasn't much better than what we were doing already. We still needed a coder whose job it was to fix asset build issues, which would pop up on a regular basis. Another issue we had was build revisions. Game builds are complicated enough and we added the additional complexity of having to specify a branch name and version number for the assets used. It doesn't seem like it would be an issue, except when you have parallel art work going on in parallel branches along with parallel dev work going on in parallel branches things break. In retrospect, we should have put our foot down and said there is only one official asset branch and all dev builds will use the latest version of that branch, but we didn't. Sorry for the essay, but I hope it was of some use to you. If you ever come up with a universal asset pipeline, sell that sucker! :)
  - thexa4 10y ago
    Good to know other people also had the same problem. If I build it I'll certainly try selling it :)

darkblackcorner 10y ago

Have you looked at Git LFS? https://git-lfs.github.com/ https://git-lfs.github.com/ GitLab have it in the Community and Commercial Editions: https://about.gitlab.com/2015/11/23/announcing-git-lfs-support-in-gitlab/ https://about.gitlab.com/2015/11/23/announcing-git-lfs-suppo...

thexa4 10y ago
I haven't yet, thanks for the suggestion, I'll look into it.

deepaksurti 10y ago

Please look at https://www.plasticscm.com/version-control-for-games.html https://www.plasticscm.com/version-control-for-games.html We are. 2 man team and have our own hackeneyed way, some in house bunch of Python scripts to build our game assets. As we do this seriously as a hobby it works :). But now we are at a stage where the 2 games are coming along well and to go full hog, I would choose plastic. It has integration with Unity and gutsync as well so migration is easy and with 7$ pm cloud plans for 5gb assets works for us. Last when I had checked plastic it was out of reach out hobbyist / indie game devs. Hope this helps.

stephenr 10y ago

For unity, there was Unity Asset Server. Not sure if it's still available/supported. More generically, I'd suggest you check out Mercurial with the Largefiles extension.