5 ms·

I'm also happy with git, but there's 3 main things that could improve on git IMO: 1) Better handling of large files than git-lfs. As in 10+ GB repos. This is n

by m12k 5y ago

I'm also happy with git, but there's 3 main things that could improve on git IMO: 1) Better handling of large files than git-lfs. As in 10+ GB repos. This is needed for game development (currently they tend to use Perforce or PlasticSCM) 2) Sparse checkout via file system integration (like Eden has) 3) Build system integration, so unchanged files and modules don't even need to be fetched to be compiled, because cached builds can be fetched from a build server instead (requires proper modularization, so e.g. C++ macro expansion doesn't just prevent anything from being cacheable) These are all features that primarily have value for repos that push the limits on size, like big monorepos (with a huge amount of files) or game development (with big asset files). But get it right, and you could massively cut down the time it takes to check out a branch and build it.

webmaven 5y ago
> These are all features that primarily have value for repos that push the limits on size, like big monorepos (with a huge amount of files) or game development (with big asset files). But get it right, and you could massively cut down the time it takes to check out a branch and build it. Would this be a good fit for large machine learning data sets? Every time a new model comes out that touts having been trained on something like a quarter of a billion images, I ask myself "how the heck does someone manage and version a pile like that?"
- zem 5y ago
  would you ever need to version individual images? at a high level, you could version the collection as a whole by filtering by timestamp, and moving deleted files to a "deleted" directory, or maintaining filename and timestamps in a database. I'm sure there are lots of corner cases that would come up when you actually tried to build such a system, but I don't think the overall scheme needs to be as conceptually complex as source code version control
  - webmaven 5y ago
    > would you ever need to version individual images? Potentially. It would depend on the image format and what metadata is (or can be) included in the file.
thomascgalvin 5y ago
ClearCase (claimed to) support points two and three way back in the 90s. It never really worked right; it was always trying to "wink-in" someone else's binary, despite the fact that it was built using a different version of the source.
411111111111111 5y ago
> Build system integration It would be a game changer if any SCM actually manages to achieve this without needing multiple developer teams just for maintenance. That would probably break gits current dominance within a relatively short timeframe... But I don't think it'll happen within the foreseeable future,
- avar 5y ago
  If you want an SCM with direct build system integration there's always GNU make with its support for RCS :) More seriously, can you describe what "build system integration" would look like? Basically like what GNU make does with RCS? I.e. considering the SCM "files" as sources. How would such a system build a dumb tarball snapshot?
  - xmcqdpt2 5y ago
    At my work we have such a system for our monorepo. Basically for each compilation unit (basically subdirectories), it takes the compilation inputs and flags etc and checks them against a remote cache. If it matches, it just pulls the results (binaries but also compiler warnings) from the remote instead of compiling anything. A new clean build is made every few commits to the main branch to populate the cache. In practice it means that if I clone the repo and build it doesn't ever invoke the compiler, so that it takes only a few minutes instead of an hour. Bazel has a similar system but I haven't used it myself, https://docs.bazel.build/versions/main/remote-caching.html https://docs.bazel.build/versions/main/remote-caching.html
    - avar 5y ago
      That seems really useful, but how is this different from what ccache and various other compilation caches that cache things as a function of their input do? The GP talks about "build system integration" being "a game changer", and I can see that being a useful thing for e.g. "priming" your cache, but that's surely just a matter of say: ccache-like-tool --prime $(git rev-parse HEAD:) I.e. give it an arbitrary "root" key and it would pre-download the cached assets for that given key, in this case the key is git-specific, but it could also be a: find -type f -name '*.[ch]' -exec cat {} \; | sha1sum | cut -d' ' -f1 Or whatever.
kvnhn 5y ago
This is by no means a perfect match for your requirements, but I'll share a CLI tool I built, called Dud[0]. At the least it may spur some ideas. Dud is meant to be a companion to SCM (e.g. Git) for large files. I was turned off of Git LFS after a couple failed attempts at using it for data science work. DVC[1] is an improvement in many ways, but it has some rough edges and serious performance issues[2]. With Dud I focused on speed and simplicity. To your three points above: 1) Dud can comfortably track datasets in the 100s of GBs. In practice, the bottleneck is your disk I/O speed. 2) Dud checks out binaries as links by default, so it's super fast to switch between commits. 3) Dud includes a means to build data pipelines -- think Makefiles with less footguns. Dud can detect when outputs are up to date and skip executing a pipeline stage. I hope this helps, and I'd be happy to chat about it. [0]: https://github.com/kevin-hanselman/dud https://github.com/kevin-hanselman/dud [1]: https://dvc.org https://dvc.org [2]: https://github.com/kevin-hanselman/dud#concrete-differences-with-dvc https://github.com/kevin-hanselman/dud#concrete-differences-...
- avar 5y ago
  I'd be curious to see if you've tried git-annex, I use it instead of git-lfs when I need to manage big binary blobs. It does the same trick with a "check out" being a mere symlink.
  - kvnhn 5y ago
    I haven't used it, no. Around the time Git LFS was released, my read from the community was that Git LFS was favored to supersede git-annex, so I focused my time investigating Git LFS. Given that git-annex is still alive and well, I may have discounted it too quickly :) Maybe I'll revisit it in the future. Thanks for sharing!
    - avar 5y ago
      Neither is favored, git-annex solves problems that git LFS doesn't even try to address (distributed big files), at the cost of extra complexity. Git LFS is intended more for a centralized "big repo" workflow, git annex's canonical usage is as a personal distributed backup system, but both can stretch into other domains. In this case git-annex seems to have a feature that git LFS doesn't have that would be useful to you.
bastardoperator 5y ago
Not handling large binary files is a feature from my perspective. Git is for source code and treating it like a disk or a catch all for your project is how people get into scaling trouble. I don't see any reason why versioned artifacts can't be attached to a build, or better, in a warm cache. I get that it's easier to keep everything together, but we can look at the roots of git and it becomes fairly obvious, Linus wasn't checking in 48GB 4K mpeg files for his next FPS.
- maccard 5y ago
  > I don't see any reason why versioned artifacts can't be attached to a build, or better, in a warm cache Because now you've got two versioning systems with different quirks and tools that behave slightly different that _must_ be kept in sync. > I get that it's easier to keep everything together, Easier is a bit of an understatement. Putting binary files outside of the project in another versioning system means you need two version control systems, _plus_ custom tooling to glue the two together. Also, the people who interact with these assets primarily are not technical at all, they're artists or designers, and they're the most likely to have issues with falling between the cracks.
hackerbrother 5y ago
Is (3) doable with the abstractions over Git that Git{la,hu}b provide?
maccard 5y ago
I work in games, and we use PlasticSCM with Unreal Engine. > Better handling of large files than git-lfs. PlasticSCM does really really well with binary files. > Sparse checkout via file system integration Windows only [0] but Plastic does this. I've been working through some issues but it's usable as a daily driver with a sane build system. > Build system integration UnrealBuildTool at the same time is both the coolest and most frustrating build systems I've ever used. It's _not_ a general purpose build system for anyone and everyone, it's tailored to Unreal, it's slow to run the actual build system, the implementation is shaky at times, but some of it's features are incredible. Two standout features are Unity Builds, and Adaptive Unity builds. Unity builds are common now [1], but adaptive unity is a game changer. It uses the Source control integrations to check what files are modified and removes them from the unity blobs, meaning that you're only ever rebuilding + relinking what's changed, and get the best of both worlds. [0] https://blog.plasticscm.com/2021/07/dynamic-workspaces-alpha-for-windows.html https://blog.plasticscm.com/2021/07/dynamic-workspaces-alpha... [1] https://en.wikipedia.org/wiki/Unity_build https://en.wikipedia.org/wiki/Unity_build

by m12k 5y ago

webmaven 5y ago

> These are all features that primarily have value for repos that push the limits on size, like big monorepos (with a huge amount of files) or game development (with big asset files). But get it right, and you could massively cut down the time it takes to check out a branch and build it. Would this be a good fit for large machine learning data sets? Every time a new model comes out that touts having been trained on something like a quarter of a billion images, I ask myself "how the heck does someone manage and version a pile like that?"

zem 5y ago
would you ever need to version individual images? at a high level, you could version the collection as a whole by filtering by timestamp, and moving deleted files to a "deleted" directory, or maintaining filename and timestamps in a database. I'm sure there are lots of corner cases that would come up when you actually tried to build such a system, but I don't think the overall scheme needs to be as conceptually complex as source code version control
- webmaven 5y ago
  > would you ever need to version individual images? Potentially. It would depend on the image format and what metadata is (or can be) included in the file.

thomascgalvin 5y ago

ClearCase (claimed to) support points two and three way back in the 90s. It never really worked right; it was always trying to "wink-in" someone else's binary, despite the fact that it was built using a different version of the source.

411111111111111 5y ago

> Build system integration It would be a game changer if any SCM actually manages to achieve this without needing multiple developer teams just for maintenance. That would probably break gits current dominance within a relatively short timeframe... But I don't think it'll happen within the foreseeable future,

avar 5y ago
If you want an SCM with direct build system integration there's always GNU make with its support for RCS :) More seriously, can you describe what "build system integration" would look like? Basically like what GNU make does with RCS? I.e. considering the SCM "files" as sources. How would such a system build a dumb tarball snapshot?
- xmcqdpt2 5y ago
  At my work we have such a system for our monorepo. Basically for each compilation unit (basically subdirectories), it takes the compilation inputs and flags etc and checks them against a remote cache. If it matches, it just pulls the results (binaries but also compiler warnings) from the remote instead of compiling anything. A new clean build is made every few commits to the main branch to populate the cache. In practice it means that if I clone the repo and build it doesn't ever invoke the compiler, so that it takes only a few minutes instead of an hour. Bazel has a similar system but I haven't used it myself, https://docs.bazel.build/versions/main/remote-caching.html https://docs.bazel.build/versions/main/remote-caching.html
  - avar 5y ago
    That seems really useful, but how is this different from what ccache and various other compilation caches that cache things as a function of their input do? The GP talks about "build system integration" being "a game changer", and I can see that being a useful thing for e.g. "priming" your cache, but that's surely just a matter of say: ccache-like-tool --prime $(git rev-parse HEAD:) I.e. give it an arbitrary "root" key and it would pre-download the cached assets for that given key, in this case the key is git-specific, but it could also be a: find -type f -name '*.[ch]' -exec cat {} \; | sha1sum | cut -d' ' -f1 Or whatever.

kvnhn 5y ago

This is by no means a perfect match for your requirements, but I'll share a CLI tool I built, called Dud[0]. At the least it may spur some ideas. Dud is meant to be a companion to SCM (e.g. Git) for large files. I was turned off of Git LFS after a couple failed attempts at using it for data science work. DVC[1] is an improvement in many ways, but it has some rough edges and serious performance issues[2]. With Dud I focused on speed and simplicity. To your three points above: 1) Dud can comfortably track datasets in the 100s of GBs. In practice, the bottleneck is your disk I/O speed. 2) Dud checks out binaries as links by default, so it's super fast to switch between commits. 3) Dud includes a means to build data pipelines -- think Makefiles with less footguns. Dud can detect when outputs are up to date and skip executing a pipeline stage. I hope this helps, and I'd be happy to chat about it. [0]: https://github.com/kevin-hanselman/dud https://github.com/kevin-hanselman/dud [1]: https://dvc.org https://dvc.org [2]: https://github.com/kevin-hanselman/dud#concrete-differences-with-dvc https://github.com/kevin-hanselman/dud#concrete-differences-...

avar 5y ago
I'd be curious to see if you've tried git-annex, I use it instead of git-lfs when I need to manage big binary blobs. It does the same trick with a "check out" being a mere symlink.
- kvnhn 5y ago
  I haven't used it, no. Around the time Git LFS was released, my read from the community was that Git LFS was favored to supersede git-annex, so I focused my time investigating Git LFS. Given that git-annex is still alive and well, I may have discounted it too quickly :) Maybe I'll revisit it in the future. Thanks for sharing!
  - avar 5y ago
    Neither is favored, git-annex solves problems that git LFS doesn't even try to address (distributed big files), at the cost of extra complexity. Git LFS is intended more for a centralized "big repo" workflow, git annex's canonical usage is as a personal distributed backup system, but both can stretch into other domains. In this case git-annex seems to have a feature that git LFS doesn't have that would be useful to you.

bastardoperator 5y ago

Not handling large binary files is a feature from my perspective. Git is for source code and treating it like a disk or a catch all for your project is how people get into scaling trouble. I don't see any reason why versioned artifacts can't be attached to a build, or better, in a warm cache. I get that it's easier to keep everything together, but we can look at the roots of git and it becomes fairly obvious, Linus wasn't checking in 48GB 4K mpeg files for his next FPS.

maccard 5y ago
> I don't see any reason why versioned artifacts can't be attached to a build, or better, in a warm cache Because now you've got two versioning systems with different quirks and tools that behave slightly different that _must_ be kept in sync. > I get that it's easier to keep everything together, Easier is a bit of an understatement. Putting binary files outside of the project in another versioning system means you need two version control systems, _plus_ custom tooling to glue the two together. Also, the people who interact with these assets primarily are not technical at all, they're artists or designers, and they're the most likely to have issues with falling between the cracks.

hackerbrother 5y ago

Is (3) doable with the abstractions over Git that Git{la,hu}b provide?

maccard 5y ago

I work in games, and we use PlasticSCM with Unreal Engine. > Better handling of large files than git-lfs. PlasticSCM does really really well with binary files. > Sparse checkout via file system integration Windows only [0] but Plastic does this. I've been working through some issues but it's usable as a daily driver with a sane build system. > Build system integration UnrealBuildTool at the same time is both the coolest and most frustrating build systems I've ever used. It's _not_ a general purpose build system for anyone and everyone, it's tailored to Unreal, it's slow to run the actual build system, the implementation is shaky at times, but some of it's features are incredible. Two standout features are Unity Builds, and Adaptive Unity builds. Unity builds are common now [1], but adaptive unity is a game changer. It uses the Source control integrations to check what files are modified and removes them from the unity blobs, meaning that you're only ever rebuilding + relinking what's changed, and get the best of both worlds. [0] https://blog.plasticscm.com/2021/07/dynamic-workspaces-alpha-for-windows.html https://blog.plasticscm.com/2021/07/dynamic-workspaces-alpha... [1] https://en.wikipedia.org/wiki/Unity_build https://en.wikipedia.org/wiki/Unity_build