6 ms·

SnowFS – a fast, scalable version control file storage for graphic files

86 points by karateka 6y ago

pjmlp 6y ago
JavaScript[0] isn't what I would call a fast storage repository, but I guess it works out for prototyping. On the context of porting to C and C++, or make it execute faster, I can see two options with minor rewrites. Use AssemblyScript and generate native code via WebAssembly AOT compilers. Try to adapt TypeScript to C++ compiler from Microsoft's MakeCode project. Implement your own C++ code generator. It would be much easier than keep using multiple code bases in parallel, plus any memory corruption issues would be most likely bugs on the code generator. [0] - Yes I know the source code is Typescript.
- sebastian_io 6y ago
  Indeed, TS/JS is great for its quick turnaround times for prototyping. But the I/O performance is executed by the underlying C/C++ layer. For the rest TS/JS is fast enough. But a full C++ backport is still on the horizon
kevlar1818 6y ago
Very interesting. I'd like to learn more about how it works. How does this compare to DVC[1], for instance? I'll throw in a shameless plug for my tool in this area, Dud[2]. Dud is to DVC what Flask is to Django. Are the mentioned benchmarks published somewhere? [1]: https://dvc.org https://dvc.org [2]: https://github.com/kevin-hanselman/dud https://github.com/kevin-hanselman/dud
- sebastian_io 6y ago
  SnowFS is still at the beginnings, so it will shape over time. For the benchmarks, these are the results on a MacBook-2019 with APFS for a 4GB file: [git add texture.psd: 20164ms] [snow add texture.psd: 4596ms] [git rm texture.psd: 575ms] [snow rm texture.psd: 111ms] [git checkout HEAD~1: 9739ms] [snow checkout HEAD~1: 1ms] You might get slightly slower speeds on NTFS for 'add' and 'checkout' but still very performant
jarym 6y ago
I need to look into this more but unless you already know C/C++ really well then Rust may be a better choice.
- sebastian_io 6y ago
  Same here, the aimed goal is a C++ backport, bc I am a C++ dev. But Rust is on my radar and happy to learn
amelius 6y ago
But what if I have software and images (e.g. as in game software)? Which version control system do I use? Honestly, I think the effort was better spent on an improved version of Git.
- sebastian_io 6y ago
  Programmers, game makers and 2D/3D artists are very different target audiences with very different needs. To name one example, the commit hash integrity which is a foundation of Git is a must-have for software projects, but might be not useful in environments like VFX or CG productions
  - mbreese 6y ago
    Also, the project is very clear that it is for individuals with no collaboration tools built in. While git does have utility for a solo developer, having a separate system for a solo artist (that isn’t working with assets for developers), starts make more sense.
  - amelius 6y ago
    > Programmers, game makers and 2D/3D artists are very different target audiences with very different needs. Yes, so if you want to make them use the same version control system (not a strange requirement, see Google monorepo), then the system should be all-encompassing, and not limited to one type of user. And it may turn out that features meant for one type of user are also useful for other types of user. For example, programmers may need to store large image files as part of a deep-learning image classification system. I'm usually not a fan of packing as many features as possible into a single tool, but in case of version control it makes sense. (A pluggable/modularized approach should of course be used if possible).
    - sebastian_io 6y ago
      You are absolutely right! A tool that can cover everyones needs is a great goal, and a pluggable or modularized approach might be even the best. SnowFS is not a one-way street, there are certain features that are currently investigated to be feature-proposed for libgit2. Means, this project is a great way to explore the possibilities without any historical or technical barriers because it is so lightweight. Best proof are the benchmarks
erlend_sh 6y ago
There is a somewhat related discussion going on about Pijul’s suitability for use with game editors: https://discourse.pijul.org/t/using-pijul-for-a-game-editor/729?u=erlend_sh https://discourse.pijul.org/t/using-pijul-for-a-game-editor/...
- sebastian_io 6y ago
  That looks really cool! I will check the project out, thanks!
mrmrcoleman 6y ago
This is great, well done. While at (now defunct) Dotscience we did a lot of work on Dotmesh which you might find interesting: https://github.com/dotmesh-io/dotmesh https://github.com/dotmesh-io/dotmesh I would also look at data science/ML as a potential use for the tool as there are real issues with using Git for training data. Last point which is more of a tip: Show don’t tell. If you did some side by side workflow walkthroughs showing the difficulties with other tools it will make it easier for people to see that this problem is real (which it definitely is)
- sebastian_io 6y ago
  Interesting project, thanks for sharing the link! I will check that out. And for the tip, agree 100%! :-)
lucideer 6y ago
From the README: > Why not Git/Git-LFS, libgit2, or SVN? > Disadvantages: > (Without Git-LFS): Heavy cost with zipping, packing, and delta-compression for larger files Given the caveat (without Git-LFS) it seems odd to include this in the list > If not properly tracked, binaries become accidentally part of "base" history That's a big "if", and not an inherent problem. This could easily be resolved by any good design-focused UI (e.g. SnowTrack), so this seems a poor argument against using Git as a backend. > Removing older commits is cumbersome due to Gits commit hashing integrity This (like the first bullet point) does not apply to Git-LFS. > Complicated rewriting history procedure What? > Issues with binaries >4GB on Windows A known bug in Git-LFS that they're working to fix. There are workarounds provided in the linked tickets (that could be leveraged by a UI / abstraction layer like SnowTrack). This is the first item in the bullet list that is a real disadvantage of Git LFS, but the workaround for it seems much less effort than developing a new VCS backend from scratch. > Slow in binary modification detection I'm not sure if this applies to Git or Git LFS; there's little detail provided. But if it's significant, this is probably the only really compelling disadvantage listed. > Git uses a restrictive license And finally we see the real reason for not using Git. --- NOTE: I don't mean to make out that building an alternative VCS to Git is not worth pursuing. Nor that it needs any specific justification. Just that listing a justification that seems (to me) mostly disingenuous is worth pointing out.
- sebastian_io 6y ago
  Thanks for listing out the points you disagree with! The project is still in early alpha, so is the Readme. Therefore anything that is ambiguous or not clear, is worth to address. The main requirement is performance (a missing point in your list). If Git would be a good candidate as a versioning system for DCC software packages, it would have been picked up by now, but it didn't happen, among others because of the reasons listed above. Git addresses a completely different target audience and lifecycle than SnowFS. The commit hash integrity is a problem in CG/VFX productions, so is the 4GB limitation, as well as the I/O performance for large binary files. The fact that these issues are still there are fully understandable, given the responsibility and dependencies of this project. That's why SnowFS tries to address the niche requirements with its light implementation. In terms of the license, this is intentionally the weakest argument of all. It doesn't prevent anyone under the GPL to ship Git as an external program with a commercial software, same counts for libgit2 with its linking exception. So there is not even a real benefit here. But the chosen MIT license is an open invitation for everyone. P.S. Certain features and technical solutions will be feature-proposed to libgit2

86 points by karateka 6y ago

JavaScript[0] isn't what I would call a fast storage repository, but I guess it works out for prototyping. On the context of porting to C and C++, or make it execute faster, I can see two options with minor rewrites. Use AssemblyScript and generate native code via WebAssembly AOT compilers. Try to adapt TypeScript to C++ compiler from Microsoft's MakeCode project. Implement your own C++ code generator. It would be much easier than keep using multiple code bases in parallel, plus any memory corruption issues would be most likely bugs on the code generator. [0] - Yes I know the source code is Typescript.

sebastian_io 6y ago
Indeed, TS/JS is great for its quick turnaround times for prototyping. But the I/O performance is executed by the underlying C/C++ layer. For the rest TS/JS is fast enough. But a full C++ backport is still on the horizon

kevlar1818 6y ago

Very interesting. I'd like to learn more about how it works. How does this compare to DVC[1], for instance? I'll throw in a shameless plug for my tool in this area, Dud[2]. Dud is to DVC what Flask is to Django. Are the mentioned benchmarks published somewhere? [1]: https://dvc.org https://dvc.org [2]: https://github.com/kevin-hanselman/dud https://github.com/kevin-hanselman/dud

sebastian_io 6y ago
SnowFS is still at the beginnings, so it will shape over time. For the benchmarks, these are the results on a MacBook-2019 with APFS for a 4GB file: [git add texture.psd: 20164ms] [snow add texture.psd: 4596ms] [git rm texture.psd: 575ms] [snow rm texture.psd: 111ms] [git checkout HEAD~1: 9739ms] [snow checkout HEAD~1: 1ms] You might get slightly slower speeds on NTFS for 'add' and 'checkout' but still very performant

jarym 6y ago

I need to look into this more but unless you already know C/C++ really well then Rust may be a better choice.

sebastian_io 6y ago
Same here, the aimed goal is a C++ backport, bc I am a C++ dev. But Rust is on my radar and happy to learn

amelius 6y ago

But what if I have software and images (e.g. as in game software)? Which version control system do I use? Honestly, I think the effort was better spent on an improved version of Git.

sebastian_io 6y ago
Programmers, game makers and 2D/3D artists are very different target audiences with very different needs. To name one example, the commit hash integrity which is a foundation of Git is a must-have for software projects, but might be not useful in environments like VFX or CG productions
- mbreese 6y ago
  Also, the project is very clear that it is for individuals with no collaboration tools built in. While git does have utility for a solo developer, having a separate system for a solo artist (that isn’t working with assets for developers), starts make more sense.
- amelius 6y ago
  > Programmers, game makers and 2D/3D artists are very different target audiences with very different needs. Yes, so if you want to make them use the same version control system (not a strange requirement, see Google monorepo), then the system should be all-encompassing, and not limited to one type of user. And it may turn out that features meant for one type of user are also useful for other types of user. For example, programmers may need to store large image files as part of a deep-learning image classification system. I'm usually not a fan of packing as many features as possible into a single tool, but in case of version control it makes sense. (A pluggable/modularized approach should of course be used if possible).
  - sebastian_io 6y ago
    You are absolutely right! A tool that can cover everyones needs is a great goal, and a pluggable or modularized approach might be even the best. SnowFS is not a one-way street, there are certain features that are currently investigated to be feature-proposed for libgit2. Means, this project is a great way to explore the possibilities without any historical or technical barriers because it is so lightweight. Best proof are the benchmarks

erlend_sh 6y ago

There is a somewhat related discussion going on about Pijul’s suitability for use with game editors: https://discourse.pijul.org/t/using-pijul-for-a-game-editor/729?u=erlend_sh https://discourse.pijul.org/t/using-pijul-for-a-game-editor/...

sebastian_io 6y ago
That looks really cool! I will check the project out, thanks!

mrmrcoleman 6y ago

This is great, well done. While at (now defunct) Dotscience we did a lot of work on Dotmesh which you might find interesting: https://github.com/dotmesh-io/dotmesh https://github.com/dotmesh-io/dotmesh I would also look at data science/ML as a potential use for the tool as there are real issues with using Git for training data. Last point which is more of a tip: Show don’t tell. If you did some side by side workflow walkthroughs showing the difficulties with other tools it will make it easier for people to see that this problem is real (which it definitely is)

sebastian_io 6y ago
Interesting project, thanks for sharing the link! I will check that out. And for the tip, agree 100%! :-)

lucideer 6y ago

From the README: > Why not Git/Git-LFS, libgit2, or SVN? > Disadvantages: > (Without Git-LFS): Heavy cost with zipping, packing, and delta-compression for larger files Given the caveat (without Git-LFS) it seems odd to include this in the list > If not properly tracked, binaries become accidentally part of "base" history That's a big "if", and not an inherent problem. This could easily be resolved by any good design-focused UI (e.g. SnowTrack), so this seems a poor argument against using Git as a backend. > Removing older commits is cumbersome due to Gits commit hashing integrity This (like the first bullet point) does not apply to Git-LFS. > Complicated rewriting history procedure What? > Issues with binaries >4GB on Windows A known bug in Git-LFS that they're working to fix. There are workarounds provided in the linked tickets (that could be leveraged by a UI / abstraction layer like SnowTrack). This is the first item in the bullet list that is a real disadvantage of Git LFS, but the workaround for it seems much less effort than developing a new VCS backend from scratch. > Slow in binary modification detection I'm not sure if this applies to Git or Git LFS; there's little detail provided. But if it's significant, this is probably the only really compelling disadvantage listed. > Git uses a restrictive license And finally we see the real reason for not using Git. --- NOTE: I don't mean to make out that building an alternative VCS to Git is not worth pursuing. Nor that it needs any specific justification. Just that listing a justification that seems (to me) mostly disingenuous is worth pointing out.

sebastian_io 6y ago
Thanks for listing out the points you disagree with! The project is still in early alpha, so is the Readme. Therefore anything that is ambiguous or not clear, is worth to address. The main requirement is performance (a missing point in your list). If Git would be a good candidate as a versioning system for DCC software packages, it would have been picked up by now, but it didn't happen, among others because of the reasons listed above. Git addresses a completely different target audience and lifecycle than SnowFS. The commit hash integrity is a problem in CG/VFX productions, so is the 4GB limitation, as well as the I/O performance for large binary files. The fact that these issues are still there are fully understandable, given the responsibility and dependencies of this project. That's why SnowFS tries to address the niche requirements with its light implementation. In terms of the license, this is intentionally the weakest argument of all. It doesn't prevent anyone under the GPL to ship Git as an external program with a commercial software, same counts for libgit2 with its linking exception. So there is not even a real benefit here. But the chosen MIT license is an open invitation for everyone. P.S. Certain features and technical solutions will be feature-proposed to libgit2