6 ms·

Oh man, if this could plug into git and be a LFS replacement, that would be awesome. I work in a field where folks run into situations where they think they nee

by Zee2 4y ago

Oh man, if this could plug into git and be a LFS replacement, that would be awesome. I work in a field where folks run into situations where they think they need LFS, and rarely does it work out well. If someone can figure out an ergonomic and durable LFS-like blob versioning system that can align with git histories, that would be incredible.

ensemblehq 4y ago
What’s the challenge with LFS?
- Aeolun 4y ago
  That it’s not baked right into git? Half of my challenges revolve around getting either everything out of LFS, or putting everything in LFS.
- penjelly 4y ago
  non-devs (designers) groan when they need to worry about LFS in my exp. Though i never understood the pain either, just sharing what I've observed
  - raincole 4y ago
    Non-devs complain about any version control tho. Perforce? Plastic SCM? Does anyone really like them?
  - Cthulhu_ 4y ago
    What version control systems are there for non-code? I never looked into it, I can imagine there's a whole world out there for things like written articles, books, but also video game assets like artwork, sprites, 3D models, textures, animations, etc. edit: Looks like e.g. the Unreal engine has support for Perforce and SVN (https://docs.unrealengine.com/5.0/en-US/collaboration-and-version-control-in-unreal-engine/ https://docs.unrealengine.com/5.0/en-US/collaboration-and-ve...), where they make use of locking a file when editing it to avoid merge issues with binary files. I can imagine that frequently goes wrong.
    - grrdotcloud 4y ago
      I am of the opinion the mind of the designer or creative does not think in terms of "versions". There are two states, in woodworking for example, the raw material and the final product. The mental model of "cutting longer" or rolling back does not exist. The idea of a filesystem level understanding of "undo' just has not been communicated or explored. While not Oxen related - TimeMachine seems to be the closest conceptual understanding. Snapshotting the drive, deltas, etc are generally not considered in my experience. I would be curious how larger or mature groups manage versioning.
- remram 4y ago
  LFS is slow (copies files during checkout, downloads/uploads files independently), inefficient (no de-duplication or delta encoding, contrary to base Git), error-prone (needs manual setup, and rewriting history if anyone gets it wrong), and otherwise not well integrated (it relies on smudge filter and other hooks, so some Git commands are aware of it and others not. You will often see the `version ... oid ...` content of the actual blob rather than the large file).
  - unshavedyak 4y ago
    > and otherwise not well integrated (it relies on smudge filter and other hooks, so some Git commands are aware of it and others not. You will often see the `version ... oid ...` content of the actual blob rather than the large file). Curious, do you know the solutions here? I've looked into integrating Git LFS-like things into Git and it felt like Git doesn't give you many tools. Smudging, as hacky as it is, feels one of a very small pool of options you have. I want to say Git Annex does something slightly different. Regardless Smudging felt fine due to how difficult Git made it. Are there better ways in your mind?
    - remram 4y ago
      I don't think you can do better if you don't want to change Git, no.
      - unshavedyak 4y ago
        Seems like an odd critique of LFS then, no? Your other points are super valid, i've wanted to replace LFS with my own impl for the same reasons _(also it was originally designed for HTTP servers, ugh)_. However the integration is roughly as good as it can be, no? Or rather, Oxen would suffer the same limitations no?
        remram 4y ago
        I'm not trying to target this critique at LFS specifically and absolve Git. It might be that LFS is a bad product because of Git's limited extension points, it remains a bad product.
- olvy0 4y ago
  In addition to what was said above: In later versions, LFS doesn't play nice with misconfigured internal networks based on Microsoft's Azure / Active Directory. The LFS project dropped support for NTLM (for good reasons) but it's difficult to convince IT to switch to Kerberos. Our team have been trying to do that for several weeks now. So the result is that if we want to use LFS we are stuck with old versions of LFS which still supported NTLM, which necessitate using old versions of GIT which supported this old version of LFS, else we run into all sorts of weird error messages. Also the same combination of old GIT + old LFS works perfectly on some machines, and fails with some weird error messages on others. Possibly due to another IT misconfiguration or something. I've been wanting for a while now to check if I can fork LFS and fix this, unfortunately my knowledge of golang is very sketchy. Hopefully we'll manage to convince IT to drop NTLM support from the on-premise Azure.
  - e12e 4y ago
    > Hopefully we'll manage to convince IT to drop NTLM support from the on-premise Azure. What year is this? ;) On a more serious note - ntlm, really? And with azure ad, not a regular (old) on-premise domain? I know it can be hard to drag windows setups into the future (hello WINS!) - But why keep ntlm auth in a post windows 7 world? Legacy systems without ldap/radius support? Genuinely curious (and a little horrified).
    - olvy0 4y ago
      Tell me about it ;) Apparently, it's related to the few win 7 machines that are still present somewhere in the organization for various reasons. All of them are airgapped (I hope) so there's no real reason to keep NTLM, but nobody has yet made the decision. Working on getting to the guy in charge to authorize it. Sigh.
- sam_goody 4y ago
  I seem to always be running into git-lfs related issues. The latest include.. If one file doesn't resolve, for any reason, git-lfs will block the entire pull. No way to say "get whatever you could, and the fact that aws is not serving that file is not a blocker" Git clone does not pull the entire lfs history of an image. Lacking the entire history, lfs will fail consistently on future actions.
gschoeni 4y ago
Creator of the project here, would love to look at what an LFS style integration with git hooks could look like. Let me know if you have any ideas there!
- Zee2 4y ago
  Honestly, I'd just trawl the interwebs for all the reasons folks have had grief with LFS, and then see if Oxen can even do a fraction better. If it can, you'll have many billion-dollar companies that would be highly interested in your solution :) A few starting points from personal experience would be - Ensuring git-aware code editors don't try and mangle blob histories when they encounter them - Modern, high-performance blob downloads/version-resolution - Human-readable/debuggable meta-files, where it's easy to recover from bad states (instead of an opaque empty file with a GUID...?)
  - yarg 4y ago
    If you want blob support, you'd probably need to introduce format specific understanding of file structures. You'd also need for the blob encoder to be exceptionally stable - if minor changes to the input lead to significant changes to output, you're going to have a lot of trouble. Something like Google's Courgette (binary patch generation) could help to simplify things, but an old thread mentioned a patent suit. https://news.ycombinator.com/item?id=2576878 https://news.ycombinator.com/item?id=2576878
  - bfm 4y ago
    Shameless plug. I open-sourced https://snapdir.org/ https://snapdir.org/ to explore how to solve these exact issues. It's an early prototype to gather feedback around the ergonomics of the interface and the manifest format, and I would love to learn if people might find it useful enough for the use cases where LFS falls short.
rajatarya 4y ago
Since launching in December we’ve had a few teams use XetHub specifically as a drop-in replacement for Git LFS and their main reasons for switching are: easier to use (no .gitattributes file management), faster performance, and less storage. Love to have you try XetHub and give us your feedback!

by Zee2 4y ago

ensemblehq 4y ago

What’s the challenge with LFS?

Aeolun 4y ago
That it’s not baked right into git? Half of my challenges revolve around getting either everything out of LFS, or putting everything in LFS.
penjelly 4y ago
non-devs (designers) groan when they need to worry about LFS in my exp. Though i never understood the pain either, just sharing what I've observed
- raincole 4y ago
  Non-devs complain about any version control tho. Perforce? Plastic SCM? Does anyone really like them?
- Cthulhu_ 4y ago
  What version control systems are there for non-code? I never looked into it, I can imagine there's a whole world out there for things like written articles, books, but also video game assets like artwork, sprites, 3D models, textures, animations, etc. edit: Looks like e.g. the Unreal engine has support for Perforce and SVN (https://docs.unrealengine.com/5.0/en-US/collaboration-and-version-control-in-unreal-engine/ https://docs.unrealengine.com/5.0/en-US/collaboration-and-ve...), where they make use of locking a file when editing it to avoid merge issues with binary files. I can imagine that frequently goes wrong.
  - grrdotcloud 4y ago
    I am of the opinion the mind of the designer or creative does not think in terms of "versions". There are two states, in woodworking for example, the raw material and the final product. The mental model of "cutting longer" or rolling back does not exist. The idea of a filesystem level understanding of "undo' just has not been communicated or explored. While not Oxen related - TimeMachine seems to be the closest conceptual understanding. Snapshotting the drive, deltas, etc are generally not considered in my experience. I would be curious how larger or mature groups manage versioning.
remram 4y ago
LFS is slow (copies files during checkout, downloads/uploads files independently), inefficient (no de-duplication or delta encoding, contrary to base Git), error-prone (needs manual setup, and rewriting history if anyone gets it wrong), and otherwise not well integrated (it relies on smudge filter and other hooks, so some Git commands are aware of it and others not. You will often see the `version ... oid ...` content of the actual blob rather than the large file).
- unshavedyak 4y ago
  > and otherwise not well integrated (it relies on smudge filter and other hooks, so some Git commands are aware of it and others not. You will often see the `version ... oid ...` content of the actual blob rather than the large file). Curious, do you know the solutions here? I've looked into integrating Git LFS-like things into Git and it felt like Git doesn't give you many tools. Smudging, as hacky as it is, feels one of a very small pool of options you have. I want to say Git Annex does something slightly different. Regardless Smudging felt fine due to how difficult Git made it. Are there better ways in your mind?
  - remram 4y ago
    I don't think you can do better if you don't want to change Git, no.
    - unshavedyak 4y ago
      Seems like an odd critique of LFS then, no? Your other points are super valid, i've wanted to replace LFS with my own impl for the same reasons _(also it was originally designed for HTTP servers, ugh)_. However the integration is roughly as good as it can be, no? Or rather, Oxen would suffer the same limitations no?
      - remram 4y ago
        I'm not trying to target this critique at LFS specifically and absolve Git. It might be that LFS is a bad product because of Git's limited extension points, it remains a bad product.
olvy0 4y ago
In addition to what was said above: In later versions, LFS doesn't play nice with misconfigured internal networks based on Microsoft's Azure / Active Directory. The LFS project dropped support for NTLM (for good reasons) but it's difficult to convince IT to switch to Kerberos. Our team have been trying to do that for several weeks now. So the result is that if we want to use LFS we are stuck with old versions of LFS which still supported NTLM, which necessitate using old versions of GIT which supported this old version of LFS, else we run into all sorts of weird error messages. Also the same combination of old GIT + old LFS works perfectly on some machines, and fails with some weird error messages on others. Possibly due to another IT misconfiguration or something. I've been wanting for a while now to check if I can fork LFS and fix this, unfortunately my knowledge of golang is very sketchy. Hopefully we'll manage to convince IT to drop NTLM support from the on-premise Azure.
- e12e 4y ago
  > Hopefully we'll manage to convince IT to drop NTLM support from the on-premise Azure. What year is this? ;) On a more serious note - ntlm, really? And with azure ad, not a regular (old) on-premise domain? I know it can be hard to drag windows setups into the future (hello WINS!) - But why keep ntlm auth in a post windows 7 world? Legacy systems without ldap/radius support? Genuinely curious (and a little horrified).
  - olvy0 4y ago
    Tell me about it ;) Apparently, it's related to the few win 7 machines that are still present somewhere in the organization for various reasons. All of them are airgapped (I hope) so there's no real reason to keep NTLM, but nobody has yet made the decision. Working on getting to the guy in charge to authorize it. Sigh.
sam_goody 4y ago
I seem to always be running into git-lfs related issues. The latest include.. If one file doesn't resolve, for any reason, git-lfs will block the entire pull. No way to say "get whatever you could, and the fact that aws is not serving that file is not a blocker" Git clone does not pull the entire lfs history of an image. Lacking the entire history, lfs will fail consistently on future actions.

gschoeni 4y ago

Creator of the project here, would love to look at what an LFS style integration with git hooks could look like. Let me know if you have any ideas there!

Zee2 4y ago
Honestly, I'd just trawl the interwebs for all the reasons folks have had grief with LFS, and then see if Oxen can even do a fraction better. If it can, you'll have many billion-dollar companies that would be highly interested in your solution :) A few starting points from personal experience would be - Ensuring git-aware code editors don't try and mangle blob histories when they encounter them - Modern, high-performance blob downloads/version-resolution - Human-readable/debuggable meta-files, where it's easy to recover from bad states (instead of an opaque empty file with a GUID...?)
- yarg 4y ago
  If you want blob support, you'd probably need to introduce format specific understanding of file structures. You'd also need for the blob encoder to be exceptionally stable - if minor changes to the input lead to significant changes to output, you're going to have a lot of trouble. Something like Google's Courgette (binary patch generation) could help to simplify things, but an old thread mentioned a patent suit. https://news.ycombinator.com/item?id=2576878 https://news.ycombinator.com/item?id=2576878
- bfm 4y ago
  Shameless plug. I open-sourced https://snapdir.org/ https://snapdir.org/ to explore how to solve these exact issues. It's an early prototype to gather feedback around the ergonomics of the interface and the manifest format, and I would love to learn if people might find it useful enough for the use cases where LFS falls short.

rajatarya 4y ago

Since launching in December we’ve had a few teams use XetHub specifically as a drop-in replacement for Git LFS and their main reasons for switching are: easier to use (no .gitattributes file management), faster performance, and less storage. Love to have you try XetHub and give us your feedback!