10 ms·

Oxen.ai: Fast Unstructured Data Version Control

177 points by sbt567 4y ago

sbt567 4y ago
Web hub (similar to GitHub): https://www.oxen.ai/ https://www.oxen.ai/
Zee2 4y ago
Oh man, if this could plug into git and be a LFS replacement, that would be awesome. I work in a field where folks run into situations where they think they need LFS, and rarely does it work out well. If someone can figure out an ergonomic and durable LFS-like blob versioning system that can align with git histories, that would be incredible.
- ensemblehq 4y ago
  What’s the challenge with LFS?
  - Aeolun 4y ago
    That it’s not baked right into git? Half of my challenges revolve around getting either everything out of LFS, or putting everything in LFS.
  - penjelly 4y ago
    non-devs (designers) groan when they need to worry about LFS in my exp. Though i never understood the pain either, just sharing what I've observed
    - raincole 4y ago
      Non-devs complain about any version control tho. Perforce? Plastic SCM? Does anyone really like them?
    - Cthulhu_ 4y ago
      What version control systems are there for non-code? I never looked into it, I can imagine there's a whole world out there for things like written articles, books, but also video game assets like artwork, sprites, 3D models, textures, animations, etc. edit: Looks like e.g. the Unreal engine has support for Perforce and SVN (https://docs.unrealengine.com/5.0/en-US/collaboration-and-version-control-in-unreal-engine/ https://docs.unrealengine.com/5.0/en-US/collaboration-and-ve...), where they make use of locking a file when editing it to avoid merge issues with binary files. I can imagine that frequently goes wrong.
      - grrdotcloud 4y ago
        I am of the opinion the mind of the designer or creative does not think in terms of "versions". There are two states, in woodworking for example, the raw material and the final product. The mental model of "cutting longer" or rolling back does not exist. The idea of a filesystem level understanding of "undo' just has not been communicated or explored. While not Oxen related - TimeMachine seems to be the closest conceptual understanding. Snapshotting the drive, deltas, etc are generally not considered in my experience. I would be curious how larger or mature groups manage versioning.
  - remram 4y ago
    LFS is slow (copies files during checkout, downloads/uploads files independently), inefficient (no de-duplication or delta encoding, contrary to base Git), error-prone (needs manual setup, and rewriting history if anyone gets it wrong), and otherwise not well integrated (it relies on smudge filter and other hooks, so some Git commands are aware of it and others not. You will often see the `version ... oid ...` content of the actual blob rather than the large file).
    - unshavedyak 4y ago
      > and otherwise not well integrated (it relies on smudge filter and other hooks, so some Git commands are aware of it and others not. You will often see the `version ... oid ...` content of the actual blob rather than the large file). Curious, do you know the solutions here? I've looked into integrating Git LFS-like things into Git and it felt like Git doesn't give you many tools. Smudging, as hacky as it is, feels one of a very small pool of options you have. I want to say Git Annex does something slightly different. Regardless Smudging felt fine due to how difficult Git made it. Are there better ways in your mind?
      - remram 4y ago
        I don't think you can do better if you don't want to change Git, no.
        unshavedyak 4y ago
        Seems like an odd critique of LFS then, no? Your other points are super valid, i've wanted to replace LFS with my own impl for the same reasons _(also it was originally designed for HTTP servers, ugh)_. However the integration is roughly as good as it can be, no? Or rather, Oxen would suffer the same limitations no?
        remram 4y ago
        I'm not trying to target this critique at LFS specifically and absolve Git. It might be that LFS is a bad product because of Git's limited extension points, it remains a bad product.
  - olvy0 4y ago
    In addition to what was said above: In later versions, LFS doesn't play nice with misconfigured internal networks based on Microsoft's Azure / Active Directory. The LFS project dropped support for NTLM (for good reasons) but it's difficult to convince IT to switch to Kerberos. Our team have been trying to do that for several weeks now. So the result is that if we want to use LFS we are stuck with old versions of LFS which still supported NTLM, which necessitate using old versions of GIT which supported this old version of LFS, else we run into all sorts of weird error messages. Also the same combination of old GIT + old LFS works perfectly on some machines, and fails with some weird error messages on others. Possibly due to another IT misconfiguration or something. I've been wanting for a while now to check if I can fork LFS and fix this, unfortunately my knowledge of golang is very sketchy. Hopefully we'll manage to convince IT to drop NTLM support from the on-premise Azure.
    - e12e 4y ago
      > Hopefully we'll manage to convince IT to drop NTLM support from the on-premise Azure. What year is this? ;) On a more serious note - ntlm, really? And with azure ad, not a regular (old) on-premise domain? I know it can be hard to drag windows setups into the future (hello WINS!) - But why keep ntlm auth in a post windows 7 world? Legacy systems without ldap/radius support? Genuinely curious (and a little horrified).
      - olvy0 4y ago
        Tell me about it ;) Apparently, it's related to the few win 7 machines that are still present somewhere in the organization for various reasons. All of them are airgapped (I hope) so there's no real reason to keep NTLM, but nobody has yet made the decision. Working on getting to the guy in charge to authorize it. Sigh.
  - sam_goody 4y ago
    I seem to always be running into git-lfs related issues. The latest include.. If one file doesn't resolve, for any reason, git-lfs will block the entire pull. No way to say "get whatever you could, and the fact that aws is not serving that file is not a blocker" Git clone does not pull the entire lfs history of an image. Lacking the entire history, lfs will fail consistently on future actions.
- gschoeni 4y ago
  Creator of the project here, would love to look at what an LFS style integration with git hooks could look like. Let me know if you have any ideas there!
  - Zee2 4y ago
    Honestly, I'd just trawl the interwebs for all the reasons folks have had grief with LFS, and then see if Oxen can even do a fraction better. If it can, you'll have many billion-dollar companies that would be highly interested in your solution :) A few starting points from personal experience would be - Ensuring git-aware code editors don't try and mangle blob histories when they encounter them - Modern, high-performance blob downloads/version-resolution - Human-readable/debuggable meta-files, where it's easy to recover from bad states (instead of an opaque empty file with a GUID...?)
    - yarg 4y ago
      If you want blob support, you'd probably need to introduce format specific understanding of file structures. You'd also need for the blob encoder to be exceptionally stable - if minor changes to the input lead to significant changes to output, you're going to have a lot of trouble. Something like Google's Courgette (binary patch generation) could help to simplify things, but an old thread mentioned a patent suit. https://news.ycombinator.com/item?id=2576878 https://news.ycombinator.com/item?id=2576878
    - bfm 4y ago
      Shameless plug. I open-sourced https://snapdir.org/ https://snapdir.org/ to explore how to solve these exact issues. It's an early prototype to gather feedback around the ergonomics of the interface and the manifest format, and I would love to learn if people might find it useful enough for the use cases where LFS falls short.
- rajatarya 4y ago
  Since launching in December we’ve had a few teams use XetHub specifically as a drop-in replacement for Git LFS and their main reasons for switching are: easier to use (no .gitattributes file management), faster performance, and less storage. Love to have you try XetHub and give us your feedback!
zachthewf 4y ago
What are the differences between this and DVC?
- gschoeni 4y ago
  Creators of the project here, the first difference is the raw speed if you have many images, video, audio, files etc. We did some benchmarking here: https://github.com/Oxen-AI/oxen-release/blob/main/Performance.md https://github.com/Oxen-AI/oxen-release/blob/main/Performanc... ~TLDR~ 200k+ images from the CelebA dataset take ~6 minutes to add, commit, push into Oxen. Same dataset takes ~3 hours for DVC.
  - whatshisface 4y ago
    Hey this is a minor thing but could you re-render that graph to a higher contrast color scheme than light grey on white? :)
    - gschoeni 4y ago
      Good call out, didn't realize how hard to see they were. Updated!
  - e12e 4y ago
    Maybe I missed it - but what is the approximate size of the dataset in GB? Would be interesting to see rclone.org (and maybe s4cmd) (should be faster access to s3?) - and also mercurial and bitkeeper.org - both should behave a little better than git on large files?
banga 4y ago
How does this compare with other systems, like DVC (https://dvc.org/ https://dvc.org/) for example?
- rsfern 4y ago
  Oxen seems more like git (with GitHub integration (Oxenhub)) for ML datasets, where DVC is a bit bit more like make (with S3, LFS, etc integration) for ML datasets. It seems like Oxen has finer granularity version control and diff capability, but as far as I can tell doesn’t have as many features to track and version derived data along with the code that produced it (like `dvc repro`)
  - gschoeni 4y ago
    We definitely have some of these features on our roadmap! Anything particularly helpful in DVC's workflow that you think we should prioritize?
    - rsfern 4y ago
      Cool! To be honest I don’t really use dvc much, but the project version control features are what really interest me. I like how data pipelines help align versioned artifacts like model checkpoints and visualizations with the datasets and code that produced. I work as a computational science and that sort of reproducibility tool is really important, and a lot of us don’t have the best software engineering skills/discipline. From your readme it seems like the oxen repo and software project repo are not as closely coupled as in dvc? It seemed like in the current state of oxen, you could do something similar with make files and oxen tracking? Oxen seems really good for longer lived data and computational science projects, where dvc seems more oriented just at analysis projects. I have a project that I want to try it out on :)
    - michaelmior 4y ago
      One thing I love about DVC is that it doesn't need its own server. I can just push/pull files via SSH. I don't really want one more service that I need to keep running. I also happen to have a lot of space available to me on a server I can't install extra services on, so oxen requiring that is a deal breaker for me.
keyle 4y ago
Link to the actual project source https://github.com/Oxen-AI/Oxen https://github.com/Oxen-AI/Oxen
jiangplus 4y ago
How does it compare to dolt? https://github.com/dolthub/dolt https://github.com/dolthub/dolt
- gschoeni 4y ago
  Dolthub is more about versioning structured SQL data tables. Oxen handles more large unstructured sets of images, video, audio, text, and DataFrame (parquet, csv, arrow, etc) files
JamesHinnek 4y ago
The comparison with DVC is biased https://github.com/Oxen-AI/oxen-release/blob/main/Performance.md https://github.com/Oxen-AI/oxen-release/blob/main/Performanc... I'd nowhere near the same performance with oxen. The analysis is very biased to help Oxen. I wish people had more integrity before trying so hard to push a half-baked product into the market.
- ttoks 4y ago
  [dead]
- ibejoeb 4y ago
  I'm reading it, and it's not apparent what that bias is. It's init, add, and push on the same data set.
  - JamesHinnek 4y ago
    It vastly depends on the remote. I'm testing Oxen (on their own hub), it has barely the same speed as dvc on S3. If I was trying to push the results to a direction being pro DVC, I would compare pushing to a local minio, I could claim 10x more speed.
    - barcoded 4y ago
      Exactly, this is because of the coupling with DAGShub as remote. Full disclosure - I'm associated with iterative (DVC creators) - we have nothing to do with DAGShub I'm afraid. They don't consult or collaborate with us at all (to this date) about how to build or optimize their server or workflows for their "hub"
aldanor 4y ago
Being realistic here, 3rd party provider for data handling will be a no-go for many firms, for infosec reasons. Whereas a hub with no ui might also be a no-go for convenience reasons. I understand that oxenhub is a way to monetise the project but is there a self-hosted 'enterprise' version of that anywhere in the plans?
JamesHinnek 4y ago
Please implement account deletion, you are violating people's privacy, GDPR and this is a dark pattern.
- ngould 4y ago
  You're a dark pattern
stvnbn 4y ago
why not just use git?
- progx 4y ago
  Fast, huge file support included.
- flohofwoe 4y ago
  I guess you never tried using git with large repos (> 100 GB or so) which are predominantly populated with big (>100 MB) binary files? On good days it's just unusably slow. On bad days it completely falls apart.
TnS-hun 4y ago
Any plans for adding exclusive locking and option to delete old versions of a file? These are really important if working with unmergeable, large files.
mmq 4y ago
On your github org, twitter link is pointing to the wrong handle: @oxen_ao -> @oxen_ai
- gorbypark 4y ago
  The Privacy Policy and the Terms and Conditions also just link to https://www.oxen.ai/repositories https://www.oxen.ai/repositories
  - gschoeni 4y ago
    Oops, thanks for catching! Updated.
- gschoeni 4y ago
  Good catch, thank you! Updated
verdverm 4y ago
How does this compare to Pachyderm.io?
rajatarya 4y ago
Great to see more people in this space! We are the authors of XetHub (posted in Dec ‘22, ShowHN: https://news.ycombinator.com/item?id=33969908 https://news.ycombinator.com/item?id=33969908) and also think a git-like workflow is perfect for ML dataset management, except that we actually integrate with git (like LFS). <A quick benchmark suggests we are 2x your published performance!>
snthpy 4y ago
I see there are already a bunch of questions about how this compares to other tools like DVC, dolt, pachyderm.io and LFS? I would just like to add one to that list: How does this compare to lakeFS?

177 points by sbt567 4y ago

Web hub (similar to GitHub): https://www.oxen.ai/ https://www.oxen.ai/

Zee2 4y ago

Oh man, if this could plug into git and be a LFS replacement, that would be awesome. I work in a field where folks run into situations where they think they need LFS, and rarely does it work out well. If someone can figure out an ergonomic and durable LFS-like blob versioning system that can align with git histories, that would be incredible.

ensemblehq 4y ago
What’s the challenge with LFS?
- Aeolun 4y ago
  That it’s not baked right into git? Half of my challenges revolve around getting either everything out of LFS, or putting everything in LFS.
- penjelly 4y ago
  non-devs (designers) groan when they need to worry about LFS in my exp. Though i never understood the pain either, just sharing what I've observed
  - raincole 4y ago
    Non-devs complain about any version control tho. Perforce? Plastic SCM? Does anyone really like them?
  - Cthulhu_ 4y ago
    What version control systems are there for non-code? I never looked into it, I can imagine there's a whole world out there for things like written articles, books, but also video game assets like artwork, sprites, 3D models, textures, animations, etc. edit: Looks like e.g. the Unreal engine has support for Perforce and SVN (https://docs.unrealengine.com/5.0/en-US/collaboration-and-version-control-in-unreal-engine/ https://docs.unrealengine.com/5.0/en-US/collaboration-and-ve...), where they make use of locking a file when editing it to avoid merge issues with binary files. I can imagine that frequently goes wrong.
    - grrdotcloud 4y ago
      I am of the opinion the mind of the designer or creative does not think in terms of "versions". There are two states, in woodworking for example, the raw material and the final product. The mental model of "cutting longer" or rolling back does not exist. The idea of a filesystem level understanding of "undo' just has not been communicated or explored. While not Oxen related - TimeMachine seems to be the closest conceptual understanding. Snapshotting the drive, deltas, etc are generally not considered in my experience. I would be curious how larger or mature groups manage versioning.
- remram 4y ago
  LFS is slow (copies files during checkout, downloads/uploads files independently), inefficient (no de-duplication or delta encoding, contrary to base Git), error-prone (needs manual setup, and rewriting history if anyone gets it wrong), and otherwise not well integrated (it relies on smudge filter and other hooks, so some Git commands are aware of it and others not. You will often see the `version ... oid ...` content of the actual blob rather than the large file).
  - unshavedyak 4y ago
    > and otherwise not well integrated (it relies on smudge filter and other hooks, so some Git commands are aware of it and others not. You will often see the `version ... oid ...` content of the actual blob rather than the large file). Curious, do you know the solutions here? I've looked into integrating Git LFS-like things into Git and it felt like Git doesn't give you many tools. Smudging, as hacky as it is, feels one of a very small pool of options you have. I want to say Git Annex does something slightly different. Regardless Smudging felt fine due to how difficult Git made it. Are there better ways in your mind?
    - remram 4y ago
      I don't think you can do better if you don't want to change Git, no.
      - unshavedyak 4y ago
        Seems like an odd critique of LFS then, no? Your other points are super valid, i've wanted to replace LFS with my own impl for the same reasons _(also it was originally designed for HTTP servers, ugh)_. However the integration is roughly as good as it can be, no? Or rather, Oxen would suffer the same limitations no?
        remram 4y ago
        I'm not trying to target this critique at LFS specifically and absolve Git. It might be that LFS is a bad product because of Git's limited extension points, it remains a bad product.
- olvy0 4y ago
  In addition to what was said above: In later versions, LFS doesn't play nice with misconfigured internal networks based on Microsoft's Azure / Active Directory. The LFS project dropped support for NTLM (for good reasons) but it's difficult to convince IT to switch to Kerberos. Our team have been trying to do that for several weeks now. So the result is that if we want to use LFS we are stuck with old versions of LFS which still supported NTLM, which necessitate using old versions of GIT which supported this old version of LFS, else we run into all sorts of weird error messages. Also the same combination of old GIT + old LFS works perfectly on some machines, and fails with some weird error messages on others. Possibly due to another IT misconfiguration or something. I've been wanting for a while now to check if I can fork LFS and fix this, unfortunately my knowledge of golang is very sketchy. Hopefully we'll manage to convince IT to drop NTLM support from the on-premise Azure.
  - e12e 4y ago
    > Hopefully we'll manage to convince IT to drop NTLM support from the on-premise Azure. What year is this? ;) On a more serious note - ntlm, really? And with azure ad, not a regular (old) on-premise domain? I know it can be hard to drag windows setups into the future (hello WINS!) - But why keep ntlm auth in a post windows 7 world? Legacy systems without ldap/radius support? Genuinely curious (and a little horrified).
    - olvy0 4y ago
      Tell me about it ;) Apparently, it's related to the few win 7 machines that are still present somewhere in the organization for various reasons. All of them are airgapped (I hope) so there's no real reason to keep NTLM, but nobody has yet made the decision. Working on getting to the guy in charge to authorize it. Sigh.
- sam_goody 4y ago
  I seem to always be running into git-lfs related issues. The latest include.. If one file doesn't resolve, for any reason, git-lfs will block the entire pull. No way to say "get whatever you could, and the fact that aws is not serving that file is not a blocker" Git clone does not pull the entire lfs history of an image. Lacking the entire history, lfs will fail consistently on future actions.
gschoeni 4y ago
Creator of the project here, would love to look at what an LFS style integration with git hooks could look like. Let me know if you have any ideas there!
- Zee2 4y ago
  Honestly, I'd just trawl the interwebs for all the reasons folks have had grief with LFS, and then see if Oxen can even do a fraction better. If it can, you'll have many billion-dollar companies that would be highly interested in your solution :) A few starting points from personal experience would be - Ensuring git-aware code editors don't try and mangle blob histories when they encounter them - Modern, high-performance blob downloads/version-resolution - Human-readable/debuggable meta-files, where it's easy to recover from bad states (instead of an opaque empty file with a GUID...?)
  - yarg 4y ago
    If you want blob support, you'd probably need to introduce format specific understanding of file structures. You'd also need for the blob encoder to be exceptionally stable - if minor changes to the input lead to significant changes to output, you're going to have a lot of trouble. Something like Google's Courgette (binary patch generation) could help to simplify things, but an old thread mentioned a patent suit. https://news.ycombinator.com/item?id=2576878 https://news.ycombinator.com/item?id=2576878
  - bfm 4y ago
    Shameless plug. I open-sourced https://snapdir.org/ https://snapdir.org/ to explore how to solve these exact issues. It's an early prototype to gather feedback around the ergonomics of the interface and the manifest format, and I would love to learn if people might find it useful enough for the use cases where LFS falls short.
rajatarya 4y ago
Since launching in December we’ve had a few teams use XetHub specifically as a drop-in replacement for Git LFS and their main reasons for switching are: easier to use (no .gitattributes file management), faster performance, and less storage. Love to have you try XetHub and give us your feedback!

zachthewf 4y ago

What are the differences between this and DVC?

gschoeni 4y ago
Creators of the project here, the first difference is the raw speed if you have many images, video, audio, files etc. We did some benchmarking here: https://github.com/Oxen-AI/oxen-release/blob/main/Performance.md https://github.com/Oxen-AI/oxen-release/blob/main/Performanc... ~TLDR~ 200k+ images from the CelebA dataset take ~6 minutes to add, commit, push into Oxen. Same dataset takes ~3 hours for DVC.
- whatshisface 4y ago
  Hey this is a minor thing but could you re-render that graph to a higher contrast color scheme than light grey on white? :)
  - gschoeni 4y ago
    Good call out, didn't realize how hard to see they were. Updated!
- e12e 4y ago
  Maybe I missed it - but what is the approximate size of the dataset in GB? Would be interesting to see rclone.org (and maybe s4cmd) (should be faster access to s3?) - and also mercurial and bitkeeper.org - both should behave a little better than git on large files?

banga 4y ago

How does this compare with other systems, like DVC (https://dvc.org/ https://dvc.org/) for example?

rsfern 4y ago
Oxen seems more like git (with GitHub integration (Oxenhub)) for ML datasets, where DVC is a bit bit more like make (with S3, LFS, etc integration) for ML datasets. It seems like Oxen has finer granularity version control and diff capability, but as far as I can tell doesn’t have as many features to track and version derived data along with the code that produced it (like `dvc repro`)
- gschoeni 4y ago
  We definitely have some of these features on our roadmap! Anything particularly helpful in DVC's workflow that you think we should prioritize?
  - rsfern 4y ago
    Cool! To be honest I don’t really use dvc much, but the project version control features are what really interest me. I like how data pipelines help align versioned artifacts like model checkpoints and visualizations with the datasets and code that produced. I work as a computational science and that sort of reproducibility tool is really important, and a lot of us don’t have the best software engineering skills/discipline. From your readme it seems like the oxen repo and software project repo are not as closely coupled as in dvc? It seemed like in the current state of oxen, you could do something similar with make files and oxen tracking? Oxen seems really good for longer lived data and computational science projects, where dvc seems more oriented just at analysis projects. I have a project that I want to try it out on :)
  - michaelmior 4y ago
    One thing I love about DVC is that it doesn't need its own server. I can just push/pull files via SSH. I don't really want one more service that I need to keep running. I also happen to have a lot of space available to me on a server I can't install extra services on, so oxen requiring that is a deal breaker for me.

keyle 4y ago

Link to the actual project source https://github.com/Oxen-AI/Oxen https://github.com/Oxen-AI/Oxen

jiangplus 4y ago

How does it compare to dolt? https://github.com/dolthub/dolt https://github.com/dolthub/dolt

gschoeni 4y ago
Dolthub is more about versioning structured SQL data tables. Oxen handles more large unstructured sets of images, video, audio, text, and DataFrame (parquet, csv, arrow, etc) files

JamesHinnek 4y ago

The comparison with DVC is biased https://github.com/Oxen-AI/oxen-release/blob/main/Performance.md https://github.com/Oxen-AI/oxen-release/blob/main/Performanc... I'd nowhere near the same performance with oxen. The analysis is very biased to help Oxen. I wish people had more integrity before trying so hard to push a half-baked product into the market.

ttoks 4y ago
[dead]
ibejoeb 4y ago
I'm reading it, and it's not apparent what that bias is. It's init, add, and push on the same data set.
- JamesHinnek 4y ago
  It vastly depends on the remote. I'm testing Oxen (on their own hub), it has barely the same speed as dvc on S3. If I was trying to push the results to a direction being pro DVC, I would compare pushing to a local minio, I could claim 10x more speed.
  - barcoded 4y ago
    Exactly, this is because of the coupling with DAGShub as remote. Full disclosure - I'm associated with iterative (DVC creators) - we have nothing to do with DAGShub I'm afraid. They don't consult or collaborate with us at all (to this date) about how to build or optimize their server or workflows for their "hub"

aldanor 4y ago

Being realistic here, 3rd party provider for data handling will be a no-go for many firms, for infosec reasons. Whereas a hub with no ui might also be a no-go for convenience reasons. I understand that oxenhub is a way to monetise the project but is there a self-hosted 'enterprise' version of that anywhere in the plans?

JamesHinnek 4y ago

Please implement account deletion, you are violating people's privacy, GDPR and this is a dark pattern.

ngould 4y ago
You're a dark pattern

stvnbn 4y ago

why not just use git?

progx 4y ago
Fast, huge file support included.
flohofwoe 4y ago
I guess you never tried using git with large repos (> 100 GB or so) which are predominantly populated with big (>100 MB) binary files? On good days it's just unusably slow. On bad days it completely falls apart.

TnS-hun 4y ago

Any plans for adding exclusive locking and option to delete old versions of a file? These are really important if working with unmergeable, large files.

mmq 4y ago

On your github org, twitter link is pointing to the wrong handle: @oxen_ao -> @oxen_ai

gorbypark 4y ago
The Privacy Policy and the Terms and Conditions also just link to https://www.oxen.ai/repositories https://www.oxen.ai/repositories
- gschoeni 4y ago
  Oops, thanks for catching! Updated.
gschoeni 4y ago
Good catch, thank you! Updated

verdverm 4y ago

How does this compare to Pachyderm.io?

rajatarya 4y ago

Great to see more people in this space! We are the authors of XetHub (posted in Dec ‘22, ShowHN: https://news.ycombinator.com/item?id=33969908 https://news.ycombinator.com/item?id=33969908) and also think a git-like workflow is perfect for ML dataset management, except that we actually integrate with git (like LFS). <A quick benchmark suggests we are 2x your published performance!>

snthpy 4y ago

I see there are already a bunch of questions about how this compares to other tools like DVC, dolt, pachyderm.io and LFS? I would just like to add one to that list: How does this compare to lakeFS?