24 ms·

Show HN: We scaled Git to support 1 TB repos

• ledauphin 4y ago
  The link takes me to a login page. It would be nice to see that fixed to somehow match the title.
  • reverius42 4y ago
    Visit https://xetdata.com https://xetdata.com for more info! (Sorry, can't edit the post link now.)
• Aperocky 4y ago
  I see a lot of reasons to version code. I see far less reasons to version data, in fact, I find reasons against versioning data and storing them in diffs.
  • ch71r22 4y ago
    What are the reasons against?
    • ltbarcly3 4y ago
      The lack of reasons for doing it IS the reason against. GIT isn't a magic 'good way' to store arbitrary data, it's a good way to collaborate on projects implemented using most programming languages which store code as plain text broken into short lines, where edits to non-sequential lines can generally be applied concurrently without careful human verification. That is an extremely specific use case, and anything outside of that very specific use case leaves git terrible, inefficient, and gives almost no benefit despite huge problems. People in ML ops use git because they aren't very sophisticated with programming professionally and they have git available to them and they haven't run into the consequences of using it to store large binary blobs, namely that it becomes impossible to live with eventually and wastes a huge amount of time and space. ML didn't invent the need for large artifacts that can't be versioned in source control but must be versioned with it, but they don't know that because they are new to professional programming and aren't familiar with how it's done.
      • ylow 4y ago
        Indeed, there is a lot of pain if you actually try to store large binary data in git. But we managed to make that work! So a question worth asking is how might things change IF you can store large binary data in git??
        • ltbarcly3 4y ago
          I think this is a foot-gun, it's a bad idea even if it works great, and I doubt it works very well. You should manage your build artifacts explicitly, not just jam them in git along with the code that generates them because you are already using it and you haven't thought it through.
          • 4y ago
• V1ndaar 4y ago
  You say you support up to 1TB repositories, but from your pricing page all I see is the free tier for up to 20GB and one for teams. The latter doesn't have a price and only a contact option and I assume likely will be too expensive for an individual. As someone who'd love to put their data into a git like system, this sounds pretty interesting. Aside from not offering a tier for someone like me who would maybe have a couple of repositories of size O(250GB) it's unclear how e.g. bandwidth would work & whether other people could simply mount and clone the full repo if desired for free etc.
  • rajatarya 4y ago
    XetHub Co-founder here. We are still trying to figure out pricing and would love to understand what sort of pricing tier would work for you. In general, we are thinking about usage-based pricing (which would include bandwidth and storage) - what are your thoughts for that? Also, where would you be mounting your repos from? We have local caching options that can greatly reduce the overall bandwidth needed to support data center workloads.
    • blagie 4y ago
      My preferences on pricing. First, it's all open-source, so I can take it and run it. Second, you provide a hosted service, and by virtue of being the author, you're the default SaaS host. You charge a premium over AWS fees for self-hosting, which works out to: 1. Enough to sustain you. 2. Less than the cost of doing dev-ops myself (AWS fees + engineer). 3. A small premium over potential cut-rate competitors. You offer value-added premium services too. Whether that's economically viable, I don't know.
      • blagie 4y ago
        As a footnote, the rationale: 1. I'm unlikely to adopt something proprietary for this sort of use. Lock-in is bad, but it's especially with a startup which can disappear tomorrow or pivot who is holding my key data. Open-source means if you disappear, I'm alive. I don't trust you, and open-source mostly means I don't need to. 2. With open-source, pricing which is more than the cost of AWS + engineer makes no sense. I'd rather host myself. However, the labor costs means that AWS + engineer gives a lot of potential profit margin for you. I'd much rather not run servers myself. 3. A cut-rate competitor will have similar per-customer cost structure as you, but you'll have somewhat higher fixed costs. For me, paying a little bit more for the reliability of going with the most competent vendor is an obvious choice (which you would be, by virtue of having written it). I wouldn't consider a cut-rate vendor unless the savings was very significant. 4. Not in my current job, but in past jobs, I'd gladly pay for service and support on top of that. A lot of things are cheaper for you to do (or explain) as the author / expert, than for my guys to figure out themselves. For this to work requires a certain economy-of-scale. That requires deep VC pockets to get to profitability, or a good beachhead (e.g. a single big customer). There are many single big customers, but I have no idea how you'd build that connection. Most are in oddball industries, and not companies you'd think of. For example, a few years back, I interacted with a major military contractor who specializes in manufacturing, and has no competence in technology. They did many billions in business, and had just paid a few million for a semi-incompetent tech consulting firm as an acqui-hire to try to build basic tech expertise. Outsourcing everything to you would have been a far better decision for them (and for many companies like them) if they could find you and vet you, and vice-versa (probably with a strategic investment as well). They were very good at what they did, but what they did was very much not tech or software.
• amadvance 4y ago
  How data is split in chunks ? Just curious.
  • deleted 4y ago
    [deleted]
  • sesm 4y ago
    They mention 'content-defined chunking', but it as far as understand it requires different chunking algorithms for different content types. Does it support plugins for chunking different file formats?
    • ylow 4y ago
      Today we just have a variation of FastCDC in production, but we have alternate experimental chunkers for some file formats (ex: a heuristic chunker for CSV files that will enable almost free subsampling). Hope to have them enter production in the next 6 months.
      • sesm 4y ago
        That's interesting. Can a CSV chunker make adding a column not affect all of the chunks?
        • ylow 4y ago
          The simplest really is to chunk row-wise so adding columns will unfortunately rewrite all the chunks. If you have a parquet file, adding columns will be cheap.
  • ylow 4y ago
    CEO/Cofounder here! Content defined chunking. Specifically a variation of FastCDC. We have a paper coming out soon with a lot more technical details.
• JZL003 4y ago
  I also have a lot of issues with versioning data. But look at git annex - it's free, self hosted and has a very easy underlying data structure [1]. So I don't even use the magic commands it has for remote data mounting/multi-device coordination, just backup using basic S3 commands and can use rclone mounting. Very robust, open source, and useful [1] When you run `git annex add` it hashes the file and moves the original file to a `.git/annex/data` folder under the hash/content addressable file system, like git. Then it replaces the original file with a symlink to this hashed file path. The file is marked as read only, so any command in any language which tries to write to it will error (you can always `git annex unlock` so you can write to it). If you have duplicated files, they easily point to the same hashed location. As long as you git push normally and back up the `.git/annex/data` you're totally version controlled, and you can share the subset of files as needed
  • kspacewalk2 4y ago
    Sounds like `git annex` is file-level deduplication, whereas this tool is block-level, but with some intelligent, context-specific way of defining how to split up the data (i.e. Content-Defined Chunking). For data management/versioning, that's usually a big difference.
    • rajatarya 4y ago
      XetHub Co-founder here. Yes, one illustrative example of the difference is: Imagine you have a 500MB file (lastmonth.csv) where every day 1MB is changed. With file-based deduplication every day 500MB will be uploaded, and all clones of the repo will need to download 500MB. With block-based deduplication, only around the 1MB that changed is uploaded and downloaded.
      • civilized 4y ago
        Does that work equally well whether the changes are primarily row-based or primarily column-based?
        • 4y ago
• wnzl 4y ago
  Just in case if you are wondering about alternatives: there is Unity’s Plastic https://unity.com/products/plastic-scm https://unity.com/products/plastic-scm which happens to use bidirectional sync with git. I’m curious how this solution compares to it! I’ll definitely give it a try over the weekend!
  • ziml77 4y ago
    I was already upset about Codice Software pulling Semantic Merge and only making it available as an integrated part of Plastic SCM. Now that I see the reason such a useful tool was taken away was to stuff the pockets of a large company, I'm fuming. I know that they're well within their rights to do this as they only ever offered subscription licensing for Semantic Merge, but that doesn't make it suck less to lose access.
• chubot 4y ago
  Can it be used to store container images (Docker)? As far as I remember they are just compressed tar files. Does the compression defeat Xet's own chunking? Can you sync to another machine without Xethub ? How about cleaning up old files?
  • ylow 4y ago
    Yeah... The compression does defeat the chunking (your mileage may vary. We do a small amount of dedupe in some experiments but never quite investigated it in detail.). That said, we have experimental preprocessors / chunkers that are file-type specific that we could potentially do something about tar.gz. Not something we have explored much yet.
• Game_Ender 4y ago
  The tl;dr is that "xet" is like GitLFS (it stores pointers in Git, with the data in a remote server and uses smudge filters to make this transparent) with some additional features: - Automatically includes all files >256KB in size - By default data is de-duplicated 16KB chunks instead of whole files (with the ability to customize this per file type). - Has a "mount" command to allow read-only browse without downloading When launching on HN it would be better if the team was a bit more transparent with the internals. I get that "we made a better GitLFS" doesn't market as well. But you can couple that with a credible vision and story about how you are a better and where you are headed next. Instead this is mostly closer to market speak of "trust our magic solution to solve your problem".
  • nightpool 4y ago
    These details seemed.... really clear to me from the post the OP made? Did you just not read it, or have they updated it since you commented? (excerpt from the OP post: > Unlike Git LFS, we don’t just store the files. We use content-defined chunking and Merkle Trees to dedupe against everything in history. This allows small changes in large files to be stored compactly. Read more here: https://xethub.com/assets/docs/how-xet-deduplication-works https://xethub.com/assets/docs/how-xet-deduplication-works)
• amelius 4y ago
  Does this fix the problem that Git becomes unreasonably slow when you have large binary files in the repo? Also, why can't Git show me an accurate progress-bar while fetching?
  • reverius42 4y ago
    Mostly! (At the moment, it doesn't fully fix the slowdown associated with storing large binary files, but reduces it by 90-99%. We're working on improving to closer to 100% that by moving even the Merkle Tree storage outside the git repo contents.) As for why git can't show you an accurate progress bar while fetching (specifically when using an extension like git-lfs or git-xet), this has to do with the way git extensions work -- each file gets "cleaned" by the extension through a Unix pipe, and the protocol for that is too simple to reflect progress information back to the user. In git-xet, we do write a percent-complete to stdout so you get some more info (but a real progress bar would be nice).
• dandigangi 4y ago
  One monorepo to rule them all and the in the darkness, pull them. - Gandalf, probably
  • irrational 4y ago
    And in the darkness merge conflicts.
• web007 4y ago
  Please consider https://sso.tax/ https://sso.tax/ before making that an "enterprise" feature.
  • IshKebab 4y ago
    I mean yeah, that's working as intended surely? Some of those price differences are pretty egregious but in general companies have to actually make money, and charging more for features that are mainly needed by richer customers is a very obvious thing to do.
    • mdaniel 4y ago
      I believe the counter-argument is that they should charge for features but that security should be available to anyone. Imagine if "passwords longer than 6 chars: now only $8/mo!" That goes double for products where paying for "enterprise" is only to get SAML, which at least in my experience causes me to go shopping for an entirely different product because I view it as extortion
      • IshKebab 4y ago
        Security is available for everyone. It's centralised security that can be easily managed by IT that isn't. I don't see an issue with charging more for SSO though as I said some of the prices are egregious.
  • Alifatisk 4y ago
    Very sad to see Bitwarden in this list
• Izmaki 4y ago
  If I had to "version control" a 1 TB large repo - and assuming I wouldn't quit in anger - I would use a tool which is built for this kind of need and has been used in the industry for decades: Perforce.
  • tinco 4y ago
    So, you wouldn't consider using a new tool that someone developed to solve the same problem despite an older solution already existing? Your advice to that someone is to just use the old solution?
    • TylerE 4y ago
      When the new solution involves voluntary use of git? Not just yea, but hell yes. I hate git.
      • xur17 4y ago
        Why do you hate git? I've been pretty happy with it for code, and wouldn't mind being able to use it for data repositories as well.
        • TylerE 4y ago
          Is it really worth re-hashing at this point? Reams have been written about the UX
          • xur17 4y ago
            It's used by the vast majority of software engineers, so apparently it's "good enough".
            • hinkley 4y ago
              Don’t ascribe positive feelings to popularity. I’m only using git until the moment there’s a viable alternative written by someone who knows what DX is.
• blobbers 4y ago
  ... why do you have 1TB of source code (you don't! mandatory hacker snark) Is git really supposed to be used for data? Or is this just a git-like interface for source control on data?
  • stevelacy 4y ago
    Game development, especially Unreal engine, can produce repos in excess of 1TB. Git LFS is used extensively for binary file support.
  • IshKebab 4y ago
    Git is only not "supposed" to be used for data because it doesn't work very well with data by default. Not because that's not a useful and sensible thing to want from a VCS.
    • TillE 4y ago
      It's a fundamentally bad idea because of how any DVCS works. You really don't want to be dragging around gigabytes of obsolete data forever. Something like git-lfs is the appropriate solution. You need a little bit of centralization.
      • IshKebab 4y ago
        Because of how Git's current implementation of DVS works. There's nothing fundamental about it. Git already supports partial clones and on-demand checkouts in some ways, it's just not very ergonomic. All that's really needed is a way to mark individual files as lazily fetched from a remote only when needed. LFS is a hacky substandard way to emulate that behaviour. It should be built in to Git.
• bastardoperator 4y ago
  I actually encountered a 4TB git repo. After pulling all the binary shit out of it the repo was actually 200MB. Anything that promotes treating git like a filesystem is a bad idea in my opinion.
  • frognumber 4y ago
    Yes... and no. The git userspace is horrible for this. The git data model is wonderful. The git userspace would need to be able to easily: 1. Not grab all files 2. Got grab the whole version history ... and that's more-or-less it. At that point, it'd do great with large files.
    • ylow 4y ago
      Exactly for the giant repo use case, we have a mount feature that will let you get a filesystem mount of any repo at any commit very very quickly.
• subnetwork 4y ago
  This feels like something that is prime for abuse. I agree with @bastardoperator, treating git for file storage is going to go nowhere good.
  • reverius42 4y ago
    git-xet follows the patterns established by other git extensions like git-annex and git-lfs, but with some UX improvements. In all of these cases, the large/binary file contents are actually stored outside of git, but available with git commands. The goal here is to make working with and versioning code and data together seamless, not to blindly use the git internals to store data. Does that still seem like a bad idea to you?
• jrockway 4y ago
  There are a couple of other contenders in this space. DVC (https://dvc.org/ https://dvc.org/) seems most similar. If you're interested in something you can self-host... I work on Pachyderm (https://github.com/pachyderm/pachyderm https://github.com/pachyderm/pachyderm), which doesn't have a Git-like interface, but also implements data versioning. Our approach de-duplicates between files (even very small files), and our storage algorithm doesn't create objects proportional to O(n) directory nesting depth as Xet appears to. (Xet is very much like Git in that respect.) The data versioning system enables us to run pipelines based on changes to your data; the pipelines declare what files they read, and that allows us to schedule processing jobs that only reprocess new or changed data, while still giving you a full view of what "would" have happened if all the data had been reprocessed. This, to me, is the key advantage of data versioning; you can save hundreds of thousands of dollars on compute. Being able to undo an oopsie is just icing on the cake. Xet's system for mounting a remote repo as a filesystem is a good idea. We do that too :)
  • ylow 4y ago
    By the way, our mount mechanism has one very interesting novelty. It does not depend on a FUSE driver on Mac :-)
    • jrockway 4y ago
      That's smart! I think users have to install a kext still?
      • ylow 4y ago
        Nope. No kernel driver needed :-) We wrote an localhost NFS server.
        • catiopatio 4y ago
          Based on unfsd or entirely in-house?
          • 4y ago
• timsehn 4y ago
  Founder of DoltHub here. One of my team pointed me at this thread. Congrats on the launch. Great to see more folks tackling the data versioning problem. Dolt hasn't come up here yet, probably because we're focused on OLTP use cases, not MLOps, but we do have some customers using Dolt as the backing store for their training data. https://github.com/dolthub/dolt https://github.com/dolthub/dolt Dolt also scales to the 1TB range and offers you full SQL query capabilities on your data and diffs.
  • ylow 4y ago
    CEO/Cofounder here. Thanks! Agreed, we think data versioning is an important problem and we are at related, but opposite parts of the space. (BTW we really wanted gitfordata.com. Or perhaps we can split the domain? OLTP goes here, Unstructured data goes there :-) Shall we chat? )
• Eleison23 4y ago
• polemic 4y ago
  There seem to be a lot of data version control systems built around ML pipelines or software development needs, but not so much on the sort of data editing that happens outside of software development & analysis. Kart (https://kartproject.org https://kartproject.org) is built on git to provide data version control for geospatial vector & tabular data. Per-row (feature & attribute) version control and the ability to collaborate with a team of people is sorely missing from those workflows. It's focused on geographic use-cases, but you can work with 'plain old tables' too, with MySQL, PostgreSQL and MSSQL working copies (you don't have to pick - you can push and pull between them).
• Wojtkie 4y ago
  Can I upload a full .pbix file to this and use it for versioning? If so, I'd use it in a heartbeat.
  • ylow 4y ago
    CEO/Cofounder here. We are file format agnostic and will happily take everything. Not too familiar with the needs around pbix, but please do try it out and let us know what you think!
• the_arun 4y ago
  Signed up & browsing "Flickr30k" repo (auto generated) repo & it was really slow for me. Like CSV, does it also supports other data formats like json, yml etc.,?
  • ylow 4y ago
    We are file format agnostic and you should be able to put anything in the repo. We have special support for CSV files for visualizations. Sorry for the UI perf... there are a lot of optimizations we need to work on.
• TacticalCoder 4y ago
  What does a Merkle Tree bring here? (honest question) I mean: for content-based addressing of chunks (and hence deduplication of these chunks), a regular tree works too if I'm not mistaken (I may be wrong but I literally wrote a "deduper" splitting files into chunks and using content-based addressing to dedupe the chunks: but I just used a dumb tree). Is the Merkle true used because it brings something else than deduplication, like chunks integrity verification or something like that?
• angrais 4y ago
  How's this differ from using git LFS?
  • ylow 4y ago
    We are significantly faster? :-) Also, block-level dedupe, scalability, perf, visualization, mounting, etc.
    • ironfootnz 4y ago
      write a blog post about these statements. Otherwise is just your opinion on it.
      • ylow 4y ago
        Even better :-) We have a paper. Its on the way.
• culanuchachamim 4y ago
  Maybe a silly question: Why do you need 1Tb for repos? What do you store inside, besides code and some images?
  • lazide 4y ago
    A whole lot of images? I personally would love to be able to store datasets next to code for regression testing, easier deployment, easier dev workstation spin up, etc.
    • culanuchachamim 4y ago
      Still, 1TB? Once you get to that amount of images it would be much easy to manage it with some files storage solution. Or I'm missing something important?
      • lazide 4y ago
        All of them require having some sort of parallel authentication, synchronization, permissions management, change tracking, etc. Which is a huge hassle, and a lot of work I’d rather not do. My current photogrammetry dataset is well over 1TB, and it isn’t a lot for the industry by any stretch of the imagination. In fact, the only thing that considers it ‘a lot’ and is hard to work with is git.
  • layer8 4y ago
    Some docker images? ;)
  • dafelst 4y ago
    Repositories for games are often larger than 1TB, and with things like UE5's Nanite becoming more viable, they're only going to get bigger.
• iFire 4y ago
  https://github.com/facebook/sapling https://github.com/facebook/sapling is doing good work and they are suggesting their git server for large repositories exists.
• COMMENT___ 4y ago
  What about SVN? Besides other features, Subversion supports representation sharing. So adding new textual or binary files with identical data won’t increase the size of your repository. I’m not familiar with ML data sets, but it seems that SVN may work great with them. It already works great for huge and small game dev projects.
• unqueued 4y ago
  I have a 1.96 TB git repo: https://github.com/unqueued/repo.macintoshgarden.org-fileset https://github.com/unqueued/repo.macintoshgarden.org-fileset (It is a mirror of a Macintosh abandoneware site) git annex info . Of course, it uses pointer files for the binary blobs that are not going to change much anyway. And the datalad project has neuro imaging repos that are tens of TB in size. Consider whether you actually need to track differences in all of your files. Honestly git-annex is one of the most powerful tools I have ever used. You can use git for tracking changes in text, but use a different system for tracking binaries. I love how satisfying it is to be able to store the index for hundreds of gigs of files on a floppy disk if I wanted.
• mattewong 4y ago
  Thank you for posting this. Is there any way to access a Xet dataset via a URL (assuming the dataset owner has opted to share in that manner) so that, for example, one could visit a web page that contains some embedded code (JS, WASM etc) which pulls the Xet data into the page for processing?