5 ms·

I see a lot of reasons to version code. I see far less reasons to version data, in fact, I find reasons against versioning data and storing them in diffs.

ch71r22 4y ago
What are the reasons against?
- ltbarcly3 4y ago
  The lack of reasons for doing it IS the reason against. GIT isn't a magic 'good way' to store arbitrary data, it's a good way to collaborate on projects implemented using most programming languages which store code as plain text broken into short lines, where edits to non-sequential lines can generally be applied concurrently without careful human verification. That is an extremely specific use case, and anything outside of that very specific use case leaves git terrible, inefficient, and gives almost no benefit despite huge problems. People in ML ops use git because they aren't very sophisticated with programming professionally and they have git available to them and they haven't run into the consequences of using it to store large binary blobs, namely that it becomes impossible to live with eventually and wastes a huge amount of time and space. ML didn't invent the need for large artifacts that can't be versioned in source control but must be versioned with it, but they don't know that because they are new to professional programming and aren't familiar with how it's done.
  - ylow 4y ago
    Indeed, there is a lot of pain if you actually try to store large binary data in git. But we managed to make that work! So a question worth asking is how might things change IF you can store large binary data in git??
    - ltbarcly3 4y ago
      I think this is a foot-gun, it's a bad idea even if it works great, and I doubt it works very well. You should manage your build artifacts explicitly, not just jam them in git along with the code that generates them because you are already using it and you haven't thought it through.
      - Game_Ender 4y ago
        Xet's initial focus appears to be on data files used to drive machine learning pipelines, not on any resulting binaries.
      - wpietri 4y ago
        I don't think you've made your case here. The practices you describe are partly an artifact of computation, bandwidth, and storage costs. But not the current ones, the ones when git was invented more than 15 years ago. In the short term, we have to conform to the computer's needs. But in the long term, it has to be the other way around.
        ltbarcly3 4y ago
        You're right! It makes way more sense, in the long run, to abuse a tool like git in a way that it isn't designed for and which it can't actually support, then instead of actually using git use a proprietary service that may or may not be around in a week. Here I was thinking short term.
        wpietri 4y ago
        You seem nice.
        ltbarcly3 4y ago
        Than you, I've worked very hard to become so. You seem nice, too!
    - sk0g 4y ago
      That is exactly what git-lfs is, a way to "version control" binary files, by storing revisions - possibly separately, while the actual repo contains text files + "pointer" files that references a binary file. It's not perfect, and still feels like a bit of a hack compared to something like p4 for the context I uses LFS in (game dev), but it works, and doesn't require expensive custom licenses when teams grow beyond an arbitrary number like 3 or 5.
      - rajatarya 4y ago
        XetHub Co-founder here. Yes, we use the same Git extension mechanism as Git LFS (clean/smudge filters) and we store pointer files in the git repository. Unlike Git LFS we do block-level deduplication (Git LFS does file-level deduplication) and this can result in a significant savings in storage and bandwidth. As an example, a Unity game repo reduced in size by 41% using our block-level deduplication vs Git LFS. Raw repo was 48.9GB, Git LFS was 48.2GB, and with XetHub was 28.7GB. Why do you think using a Git-based solution is a hack compared to p4? What part of the p4 workflow feels more natural to you?
        sk0g 4y ago
        The centralised model of Perforce is more of a natural fit for one thing, since by default it allows you to clone subsets, and just the latest version of files. File locking is much more integrated into the p4 workflow as well, in git you can still modify files locally, then commit them. The check happens on push, and sometimes git fails to send the lock notification upstream. Oh and it breaks down entirely if you use branches. Some of these have workarounds and hacks for more experienced users. I'm not about to run around teaching people the intricacies of arcane git incantations, while p4 functions, by default, how you'd want to. The programming side is better on git though, yeah.
        reverius42 4y ago
        (XetHub engineer here) We're working on perforce-style locking on XetHub, and I believe git already supports things like only cloning the latest version of files. Cloning the full repo without "smudging" (pulling in binary file contents) is already possible, and cloning while smudging a subset is on our roadmap. We're definitely on a path to making git UX for dealing with large binary files as easy as perforce, and there are lots of advantages to keeping a git-based workflow for teams that already work with git.
  - mardifoufs 4y ago
    I literally don't know anyone or any team in ML using git as a data versioning tool. It doesn't even make sense to me, and most mlops people I have talked to would agree. Is that really the point of this tool? To be a general purpose data store for mlops? I thought it is for very specialized ML use cases. Because even 1TB isn't much for ML data versioning Mlops people are very aware of tools that are more suited for the job... even too aware in fact. The entire field is full of tools, databases, etc to the point where it's hard make sense of it. So your comment is a bit weird to me
    - ltbarcly3 4y ago
      I think you'll find varying levels of maturity in ML ops. Anyway I think we basically agree, if you use something like this you aren't that mature, and if you are mature you would avoid this thing.
    - mickkygoel 4y ago
      Building mlops solutions for a big tech. Agree, most mature ml teams are not using git for ml data versioning, but in my experience and user research it’s not due to lack of intent. Teams have been forced to move to other ml data tools in absence of scalable git solution, most of which come with a lot of cognitive overhead for ml engineers who don’t want to spend time adopting several custom tools in their ml pipelines.
guardian5x 4y ago
As always it depends on the application. It can definitely be useful in some applications.
substation13 4y ago
Versioning data is great, but storing as diffs is inefficient when 99% of the file changes each version.
- reverius42 4y ago
  We don't store as diffs, we store as snapshots -- and it's efficient thanks to the way we do dedupe. See https://xethub.com/assets/docs/how-xet-deduplication-works/ https://xethub.com/assets/docs/how-xet-deduplication-works/
ylow 4y ago
Cofounder/CEO here! I think it less about "versioning", but the ability to modify with confidence knowing that you can go back in time anytime. (Minor clarification: we are not quite storing diffs; holding snapshots just like Git + a bunch of data dedupe)
- krageon 4y ago
  > the ability to modify with confidence knowing that you can go back in time anytime This is versioning
  - rafael09ed 4y ago
    Versioning is a technique. Backups, copy+paste+rename also does it
treeman79 4y ago
Anything that might be audited. Being able to look at things how they were how they changed to find out how they got to where they currently are, and who did what; is amazing for many application. Finance, healthcare, elections, Etc. Well unless fraud is the goal.
- bfm 4y ago
  Shameless plug for https://snapdir.org https://snapdir.org which focuses on this particular use case using regular git and auditable plain text manifests
oftenwrong 4y ago
One use-case would be for including dependencies in a repo. For example, it is common for companies to operate their own artifact caches/mirrors to protect their access to artifacts from npm, pypi, dockerhub, maven central, pkg.go.dev, etc. With the ability to efficiently work with a big repo, it would be possible to store the artifacts in git, saving the trouble of having to operate artifact mirrors. Additionally, it guarantees that the artifacts for a given, known-buildable revision are available offline.
zachmu 4y ago
You're suffering from a failure of imagination, maybe because you've never been able to version data usefully before. There are already lots of interesting applications, and it's still quite new. https://www.dolthub.com/blog/2022-07-11-dolt-case-studies/ https://www.dolthub.com/blog/2022-07-11-dolt-case-studies/
WorldMaker 4y ago
Something that I've experienced from many years in enterprise software: 90% of enterprise software is about versioning data in some way. SharePoint is half as complicated as it is because it has be a massive document and data version manager. (Same with Confluence and other competitors.) "Everyone" needs deep audit logs for some likely overlap of SOX compliance, PCI compliance, HIPAA compliance, and/or other industry specific standards and practices. Most business analysts want accurate "point in time" reporting tools to revisit data as it looked at almost any point in the past, and if you don't build it for them they often build it as ad hoc file cabinets full of Excel export dumps for themselves. The wheels of data versioning just get reinvented over and over and over again, with all sorts of slightly different tools. Most of the job of "boring CRUD app development" is data version management and some of the "joy" is how every database you ever encounter is often its own little snowflake with respect to how it versions its data. There have been times I've pined for being able to just store it all in git and reduce things to a single paradigm. That said, I'd never actually want to teach business analysts or accountants how to use git (and would probably spend nearly as much time building custom CRUD apps against git as against any other sort of database). There are times though where I have thought for backend work "if I could just checkout the database at the right git tag instead needing to write this five table join SQL statement with these eighteen differently named timestamp fields that need to be sorted in four different ways…". Reasons to version data are plenty and most of the data versioning in the world is ad hoc and/or operationally incompatible/inconsistent across systems. (Ever had to ETL SharePoint lists and its CVC-based versioning with a timestamp based data table? Such "fun".) I don't think git is necessarily the savior here, though there remains some appeal in "I can use the same systems I use for code" two birds with one stone. Relatedly, content-addressed storage and/or merkle trees are a growing tool for Enterprise and do look a lot like a git repository and sometimes you also have the feeling like if you are already using git why build your own merkle tree store when git gives you a swiss army knife tool kit on top of that merkle tree store.

ch71r22 4y ago

What are the reasons against?

ltbarcly3 4y ago
The lack of reasons for doing it IS the reason against. GIT isn't a magic 'good way' to store arbitrary data, it's a good way to collaborate on projects implemented using most programming languages which store code as plain text broken into short lines, where edits to non-sequential lines can generally be applied concurrently without careful human verification. That is an extremely specific use case, and anything outside of that very specific use case leaves git terrible, inefficient, and gives almost no benefit despite huge problems. People in ML ops use git because they aren't very sophisticated with programming professionally and they have git available to them and they haven't run into the consequences of using it to store large binary blobs, namely that it becomes impossible to live with eventually and wastes a huge amount of time and space. ML didn't invent the need for large artifacts that can't be versioned in source control but must be versioned with it, but they don't know that because they are new to professional programming and aren't familiar with how it's done.
- ylow 4y ago
  Indeed, there is a lot of pain if you actually try to store large binary data in git. But we managed to make that work! So a question worth asking is how might things change IF you can store large binary data in git??
  - ltbarcly3 4y ago
    I think this is a foot-gun, it's a bad idea even if it works great, and I doubt it works very well. You should manage your build artifacts explicitly, not just jam them in git along with the code that generates them because you are already using it and you haven't thought it through.
    - Game_Ender 4y ago
      Xet's initial focus appears to be on data files used to drive machine learning pipelines, not on any resulting binaries.
    - wpietri 4y ago
      I don't think you've made your case here. The practices you describe are partly an artifact of computation, bandwidth, and storage costs. But not the current ones, the ones when git was invented more than 15 years ago. In the short term, we have to conform to the computer's needs. But in the long term, it has to be the other way around.
      - ltbarcly3 4y ago
        You're right! It makes way more sense, in the long run, to abuse a tool like git in a way that it isn't designed for and which it can't actually support, then instead of actually using git use a proprietary service that may or may not be around in a week. Here I was thinking short term.
        wpietri 4y ago
        You seem nice.
        ltbarcly3 4y ago
        Than you, I've worked very hard to become so. You seem nice, too!
  - sk0g 4y ago
    That is exactly what git-lfs is, a way to "version control" binary files, by storing revisions - possibly separately, while the actual repo contains text files + "pointer" files that references a binary file. It's not perfect, and still feels like a bit of a hack compared to something like p4 for the context I uses LFS in (game dev), but it works, and doesn't require expensive custom licenses when teams grow beyond an arbitrary number like 3 or 5.
    - rajatarya 4y ago
      XetHub Co-founder here. Yes, we use the same Git extension mechanism as Git LFS (clean/smudge filters) and we store pointer files in the git repository. Unlike Git LFS we do block-level deduplication (Git LFS does file-level deduplication) and this can result in a significant savings in storage and bandwidth. As an example, a Unity game repo reduced in size by 41% using our block-level deduplication vs Git LFS. Raw repo was 48.9GB, Git LFS was 48.2GB, and with XetHub was 28.7GB. Why do you think using a Git-based solution is a hack compared to p4? What part of the p4 workflow feels more natural to you?
      - sk0g 4y ago
        The centralised model of Perforce is more of a natural fit for one thing, since by default it allows you to clone subsets, and just the latest version of files. File locking is much more integrated into the p4 workflow as well, in git you can still modify files locally, then commit them. The check happens on push, and sometimes git fails to send the lock notification upstream. Oh and it breaks down entirely if you use branches. Some of these have workarounds and hacks for more experienced users. I'm not about to run around teaching people the intricacies of arcane git incantations, while p4 functions, by default, how you'd want to. The programming side is better on git though, yeah.
        reverius42 4y ago
        (XetHub engineer here) We're working on perforce-style locking on XetHub, and I believe git already supports things like only cloning the latest version of files. Cloning the full repo without "smudging" (pulling in binary file contents) is already possible, and cloning while smudging a subset is on our roadmap. We're definitely on a path to making git UX for dealing with large binary files as easy as perforce, and there are lots of advantages to keeping a git-based workflow for teams that already work with git.
- mardifoufs 4y ago
  I literally don't know anyone or any team in ML using git as a data versioning tool. It doesn't even make sense to me, and most mlops people I have talked to would agree. Is that really the point of this tool? To be a general purpose data store for mlops? I thought it is for very specialized ML use cases. Because even 1TB isn't much for ML data versioning Mlops people are very aware of tools that are more suited for the job... even too aware in fact. The entire field is full of tools, databases, etc to the point where it's hard make sense of it. So your comment is a bit weird to me
  - ltbarcly3 4y ago
    I think you'll find varying levels of maturity in ML ops. Anyway I think we basically agree, if you use something like this you aren't that mature, and if you are mature you would avoid this thing.
  - mickkygoel 4y ago
    Building mlops solutions for a big tech. Agree, most mature ml teams are not using git for ml data versioning, but in my experience and user research it’s not due to lack of intent. Teams have been forced to move to other ml data tools in absence of scalable git solution, most of which come with a lot of cognitive overhead for ml engineers who don’t want to spend time adopting several custom tools in their ml pipelines.

guardian5x 4y ago

As always it depends on the application. It can definitely be useful in some applications.

substation13 4y ago

Versioning data is great, but storing as diffs is inefficient when 99% of the file changes each version.

reverius42 4y ago
We don't store as diffs, we store as snapshots -- and it's efficient thanks to the way we do dedupe. See https://xethub.com/assets/docs/how-xet-deduplication-works/ https://xethub.com/assets/docs/how-xet-deduplication-works/

ylow 4y ago

Cofounder/CEO here! I think it less about "versioning", but the ability to modify with confidence knowing that you can go back in time anytime. (Minor clarification: we are not quite storing diffs; holding snapshots just like Git + a bunch of data dedupe)

krageon 4y ago
> the ability to modify with confidence knowing that you can go back in time anytime This is versioning
- rafael09ed 4y ago
  Versioning is a technique. Backups, copy+paste+rename also does it

treeman79 4y ago

Anything that might be audited. Being able to look at things how they were how they changed to find out how they got to where they currently are, and who did what; is amazing for many application. Finance, healthcare, elections, Etc. Well unless fraud is the goal.

bfm 4y ago
Shameless plug for https://snapdir.org https://snapdir.org which focuses on this particular use case using regular git and auditable plain text manifests

oftenwrong 4y ago

One use-case would be for including dependencies in a repo. For example, it is common for companies to operate their own artifact caches/mirrors to protect their access to artifacts from npm, pypi, dockerhub, maven central, pkg.go.dev, etc. With the ability to efficiently work with a big repo, it would be possible to store the artifacts in git, saving the trouble of having to operate artifact mirrors. Additionally, it guarantees that the artifacts for a given, known-buildable revision are available offline.

zachmu 4y ago

You're suffering from a failure of imagination, maybe because you've never been able to version data usefully before. There are already lots of interesting applications, and it's still quite new. https://www.dolthub.com/blog/2022-07-11-dolt-case-studies/ https://www.dolthub.com/blog/2022-07-11-dolt-case-studies/

WorldMaker 4y ago

Something that I've experienced from many years in enterprise software: 90% of enterprise software is about versioning data in some way. SharePoint is half as complicated as it is because it has be a massive document and data version manager. (Same with Confluence and other competitors.) "Everyone" needs deep audit logs for some likely overlap of SOX compliance, PCI compliance, HIPAA compliance, and/or other industry specific standards and practices. Most business analysts want accurate "point in time" reporting tools to revisit data as it looked at almost any point in the past, and if you don't build it for them they often build it as ad hoc file cabinets full of Excel export dumps for themselves. The wheels of data versioning just get reinvented over and over and over again, with all sorts of slightly different tools. Most of the job of "boring CRUD app development" is data version management and some of the "joy" is how every database you ever encounter is often its own little snowflake with respect to how it versions its data. There have been times I've pined for being able to just store it all in git and reduce things to a single paradigm. That said, I'd never actually want to teach business analysts or accountants how to use git (and would probably spend nearly as much time building custom CRUD apps against git as against any other sort of database). There are times though where I have thought for backend work "if I could just checkout the database at the right git tag instead needing to write this five table join SQL statement with these eighteen differently named timestamp fields that need to be sorted in four different ways…". Reasons to version data are plenty and most of the data versioning in the world is ad hoc and/or operationally incompatible/inconsistent across systems. (Ever had to ETL SharePoint lists and its CVC-based versioning with a timestamp based data table? Such "fun".) I don't think git is necessarily the savior here, though there remains some appeal in "I can use the same systems I use for code" two birds with one stone. Relatedly, content-addressed storage and/or merkle trees are a growing tool for Enterprise and do look a lot like a git repository and sometimes you also have the feeling like if you are already using git why build your own merkle tree store when git gives you a swiss army knife tool kit on top of that merkle tree store.