7 ms·

I think git is a good model for what would otherwise be "laggy async and mismatched" distributed systems. It has a fast sync algorithm, and after you sync, eve

by chubot 6y ago

I think git is a good model for what would otherwise be "laggy async and mismatched" distributed systems. It has a fast sync algorithm, and after you sync, everything works locally on a fast file system. You explicitly know when you're hitting the network, rather than hitting it ALL THE TIME. ----- I would like to use something like git to store the source code to every piece of software I use, and the binaries. That is, most of a whole Linux distro. I have been loosely following some "git for binary data" projects for a number of years. I looked at IPFS like 5 years ago but it seems to have gone off the rails. The dat project seems to have morphed into something else? Are there any new storage projects in that vein? I think the OP is identifying a real problem -- distributed systems are unreliable, and you can get a lot done on a single machine. But we are missing some primitives that would enable that. Every application is littered with buggy and difficut network logic, rather than having a single tool like git (or rsync) which would handle the problem in a focused and fast way. It would be like if Vim/Emacs and GCC/Clang all were "network-enabled"... that doesn't really make sense. Instead they all use the file system, and the file system can be sync'd as an orthogonal issue. Sort of related is a fast distro here I'm looking at: https://michael.stapelberg.ch/posts/2019-08-17-introducing-distri/ https://michael.stapelberg.ch/posts/2019-08-17-introducing-d...

trishume 6y ago
This is also an interesting system in that it's an example of how you can get away with a non-distributed system if your problem is small enough but eventually that falls over. Once you get to large corporation monorepos git operations start to get real slow and use too much hard disk, so you end up with them either creating/using a new VCS or doing some complicated undertaking like https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-planet/ https://devblogs.microsoft.com/bharry/the-largest-git-repo-o...
- chubot 6y ago
  Yeah I agree, it's sort of an open problem, but I guess a bunch of arrows are pointing toward FUSE. Distri uses FUSE and it appears Microsoft's GVFS uses FUSE or whatever Windows technology is the equivalent. (My teammates developed Google's equivalent about 14 years ago, using FUSE, so it's something I've used / seen several times.) FUSE requires some kernel support (a module), which git of course doesn't require. That is a barrier, but perhaps not an insurmountable one. Basically I would like to offload all the "network" work to the OS, so applications are free of that logic.
gumby 6y ago
Actually git “suffers” from two of the problems he listed: cache coherency and, as with all filesystem based approaches, serialization. These don’t matter for git which manages to push all the coherency issues onto the user and which can afford to operate (in computational terms, not human terms) very slowly on small amounts of data. I’m not saying git is slow (it’s gratifyingly fast) but it has a remarkably smaller problem domain than the one described in the article.
- chubot 6y ago
  I don't agree with that framing... When you say "cache coherency" you're implicitly assuming some authoritative state. The point of git is that there isn't a single authoritative state. Not all apps will work with that model, but more apps than you think would. I guess the difference is how fine-grained you want the updates to be. For example something like Figma in the browser (a collaborative photoshop) implements a lot of custom application-specific sync with CRDTs and so forth. Maybe you need the really fine-grained updates, or maybe you just need some Github-like site which allows coarse-grained collaboration. In other words, there can be a network model between "e-mail a .PSD file" and Figma. I think this "in between" would scale to more applications than reinventing sync inside every app. Imagine audio editors, video editors, 3D modellers, etc. Rewriting all those in the style of Figma is prohibtive. I would rather have open file formats and application-agnostic sync, like git. And I think it's a lot cheaper to develop, although current software business models don't really support its development.
  - Someone 6y ago
    “The point of git is that there isn't a single authoritative state.“ I think you’re missing the point of the post you refer to. For git, there _technically_ isn’t a single authoritave state, but for many, if not most, git use cases, there _sociologically_ is. Projects typically have one repo that is _the_repo_: the repo most merges are done to, that releases get built from, whose url you give when you tell people where the project lives, etc. All other clones of that repo, sociologically, are just caching the main repo and changes you make. Then, humans have to decide when to flush what part(s) of the write cache to the main server, and when that results in conflicts, humans have to resolve them. That’s, I think, what “which manages to push all the coherency issues onto the user” means.
dimatura 6y ago
I have been interested in "git for binary data" for a while, mostly for ML/computer vision purposes. I've tried quite a few systems. Of course, there's git-lfs (which keeps "pointer" files and blobs in a cache), which I do use sometimes - but it has a quite few things I don't like. It doesn't give you a lot of control on where the files are stored and how the storage is managed on the remote side. The way it works means there'll be two copies of your data, which is not great for huge datasets. Git-annex (https://git-annex.branchable.com/ https://git-annex.branchable.com/) is pretty great, and ticks almost every checkbox I want. Unlike git-lfs, it uses symlinks instead of pointer files (by default) and gives you a lot of control in managing multiple remote repositories. On the other hand, using it outside of Linux (e.g., MacOS) has always been a bit painful, specially when trying to collaborate with less technical users. I also get the impression that the main developer doesn't have much time for it (understandably - I don't think he makes any money off it, even if there were some early attempts). My current solution is DVC (https://dvc.org/ https://dvc.org/). It's explicitly made with ML in mind, and implements a bunch of stuff beyond binary versioning. It does lack a few of the features of git-annex, but has the ones I do care about most - namely, a fair amount of flexibility on how the remote storage is implemented. And the one thing I like the most is that it can work either like git-lfs (with pointer files), like git-annex (with soft- or hard-links), or -- my favorite -- using reflinks, when running on filesystems that support it (e.g. APFS, btrfs). It also is being actively developed by a team at a company, though so far there doesn't seem to be any paid features or services around it. Pachyderm (https://www.pachyderm.com https://www.pachyderm.com) also seems quite interesting, and pretty ideal for some workflows. Unfortunately it's also more opinionated, in that it requires using docker for the filesystem, as far as I can tell. Edit: a rather different alternative I've resorted to in the past -- which of course lacks a lot of the features of "git for binary data" -- is simply to do regular backups of data to either borg or restic, which are pretty good deduplicating backup systems. Both allow you to mount past snapshots with FUSE, which is a nice way of accessing earlier versions of your data (read-only, of course). These days, this kind of thing can also be done with ZFS or btrfs as well, though.
- chubot 6y ago
  Thanks for the response! I have heard of dvc and git annex, and it's probably time to give them another try :)
benibela 6y ago
I have searched something like this for backups, but have not found anything good. But I also have conflicting wishes. I want it to store the history (so that a backup of directory A cannot be overridden by a backup of an unrelated directory B, and it can apply renames without copying large files again), and then I do not want it to store a history (so that large files that are removed from the main system can be permanently removed from the backup)
hobofan 6y ago
Not sure how well if fits your use-case, but I've been very happy with git-lfs in combination with a NAS I have at home. The NAS is just mounted as a normal network drive and available to use with LFS via lfs-filestore[0] and available on the go via the builtin DynDNS + VPC of the NAS. I've been using it for a repo where I store all my university/academic stuff like lectures (recordings), PDFs of books and papers, Anki decks, etc., and it has now grown to be ~120GB big. Biggest issue was that I'm always running low on disk space with my laptop, and git-lfs doesn't have a good built-in way to only checkout part of the files on your machine, so I built a small tool to make that easier[1]. Since I've been using that it's been a pretty smooth ride. [0]: https://github.com/sinbad/lfs-folderstore https://github.com/sinbad/lfs-folderstore [1]: https://github.com/hobofan/lfs-unload https://github.com/hobofan/lfs-unload
nixpulvis 6y ago
I agree that git is a great model, however it's often hard to explain to new users why merge conflict require time to resolve, and nothing saves you from the added work. Life just isn't completely decentralizable... sorry.
- chubot 6y ago
  git isn't completely centralized either. You can centralize it like Github, and that plays an important role in the ecosystem. Decentralization is a spectrum, not the opposite of centralization. ---- One way to partially address to merge conflict problem is to explode application files into directory hierarchies. For example, Word .doc files and Photoshop .PSD files are basically huge hierchical data structures inside a single file. I believe video formats also have significant hierarchical structure. A lot of them even have immutable portions and mutable portions -- e.g. for storing an entire version history. So if those were exploded into something that the OS (or tools like git/rsync) could understand, like a tree of files, then you would have a lot fewer merge conflicts. That's how people tend structure their git repos too. If you have a frequently edited file that's hard to merge, then that's a smell, and you can fix it. This won't solve every problem but again there needs to be something in between "rewrite Photoshop as Figma" and "email around a bunch of PSD files" (which is a very flaky distributed system on top of e-mail.)
nanomonkey 6y ago
There is git/github on top of Secure Scuttlebutt: https://github.com/noffle/git-ssb-intro https://github.com/noffle/git-ssb-intro
j88439h84 6y ago
> most of a whole Linux distro. It's NixOS.
jazzyjackson 6y ago
I've been bouncing an idea around for a while, on how could I use git as a back-end for a filesharing/chat/collaboration suite -- I think it would work to have a git hook, pre-commit, that replaces all binary files / blacklisted file extensions, with a text file whose contents are the magnet address to download the large binary file over torrent - so the file name, permissions etc don't change, but a small text file is committed instead of the binary. So, as a consumer of the repo, I would clone the repo, and then have a post-checkout hook check the blacklisted file extensions and grab their contents: the magnet link, all that's left is to find peers, download the file and replace the text file with a hardlink to the binary. Maybe this is all too laggy, asynchronous, and mismatched, but I think git+magnet could be a cool combination. I found this repo [1] that generates the magnet link for you, I would just need to find away to use the repo to make all the contributors peers to each other, so we can download the binaries from whoever is nearest / highest bandwidth from us. [1] https://github.com/casey/intermodal https://github.com/casey/intermodal
- robochat 6y ago
  I think that a version of this idea underpins the operation of the matrix messaging protocol. Rather than just sending the latest messages, the conversation is synced between clients to ensure that everyone sees the same history. [1] https://matrix.org https://matrix.org
- nybble41 6y ago
  What you are describing sounds a lot like git-annex[1] with the IPFS remote[2]. There is also a bittorrent remote[3] but it doesn't handle uploading content; you have to create the .torrent file or magnet link yourself and add it to the repo, at which point git-annex will download the content automatically on checkout. [1] https://git-annex.branchable.com/ https://git-annex.branchable.com/ [2] https://git-annex.branchable.com/special_remotes/ipfs/ https://git-annex.branchable.com/special_remotes/ipfs/ [3] https://git-annex.branchable.com/special_remotes/bittorrent/ https://git-annex.branchable.com/special_remotes/bittorrent/
tsimionescu 6y ago
> It would be like if Vim/Emacs and GCC/Clang all were "network-enabled"... that doesn't really make sense. Instead they all use the file system, and the file system can be sync'd as an orthogonal issue. Well, you may already be aware, but Emacs is actually 'network-enabled' in this way through TRAMP, and to a lesser extent, the emacs client/server protocol. There are also many issues other than file syncing that networking has to solve. And also, even for regular files, there are many ways to interact with them, and many protocols that existing systems can already speak.
microtherion 6y ago
git has some robustness problem over slow, fragile connections, though. It will not corrupt data, but if your connection cannot stay up long enough to clone an entire repo, you need to resort to ugly workarounds.

by chubot 6y ago

trishume 6y ago

This is also an interesting system in that it's an example of how you can get away with a non-distributed system if your problem is small enough but eventually that falls over. Once you get to large corporation monorepos git operations start to get real slow and use too much hard disk, so you end up with them either creating/using a new VCS or doing some complicated undertaking like https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-planet/ https://devblogs.microsoft.com/bharry/the-largest-git-repo-o...

chubot 6y ago
Yeah I agree, it's sort of an open problem, but I guess a bunch of arrows are pointing toward FUSE. Distri uses FUSE and it appears Microsoft's GVFS uses FUSE or whatever Windows technology is the equivalent. (My teammates developed Google's equivalent about 14 years ago, using FUSE, so it's something I've used / seen several times.) FUSE requires some kernel support (a module), which git of course doesn't require. That is a barrier, but perhaps not an insurmountable one. Basically I would like to offload all the "network" work to the OS, so applications are free of that logic.

gumby 6y ago

Actually git “suffers” from two of the problems he listed: cache coherency and, as with all filesystem based approaches, serialization. These don’t matter for git which manages to push all the coherency issues onto the user and which can afford to operate (in computational terms, not human terms) very slowly on small amounts of data. I’m not saying git is slow (it’s gratifyingly fast) but it has a remarkably smaller problem domain than the one described in the article.

chubot 6y ago
I don't agree with that framing... When you say "cache coherency" you're implicitly assuming some authoritative state. The point of git is that there isn't a single authoritative state. Not all apps will work with that model, but more apps than you think would. I guess the difference is how fine-grained you want the updates to be. For example something like Figma in the browser (a collaborative photoshop) implements a lot of custom application-specific sync with CRDTs and so forth. Maybe you need the really fine-grained updates, or maybe you just need some Github-like site which allows coarse-grained collaboration. In other words, there can be a network model between "e-mail a .PSD file" and Figma. I think this "in between" would scale to more applications than reinventing sync inside every app. Imagine audio editors, video editors, 3D modellers, etc. Rewriting all those in the style of Figma is prohibtive. I would rather have open file formats and application-agnostic sync, like git. And I think it's a lot cheaper to develop, although current software business models don't really support its development.
- Someone 6y ago
  “The point of git is that there isn't a single authoritative state.“ I think you’re missing the point of the post you refer to. For git, there _technically_ isn’t a single authoritave state, but for many, if not most, git use cases, there _sociologically_ is. Projects typically have one repo that is _the_repo_: the repo most merges are done to, that releases get built from, whose url you give when you tell people where the project lives, etc. All other clones of that repo, sociologically, are just caching the main repo and changes you make. Then, humans have to decide when to flush what part(s) of the write cache to the main server, and when that results in conflicts, humans have to resolve them. That’s, I think, what “which manages to push all the coherency issues onto the user” means.

dimatura 6y ago

I have been interested in "git for binary data" for a while, mostly for ML/computer vision purposes. I've tried quite a few systems. Of course, there's git-lfs (which keeps "pointer" files and blobs in a cache), which I do use sometimes - but it has a quite few things I don't like. It doesn't give you a lot of control on where the files are stored and how the storage is managed on the remote side. The way it works means there'll be two copies of your data, which is not great for huge datasets. Git-annex (https://git-annex.branchable.com/ https://git-annex.branchable.com/) is pretty great, and ticks almost every checkbox I want. Unlike git-lfs, it uses symlinks instead of pointer files (by default) and gives you a lot of control in managing multiple remote repositories. On the other hand, using it outside of Linux (e.g., MacOS) has always been a bit painful, specially when trying to collaborate with less technical users. I also get the impression that the main developer doesn't have much time for it (understandably - I don't think he makes any money off it, even if there were some early attempts). My current solution is DVC (https://dvc.org/ https://dvc.org/). It's explicitly made with ML in mind, and implements a bunch of stuff beyond binary versioning. It does lack a few of the features of git-annex, but has the ones I do care about most - namely, a fair amount of flexibility on how the remote storage is implemented. And the one thing I like the most is that it can work either like git-lfs (with pointer files), like git-annex (with soft- or hard-links), or -- my favorite -- using reflinks, when running on filesystems that support it (e.g. APFS, btrfs). It also is being actively developed by a team at a company, though so far there doesn't seem to be any paid features or services around it. Pachyderm (https://www.pachyderm.com https://www.pachyderm.com) also seems quite interesting, and pretty ideal for some workflows. Unfortunately it's also more opinionated, in that it requires using docker for the filesystem, as far as I can tell. Edit: a rather different alternative I've resorted to in the past -- which of course lacks a lot of the features of "git for binary data" -- is simply to do regular backups of data to either borg or restic, which are pretty good deduplicating backup systems. Both allow you to mount past snapshots with FUSE, which is a nice way of accessing earlier versions of your data (read-only, of course). These days, this kind of thing can also be done with ZFS or btrfs as well, though.

chubot 6y ago
Thanks for the response! I have heard of dvc and git annex, and it's probably time to give them another try :)

benibela 6y ago

I have searched something like this for backups, but have not found anything good. But I also have conflicting wishes. I want it to store the history (so that a backup of directory A cannot be overridden by a backup of an unrelated directory B, and it can apply renames without copying large files again), and then I do not want it to store a history (so that large files that are removed from the main system can be permanently removed from the backup)

hobofan 6y ago

Not sure how well if fits your use-case, but I've been very happy with git-lfs in combination with a NAS I have at home. The NAS is just mounted as a normal network drive and available to use with LFS via lfs-filestore[0] and available on the go via the builtin DynDNS + VPC of the NAS. I've been using it for a repo where I store all my university/academic stuff like lectures (recordings), PDFs of books and papers, Anki decks, etc., and it has now grown to be ~120GB big. Biggest issue was that I'm always running low on disk space with my laptop, and git-lfs doesn't have a good built-in way to only checkout part of the files on your machine, so I built a small tool to make that easier[1]. Since I've been using that it's been a pretty smooth ride. [0]: https://github.com/sinbad/lfs-folderstore https://github.com/sinbad/lfs-folderstore [1]: https://github.com/hobofan/lfs-unload https://github.com/hobofan/lfs-unload

nixpulvis 6y ago

I agree that git is a great model, however it's often hard to explain to new users why merge conflict require time to resolve, and nothing saves you from the added work. Life just isn't completely decentralizable... sorry.

chubot 6y ago
git isn't completely centralized either. You can centralize it like Github, and that plays an important role in the ecosystem. Decentralization is a spectrum, not the opposite of centralization. ---- One way to partially address to merge conflict problem is to explode application files into directory hierarchies. For example, Word .doc files and Photoshop .PSD files are basically huge hierchical data structures inside a single file. I believe video formats also have significant hierarchical structure. A lot of them even have immutable portions and mutable portions -- e.g. for storing an entire version history. So if those were exploded into something that the OS (or tools like git/rsync) could understand, like a tree of files, then you would have a lot fewer merge conflicts. That's how people tend structure their git repos too. If you have a frequently edited file that's hard to merge, then that's a smell, and you can fix it. This won't solve every problem but again there needs to be something in between "rewrite Photoshop as Figma" and "email around a bunch of PSD files" (which is a very flaky distributed system on top of e-mail.)

nanomonkey 6y ago

There is git/github on top of Secure Scuttlebutt: https://github.com/noffle/git-ssb-intro https://github.com/noffle/git-ssb-intro

j88439h84 6y ago

> most of a whole Linux distro. It's NixOS.

jazzyjackson 6y ago

I've been bouncing an idea around for a while, on how could I use git as a back-end for a filesharing/chat/collaboration suite -- I think it would work to have a git hook, pre-commit, that replaces all binary files / blacklisted file extensions, with a text file whose contents are the magnet address to download the large binary file over torrent - so the file name, permissions etc don't change, but a small text file is committed instead of the binary. So, as a consumer of the repo, I would clone the repo, and then have a post-checkout hook check the blacklisted file extensions and grab their contents: the magnet link, all that's left is to find peers, download the file and replace the text file with a hardlink to the binary. Maybe this is all too laggy, asynchronous, and mismatched, but I think git+magnet could be a cool combination. I found this repo [1] that generates the magnet link for you, I would just need to find away to use the repo to make all the contributors peers to each other, so we can download the binaries from whoever is nearest / highest bandwidth from us. [1] https://github.com/casey/intermodal https://github.com/casey/intermodal

robochat 6y ago
I think that a version of this idea underpins the operation of the matrix messaging protocol. Rather than just sending the latest messages, the conversation is synced between clients to ensure that everyone sees the same history. [1] https://matrix.org https://matrix.org
nybble41 6y ago
What you are describing sounds a lot like git-annex[1] with the IPFS remote[2]. There is also a bittorrent remote[3] but it doesn't handle uploading content; you have to create the .torrent file or magnet link yourself and add it to the repo, at which point git-annex will download the content automatically on checkout. [1] https://git-annex.branchable.com/ https://git-annex.branchable.com/ [2] https://git-annex.branchable.com/special_remotes/ipfs/ https://git-annex.branchable.com/special_remotes/ipfs/ [3] https://git-annex.branchable.com/special_remotes/bittorrent/ https://git-annex.branchable.com/special_remotes/bittorrent/

tsimionescu 6y ago

> It would be like if Vim/Emacs and GCC/Clang all were "network-enabled"... that doesn't really make sense. Instead they all use the file system, and the file system can be sync'd as an orthogonal issue. Well, you may already be aware, but Emacs is actually 'network-enabled' in this way through TRAMP, and to a lesser extent, the emacs client/server protocol. There are also many issues other than file syncing that networking has to solve. And also, even for regular files, there are many ways to interact with them, and many protocols that existing systems can already speak.

microtherion 6y ago

git has some robustness problem over slow, fragile connections, though. It will not corrupt data, but if your connection cannot stay up long enough to clone an entire repo, you need to resort to ugly workarounds.