4 ms·
I want a distributed (not cloud) file system that handles many files and WAN latency (i.e. not HDFS). It might be a cross between Git and BitTorrent. It feels
by chubot 9y ago
I want a distributed (not cloud) file system that handles many files and WAN latency (i.e. not HDFS). It might be a cross between Git and BitTorrent.
It feels like deb repos, PyPI, NPM, CPAN, CRAN, etc. should be put in there, with the addition of binaries for popular architectures. And probably Docker-like images, although I think if they are not opaque blobs, it would be better for rsync-like differential compression (which Git implicitly provides).
There will be some small files and some big files. I want it to be like Git so I can clone locally, not just go to the cloud. The way that Git is trivial to set up and clone through SSH is nice too.
It probably has to have a notion of "user", like a local file system. (This makes the problem a lot harder; git doesn't really have permissions.)
BTW Julia's package manager just used git, but I watched a talk that said this ended up being a really bad idea, especially on Windows.
As far as I understand IPFS has some of these properties. Has anybody used it? Could I use it for the package repository use case?
I don't know much about it, but Project Atomic sounds similar too: https://www.projectatomic.io/ https://www.projectatomic.io/
Any other projects that seem like a close fit?
BTW I think this would also be useful in the data center, as some companies like Twitter apparently use BitTorrent in data centers to start large jobs quickly (i.e replicate the same 500 MB binary to 1000 machines).
- dsnuh 9y agoIPFS does sound like it might be close to what you are looking for, just based off my little experience playing with it. I want to try to leverage a private deployment for dataset sharing for analytics, etc.
- antoncohen 9y agoBasically you want S3, which has BitTorrent BTW. S3 has multiple widely used command line tools that support syncing (cloning, pushing). There are multiple heavily used S3-like systems, open and closed source, some are S3 API compatible. Examples include Ceph and OpenStack Swift. The reason people use cloud services instead of running these themselves is because the cloud storage fee is usually less than paying someone to run a distributed system.
- chubot 9y agoRight, but I can run my own git "server" with no problem. That's why I use that analogy. Plenty of people run their own BitTorrent trackers too. In my mind, "cloud" open source means: you need a team of experts to run it. git and BitTorrent are different -- they are designed for you to run it yourself. Also, the model is to "sync" and then "read/write", as with git (and BitTorrent). Not just read/write remotely. So maybe I should call it a "replicated file system" rather than a distributed one. I guess the main difference with git is that it should handle large binaries / many files, you shouldn't have to clone the entire repo, and maybe users/permissions.
- stephenr 9y agoIts not a solution to all problems but Minio (https://minio.io https://minio.io) will provide s3 compatible data storage on your own server(s).
- zukzuk 9y agoSyncthing is pretty nice. It's not a filesystem, and it doesn't deal with conflicts particularly well, but it's as close to a cross-platform, distributed solution that "just works" as I've been able to find.
- _eht 9y agoI feel like a major repo issue has yet to show it's head. We got a sample of what it might feel like with the NPM namespace fiasco that happened recently, albeit a different problem than scale and accessibility.
- burkemw3 9y agoA few random thoughts: IPFS sounds a lot like what you want. Tahoe LAFS plays in this space a bit The newer distributed file systems I've seen don't like permissions. They like cryptographically-backed capabilities. You have the permission to read the file because you have the ability to decrypt, through the key. (Of course, key management is easy </sarc>). Some user-facing distributed filesystem talks drift into FUSE (or similar) territory. A TahoeLAFS dev talks about how users probably don't actually know what they want: https://plus.google.com/108313527900507320366/posts/ZrgdgLhV3NG https://plus.google.com/108313527900507320366/posts/ZrgdgLhV... (QUIBBLES: REAL FILESYSTEM VS. STORAGE APP section). This is probably less relevant for a package manager. The first time I read about BitTorrent based deployment was from Facebook.
- chubot 9y agoThanks for the link! I've heard about Tahoe LAFS, but like a lot of these projects (Upspin, IPFS), I don't actually know anyone who uses them!
- jimmy1 9y agoUpspin? https://upspin.io/ https://upspin.io/
- chubot 9y agoYes I watched a talk on it... it seems interesting for sure. It relies on FUSE I believe, whereas I might want an explicit "sync". Still, I'm definitely interested in whether I could use it to upload say 1 TB of source code, and maybe 5 TB of binaries, and have people sync efficiently (and partially).
- mhacct 9y agoI’m sure it misses some part of what you’re looking for but there’s an old project on github called DrFTPD that does at least part of what you’re looking for, in a relatively archaic manner. It’s designed for a distributed FTPD across WAN links. It also has mirroring / striping, user accounts, etc. I’m sure you could find some tool to help it look local if needed.
- anon1253 9y agohttps://www.resilio.com/ https://www.resilio.com/ (formerly bitorrent-sync)?