10 ms·
Show HN: Zero – Local file system transparently swapping to the cloud
- hilyen 8y agoNot seeing, but does it support encrypting files/folders on cloud storage?
- yjftsjthsd-h 8y agoWorst case, could layer with encfs or equivalent. Be very careful to understand the exact threat model that covers (for starters, it leaves you painfully exposed to metadata issues), but it would work easily enough.
- konschubert 8y agoMy plan is to use it with fusecrypt for now and eventually include encryption directly.
- deleted 8y ago[deleted]
- starkruzr 8y agoI can't see how this is used. Can we get a "here's how to set up the file system" guide somewhere? I see the bit about the config file. Then what? I'd primarily like to use this to back up a couple of Proxmox hosts.
- konschubert 8y agoYes, I'll add a guide and more description but please don't use it yet, it's work in progress.
- corybrown 8y agoBeen using rclone mount for something similar, curious how this compares
- konschubert 8y agoI think with rclone you need as much local space as you take up in the cloud?
- dpacmittal 8y agoRclone has a cache backend in the latest version which works exactly like OP.
- konschubert 8y agoCool! Do you have a link?
- mirceal 8y agoare there any plans to support S3?
- h1d 8y agoThey should support providers such as Wasabi where they have unlimited egress to feel safe with a fixed pricing (per GB).
- konschubert 8y agoHi, the Back-Ends are in principle pluggable so I'm very happy to incorporate a PR for an s3 Back end. I may also do it myself eventually. I just find s3 very expensive for long term storage of personal data.
- Tloewald 8y agoInteresting idea. But is the cloud version a complete image (perhaps out of sync)? If so then it’s a performance disaster, if not it’s very fragile. It seems to me what we really want is a cloud file system with local cache (like Dropbox or iCloud conceptually) so that if our local device is vaporized we have a pretty much up to date logical store alive and well (and we can work on any number of machines). The word “swapping” seems to me to be based on the virtual memory model which means that if anything goes wrong you have two disconnected piles of crap. At a file level you could theoretically have a giant file that is never wholly local, but how useful is this as a feature in real terms?
- nine_k 8y agoI think Borg or Tarsnap use the right approach here: a map of blocks, updating a file updates only the changed block(s). It balances the efficiency of updates and the completeness of the copy. Sort of like FAT filesystem, only with block-level deduplication built in. Of course you don't get a nice mirror of your files right in the cloud, unless you run a separate server that reconstructs it and makes available as traditional buckets.
- deleted 8y ago[deleted]
- AmrMostafa 8y agoI am wondering if there is a backup solution that works that way but without requiring a manual time consuming invocation. Using something like inotify to record changed files and a worker in the background to immediately sync. Like dropbox.
- hemancuso 8y agohttp://www.arqbackup.com http://www.arqbackup.com
- h1d 8y agorestic and duplicacy are the newer implementations of block level dedup encrypted backup. From what I tested, restic has friendlier command line options but duplicacy is technically superior at this point (restore works way faster)
- ArtWomb 8y agoGcloud also has a FUSE adapter for cloud storage buckets https://cloud.google.com/storage/docs/gcs-fuse https://cloud.google.com/storage/docs/gcs-fuse
- hemancuso 8y agoVery cool project! Close to my heart. I write ExpanDrive http://www.expandrive.com http://www.expandrive.com Which operates with the same idea, up to a 10Gb local cache and will stream out larger files. Supports a huge number of storage back ends (Dropbox, Google Drive/GCS, Amazon S3/Drive, OneDrive, SFTP, etc). Also can pin files and trees into the cache.
- deleted 8y ago[deleted]
- drtyolmck 8y agoI use expandrive everyday. Thanks man.
- kyrra 8y agoIs that similar to the Drive FS client then (except it's obviously multi-cloud and backend)? https://support.google.com/a/answer/7491144?hl=en https://support.google.com/a/answer/7491144?hl=en
- deadonarrival 8y agoAny progress on the Linux version?
- hemancuso 8y agohttps://twitter.com/expandrive/status/1000012200612950017 https://twitter.com/expandrive/status/1000012200612950017 Yup!
- rosege 8y agoDoes everything get stored in the cloud - even files that are in the 10Gb local cache?
- hemancuso 8y agoYes, unless you're offline - in which case it stays in the cache until a connection is available.
- deleted 8y ago[deleted]
- alexnewman 8y agoMinio can do disk caching
- j88439h84 8y agorclone.org is a great tool for syncing to/from cloud storage.
- mbrumlow 8y agoJust wondering. Would anybody want a block devise backed by s3 or other object storage? Local cache with snapshots that can be rolled back ? Maybe giving you a exobyte of addrisable storage ? This would be a actual block device not a fuse file system.
- deleted 8y ago[deleted]
- konschubert 8y agoI was actually thinking of doing this as a block device instead of a file system and I agree that it may be the more "natural" solution. However, the software tries to make predictions for which files will be accessed and doing this on a block level would be much harder. Plus, I had to learn fuse to write this and I thought I'd start off easy :D
- chrisper 8y agoYou mean like iSCSI in the cloud?
- mbrumlow 8y agoYeah like iSCSI backed by the cloud but with local cache for recently used data. And the ability to roll the entire device back to a state in the past (depending on how big you want your S3 bill to be).
- burmecia 8y agoNice work!. I also made a similar file system Zbox: https://github.com/zboxfs/zbox https://github.com/zboxfs/zbox. The difference is Zbox is a in-app file system focused on privacy, so FUSE is not supported intentionally. Although it already supports key-value store now, I am currently trying to extend its capability to cloud storage.
- johntash 8y agoZbox looks pretty interesting, thanks for linking it! Have you done any performance testing to compare zbox vs xfs/ext4/zfs/whatever on the same system? I saw the benchmark in the readme, but that doesn't necessarily show how much overhead or loss of performance there is over the native filesystem.
- burmecia 8y agoNo, I didn't. And I don't think it is necessary as Zbox is much more like an 'application-level' fs, obviously can't match the system level fs.
- mkl 8y agoI've been using SyncThing [1] recently, which does a similar thing but between your own devices (anything from Android to desktop to servers in the cloud). I've been using it on Linux and Windows, and it seems pretty good. [1] https://syncthing.net https://syncthing.net
- luhn 8y agoI don't think Syncthing is the same. As far as I recall, it stores the full sync folder to disk, similar to traditional sync services. This project only stores a cache on the disk.
- mkl 8y agoYou are correct. With SyncThing each device gets a full copy of whatever folders you've asked it to sync there. The FUSE aspect of Zero lets it do just-in-time file transfer and have no solid upper limit on storage, saving bandwidth and disk space at the cost of portability, latency, and redundancy.
- burkemw3 8y agoI built syncthingfuse to do partial syncs. Currently unmaintained, though. https://github.com/burkemw3/syncthingfuse https://github.com/burkemw3/syncthingfuse
- quantumwoke 8y agoAwesome idea! Recently I've been looking for a solution to automatically back up ~GBs of scraped data that is updated daily. Is this solution trustworthy enough? I was burned by OneDrive silently deleting data on a previous attempt.
- tempay 8y agoThe README states "Do not use in production" so I wouldn't trust it yet. For ~GBs of data per day is it necessary to use something that avoids having a full local copy? I'd have thought you could have a full local mirror backed up with Dropbox, Backblaze, rsync, rclone, etc.
- quantumwoke 8y agoPricing and reliability are my two main problems. I have access to 1Tb OneDrive but after it silently deleted 40Gb of irretrievable data I will never trust it again. Pricing wise Dropbox GDrive and others seem too expensive. Backblaze is the current frontrunner for sure, particularly with its deduplication facilities.
- konschubert 8y agoPlease don't use it in prod, this is work in progress.
- chrisper 8y agoI would recommend you borg (has deduplication) + rclone.
- Fnoord 8y agoSo, what all these filesystems do is having a local cache (in RAM or on SSD for better performance). The successors of NFS in the Linux kernel have been doing that all along. Not at all interesting IMO. WebDAV and Nextcloud already work (just like Rsync). What's interesting is applying some kind of encryption on top of it. For that, I use Cryptomator [1] which also works on mobile devices. [1] https://cryptomator.org https://cryptomator.org
- Shorel 8y agoThis is one of the reasons I use pCloud. It adds a huge disk device that is actually remote and only the recent files are locally cached. The other reason is Linux support.
- konschubert 8y agoAuthor here. Very glad to see that nobody has noticed how slow it is and nobody has made a comment about how messy code is in some places :D Working on both issues. But first I will add some instructions on how to run it.
- ohiovr 8y agoI think you should try funding your project by encouraging users to follow an affiliate link to buy storage from backblaze.
- tader 8y agoWhy would one prefer this over s3ql (https://bitbucket.org/nikratio/s3ql/ https://bitbucket.org/nikratio/s3ql/)?
- konschubert 8y agoZero has a cache where it keeps the most recently accessed files. For example, your latest 100GB of raw video recordings. Does s3ql do that? I skimmed over their documentation and could not see it but maybe I didn't look long enough?
- tader 8y ago"S3QL splits file contents into smaller blocks and caches blocks locally." -- http://www.rath.org/s3ql-docs/about.html#features http://www.rath.org/s3ql-docs/about.html#features It allows you to configure an arbitrary cache size, I've been using it with 60GB local cache.
- konschubert 8y agoBut are writes and reads really 100% local or do they require synchronous networking?
- fazilakhtar 8y agoKeybase does something similar with their file system kbfs(which is encrypted), they use fuse too. Local caching is limited to 10% of your disk space (if I remember correctly). Cool project though. Will definitely keep it in my radar.
- konschubert 8y agoDoes keybase sell could storage packages? Btw, I really love the idea of keybase, I hope they take off.
- fazilakhtar 8y agoNo, they have 250GB per user. IMO that is good for backups for now till they maybe start selling more storage? Same here, they're doing some really good work.
- swalsh 8y agoI built something similar years ago when I had a laptop with a really tiny hard drive. I would move a bunch of files to a cloud server, and left a bunch of files with no data on the hard drive. Then I wrote a file system filter driver which would watch for the files opening, catch the open request, and quickly download. It was better than walking around with an external drive. But it was super slow. After I upgraded the laptop, I had no further use for it.
- __adrien 8y agoReminds me of https://github.com/Azure/azure-storage-fuse https://github.com/Azure/azure-storage-fuse The code is good, maybe you'll find inspiration there.
- balkierode 8y agoNFS again?
- ohiovr 8y agoVery cool idea. I will be keeping an eye on this one. 1 tb of practical storage for 5 bucks a month. You could have a petabyte of storage for $5000 a month!