6 ms·
UtahFS: Encrypted File Storage
- blu_ 6y agoWill keep an eye on this!
- wakkaflokka 6y agoLove the idea of this. I've tried to use something similar (at least in function), called Cryptomator. However, Cryptomator would always choke on larger files or directories with thousands of items (e.g. my Google Photos gallery). Will be interested to see if this or another solution is what I'm looking for.
- bootlooped 6y agoBoxcryptor is a paid solution, similar to Cryptomator. I think I gave each one a try out on my phone, but ultimately just went back to using plain Google Drive and Google Photos. It fell into the category of "what is the real tangible benefit of this to me actually?" The real cost was added complexity, decreased reliability, a worse photos experience, and possibly another subscription cost.
- msh 6y agoThats also rather large. I have a cryptomator “vault” of approx 40 GB, performance on linux, windows and iOS seems fine.
- secfirstmd 6y agoTrue, the messy bit with Cryptomator is that it's not really very easily to share files with others. I've found it more of a tool really for an individual to use. I find myself sometimes training people on it a recommending it to people and then switching them to something like Tresorit when they have problems.
- jerednel 6y agoI just started using Cryptomator for sensitive files on a self-hosted NextCloud (since i access it via internet). The photos thing was an issue so I just sync those unencrypted and use Cryptomator for non-media files - i suppose if I had a photo that I wouldn't want exposed i'd just move it individually over.
- bronco21016 6y agoHow do you have it setup? I had similar issues when I was trying to access my WebDAV vault directly from Cryptomator on Ubuntu. I setup Nextcloud to sync directly with my local Ubuntu install and pointed Cryptomator at the vault in the synced directory and it works great! It opens all the files in the vault almost instantly and they’re navigable as if they're local files. Mobile is still a problem though. A gallery system would be a big help but you’re always stuck with a bandwidth limit unless you cache the vault locally which for my vault isn’t feasible and also undesirable as I don’t want the files on my device.
- imglorp 6y agoIPFS already has a storage and data layer as well as integrity and other properties. I wonder if it would be a good backend for this filesystem.
- jbverschoor 6y agoUntil your pk is compromised
- zaarn 6y agoYou'd have to find someone to permanently pin the data (nobody will be interested in random encrypted databoxes without being paid for it). And at that point you're just loosing performance compared to S3 or WebDAV.
- imglorp 6y agoYes you can pay to have someone pin it--there are pinning services--but you can also just replicate it among your own nodes. It would be comparable to Dropbox plus encryption. Another thought less well explored would be to swap space with friends such that you hold each others pins but you can't read your friends' encrypted blocks. This is kind of the premise behind filecoin, which lets you buy space (ie if you want someone to hold your blocks) and sell space (ie, mining is equal to holding someone's blocks).
- zaarn 6y agoI would point out that IPFS provides no encryption out of the box, you're just layering this below something that encrypts that could equally well work on Dropbox. Filecoin atleast has a integrated payment structure but modern storage providers are pretty damn cheap (I pay ~10$ for 3TB of data including ingress/egress) When I last checked by filecoin they could only barely compete with AWS storage options (which quickly get very expensive)
- autonome 6y agoFor clarification: IPFS has transport encryption by default. IPFS does not encrypt content itself for you.
- mtgx 6y agoDropbox and Google could easily encrypt the files locally with your own key prior to uploading them to their servers. But of course they won't do that, because then how else will they mine your data? Also, you probably shouldn't trust them even if they did offer this anyway.
- yjftsjthsd-h 6y agoAlso (and more charitably), I'm pretty sure at least Dropbox deduplicates files between customers.
- jtbayly 6y agoWhich allows them to charge many people for “storage space” that they aren’t actually using.
- deleted 6y ago[deleted]
- ec109685 6y agoIf the key for the file is the hash of the file, which they encrypt with the user’s passphrase, they can prevent anyone who didn’t originally have the file from being able to read it, including themselves.
- jeffbee 6y agoSure, if you don't want search or sharing or 90% of the other features of Google Drive then encrypting on the client side makes great sense. For people who demand that their computers are useful, it doesn't make any sense.
- rcxdude 6y agoSharing can be accomplished like encrypted file-hosting sites do: encoding a decryption key in the anchor of the URL. Search of course is harder, but it's a bit unreasonable that it's not an option at all.
- unethical_ban 6y agomtgx - you are dead. I haven't looked at native OpenZFS encryption... The method I used for a while was having a Veracrypt volume locally of all my files, and zfs-sending it to a remote location. It is a large file, several gigabytes. However, any updates made locally would get synced remotely at a block level vs. a file level, so the whole encrypted file didn't need synced. Much better than rsync, for example. I did not read the entire F'in article yet, I just stepped in while a CI/CD pipe was running at work :) BUT! this looks like a pretty accessible article about filesystems. I also wonder if it is named UtahFS because of the NSA facility near Salt Lake City.
- diablo1 6y ago> I also wonder if it is named UtahFS because of the NSA facility near Salt Lake City. Yeah I wondered this too. They don't say why they chose that wording. Maybe it's a subtle nod to that datacenter
- Kye 6y agoI'm not sure what the karma threshold is, but you might be able to open the dead comment and click vouch to make it replyable. I went ahead and did that since it seems to be a good comment. I don't know why so many of their recent comments are dead. Some I can understand, but others...
- linsomniac 6y agoI don't know about the crypto design of OpenZFS, but I have been running Ubuntu 20.04 with root pool using OpenZFS crypto since it's release. You have to make a small hack to the installer to do it, but otherwise it seems to be working great. I documented the installer hack at: https://linsomniac.gitlab.io/post/2020-04-09-ubuntu-2004-encrypted-zfs/ https://linsomniac.gitlab.io/post/2020-04-09-ubuntu-2004-enc... Before that I was running my storage server using LUKS on the raw devices and ZFS on top of that, since 2008, and that worked well.
- mhogomchungu 6y agoThere a couple of solutions that are fuse based and dealing with folder based encryption and gocryptfs has a page that compares them here[1]. It will be interesting to see how this project does against its competition. SiriKali[2] is a front end to a majority of these projects and it works on Linux, MACOS and windows. [1] https://nuetzlich.net/gocryptfs/comparison/ https://nuetzlich.net/gocryptfs/comparison/ [2] https://mhogomchungu.github.io/sirikali/ https://mhogomchungu.github.io/sirikali/
- deleted 6y ago[deleted]
- diablo1 6y agoInteresting name for a file system. Reminds me of this: https://en.wikipedia.org/wiki/Utah_Data_Center https://en.wikipedia.org/wiki/Utah_Data_Center
- Legogris 6y agoAlso https://en.wikipedia.org/wiki/Tahoe-LAFS https://en.wikipedia.org/wiki/Tahoe-LAFS
- linsomniac 6y agoWhile reading this I kept thinking "Isn't Utah the name of Zooko's FS?" Took me a minute to remember it was Tahoe.
- ianopolous 6y agoThey should use Peergos [1], which satisfies all their requirements. It's built on top of ipfs, encrypts metadata as well as data, and hides file sizes and directory structure too. A directory is indistinguishable from a small file. We can also use S3 compatible object storage, including allowing direct reads and writes from the browser. Disclaimer: I'm the founder. 1. https://github.com/peergos/peergos https://github.com/peergos/peergos
- libeclipse 6y agoPeergos doesn't hide access patterns as far as I can tell.
- ianopolous 6y agoWe use a similar structure to their oram, a merkle champ, with random labels. Some operations do move blocks to new labels, but reads currently don't because of the overhead as well as wanting to support multiple concurrent readers.
- slaymaker1907 6y agoORAM is pretty costly for a feature that many people don’t actually need. Yes it is log(n) for reads and writes, but the bigger downside is that ORAM by design thrashes any cache you happen to be using. I imagine this would be particularly noticeable for filename resolution with deep directories. Plus, ORAM provides no protection out of the box for timing based attacks.
- oarsinsync 6y agoFor anyone thinking about playing with this, please make sure you note this line: > Keep in mind that this system is not used in production at Cloudflare: it’s a proof-of-concept that I built
- jgrahamc 6y agoThat's right. The Research Team does all sorts of stuff. Some of it ends up as open source projects, some of it as production software.
- Tepix 6y agoOne way to achieve encrypted remote file storage is to combine Linux's EncFS² with some other type of remote storage protocol such as CIFS. The basic idea is to mount a CIFS volume from a server you don't trust with your unencrypted files, then you mount that directory again with EncFS to get to the unencrypted files. The main drawbacks are that EncFS will not hide your directory structure and file sizes and also that not all of the EncFS vulnerabilities that were discovered in 2014 have been addressed with an EncFS 2.0 release. -- ² or one of its similar alternatives of course
- eric1293 6y agoEncFS 2.0 does not seem to have been released. I don't recommend it due to security problems.
- eric1293 6y agoI find it difficult to set up a good system for encrypted storage. 1. Cryptomator: it's immature and buggy, especially the 1.5 version. See comments in forum.cryptomator. The files and folders disappear, vaults crash, vaults fail to mount, etc. Boxcryptor is the paid version and not buggy. But it's not open source. 2. EncFS. Has security issues that haven't been resolved. 3. CryFS. Too slow and immature. 4. Encrypted backup, like rclone or duplicity. These are not sync tools. 5. eCryptfs: Used for Ubuntu home encryption (even then somewhat outdated), not for cloud. 6. AWS KMS: server side encryption; amazon has the keys. 7. Gocryptfs: It's OK. Reasonably fast. Cons: command line only, and for Linux. Uses OpenSSL library which isn't all that secure. It seems to me gocrytfs is the best among these.
- Legogris 6y agoWhat’s wrong with LUKS / tomb?
- eric1293 6y agoIt's one big container. A small change means the whole file has to be uploaded again. If you are lucky for Dropbox blocks are synced only. Also it lacks authentication. The snapshots of the XTS mode are prone to certain attacks.
- Legogris 6y agoRegarding your first paragraph, that’s always going to be a tradeoff that has to be assessed based on threat model and resource budget, since doing it differently will inherently leak sidechannel information, right? The second one, TIL and good points!
- fbriff 6y agoI would also add: 8. Securefs [1] Gocryptfs has a comparison of these projets, here [2]. Focused on Windows only, to my experience, securefs is the one that is working the best as it is not using dokany but winfsp (FUSE for Windows). With all other solutions using dokany, the copy or sync of large number of files is damn slow or hanging. [1] https://github.com/netheril96/securefs https://github.com/netheril96/securefs [2] https://nuetzlich.net/gocryptfs/comparison/ https://nuetzlich.net/gocryptfs/comparison/
- convivialdingo 6y agoAwesome job! I prototyped a very similar FS for IBM Research a few years back, but I like your design better! They used it for Hadoop. Love the crypto fs work you're doing at Cloudflare.