3 ms·
I have absolutely no intention to spread FUD about client-side encryption, and in fact I don't think Dropbox's speed advantage has anything to do with (lack of)
by kijin 13y ago
I have absolutely no intention to spread FUD about client-side encryption, and in fact I don't think Dropbox's speed advantage has anything to do with (lack of) encryption. Encryption shouldn't have any noticeable impact on sync speed, since modern CPUs can encrypt a file a hundred times over in the time it takes to upload/download it. I agree with you and suspect that it has more to do with crappy architecture.
Anecdote: SpiderOak takes 5 minutes synchronizing metadata with the server if I start it on a laptop that I only use occasionally. That's how long it takes for the client to find out what files I changed in my other machines in the meantime, including all the files that have nothing to do with this particular machine. Nothing gets synched during this time, and the client sometimes consumes a whole CPU core for extended periods. It's very annoying when I just want to see a few files that I know I changed in another machine. I understand that the protocol needs to be different if all the encryption is done by the client, but this is just ridiculous. Wuala is no better in my experience, although YMMV depending on your use case and filesystem structure. (I have over 50GB of data in my SpiderOak account, made up of a quarter million files. At one point I had over 100GB stored in there, and it was a PITA to sync any files at all.)
As an interesting comparison, Microsoft SkyDrive performs just as terribly as SpiderOak does when you give it a few hundred folders with ~100K files to sync, despite the fact that it does no client-side encryption. It just sits idle for half an hour or more, scarcely utilizing any network resources. I have no idea what it's doing. Maybe it does sleep(random()); between each upload. In any case, encryption isn't making it slow. It's just bad app design, bad protocol design, and perhaps overloaded servers.
Dropbox, on the other hand, immediately begins to upload/download my files, and once it's done scanning any large folders, its speed is only limited by the bandwidth of my Internet connection. When I need to do a presentation, I can just work on the .pptx file in my Dropbox on my desktop PC, take my laptop to the conference room, and expect the .pptx to be there as soon as I wake my laptop. That's the meaning of convenience. By the same metric, none of Dropbox's commercial competitors (perhaps with the exception of Google Drive, which I don't use) are even close to convenient.
- objectivefs 13y agoClient-side encryption can be very fast when done right. For your use case a filesystem stored in the cloud might be a better solution, that way only the data you are using is downloaded to each machine and the writes are done immediately. Give ObjectiveFS[1] a try, the free preview is available for a few more days, and see how fast client-side encryption can be. [1] https://objectivefs.com https://objectivefs.com
- kijin 13y agoInteresting, but an all-cloud solution would seem to be a better fit for servers with fat pipes (especially EC2) than for individuals with crappy residential connections. I have fiber at home, though, so I might try it out sometime.
- rarrrrrr 13y agoSpiderOak cofounder here. It's not really about the encryption directly; it's about the way the database design has to change to support zero knowledge (the server can no longer do the database work.) Things like garbage collection have to happen client side. Described in detail here: https://spideroak.com/blog/20091026143000-why-and-how-spideroak-architecture-is-different-than-other-online-storage-services-the-surprising-consequences-on-database-design-from-our-zero-knowledge-approach-to-privacy https://spideroak.com/blog/20091026143000-why-and-how-spider... FYI, when SpiderOak first starts running, like any backup software, it has to scan your filesystem to see what may have changed while it was shutdown, and that can take a lot of IO and CPU if you have many folders and/or many small files (hundreds of thousands). (SpiderOak works with any folders you like, including externa/network drives, not just one folder like Dropbox.) In a profile, basically none of this CPU time is spent in the crypto module, just mostly for unserializing the per-folder journal to compare state. SpiderOak does some optimizations with directory hashing to avoid having to open the journal to scan for specific changes in unchanged folders, but there are some situations where that optimization is defeated (such as if you remount a file system with changed gids, inode IDs, timestamps, etc.) Anyway, once it's running, it uses the operating system facilities on Linux/Mac/Windows to notice changes automatically so rescanning the whole backup selection after startup is not usually necessary. In any case, thanks for your interest in SpiderOak. :)
- Pyramids 13y agoHello Alan, I have a quick question for you which might be of interest to others as well: Why isn't SpiderOak open source yet? I've read your FAQ answer[1] on this, however it doesn't really give a concrete explanation, besides "soon" and "licencing concerns" which is definitely disappointing. I currently use my own, EncFS based sync solution, however I'd love to be able to use SpiderOak, or a third party open source application which supported syncing with SpiderOak (if you'd ever consider exposing an API/Protocol which allowed such.) This is the only reason I'm not using your service currently, however I love the concept and hope you'll take it into consideration. It's likely I'm not the only one with similar concerns. [1]: https://spideroak.com/faq/questions/35/why_isnt_spideroak_open_source_yet_when_will_it_be/ https://spideroak.com/faq/questions/35/why_isnt_spideroak_op...
- 13y ago