8 ms·
S3QL: an S3 FS with encryption, de-dup, immutable trees and snapshotting
- spydum 11y agoNot sure why it isn't listed in the overview, but this is based on SQLite (hence the name). HVent looked into the details yet but it looks pretty fun
- eloff 11y agoI think SQLite is only used for the metadata, which seems like a reasonable choice. The speed of metadata operations won't matter next to the slowness of the read/write operations (slow relative to local SSDs.)
- atmosx 11y agoIsn't that kind limited? How exactly are two different clients going to access the metadata at the same time, if SQLite3 is used?!
- hvidgaard 11y agoI don't think that's the aim of the project. They do state that it's "especially suitable for online backup and archival". We typically have one network location where we store data for backup and archival, so only one machine would be having a connection anyway.
- viraptor 11y agoThey won't: https://bitbucket.org/nikratio/s3ql/wiki/FAQ#!can-i-access-an-s3ql-file-system-on-multiple-computers-simultaneously https://bitbucket.org/nikratio/s3ql/wiki/FAQ#!can-i-access-a... """In principle, both VPN and NFS/CIFS-alike functionality could be integrated into S3QL to allow simultaneous mounting using mount.s3ql directly. However, consensus among the S3QL developers is that this is not worth the increased complexity."""
- nbevans 11y agoSQLite supports multiple readers when Write Ahead Logging (WAL) is enabled. This mode is so good and transforms the whole operation of SQLite that it is shame that 1) it isn't the default, 2) more people don't know about it.
- aidos 11y agoThe way s3ql works is that it copies the db from s3 to your local machine. You make changes and then write the new version back to s3 later with the data. So it won't help in this case.
- nbevans 11y agoI see. Out of interest how does it write back the new version of the SQLite database file with only the differences? Does S3 support binary patching?
- aidos 11y agoI believe it uploads the whole thing each time. I think it may even upload it with a new index counter in the name to version it (but I can't find that in the docs now). http://www.rath.org/s3ql-docs/impl_details.html#metadata-storage http://www.rath.org/s3ql-docs/impl_details.html#metadata-sto...
- tobias3 11y agoI think you found the Achilles heel. This won't scale, so it is only suitable for small file systems.
- nbevans 11y agoIndeed, I'm good at that ;) It could probably be improved (subject to the nuances of S3 which I'm not fully familiar with). One way to fix it would be to copy the concept of SQLite's WAL mode. Use an appended write operation on S3 (if it supports it) to append to an existing file that contains the transaction log. Then at certain intervals (say every few thousand transactions) one can finally flush that log to be stored in the main database file. This would substantially reduce the number of times the database would need to be re-uploaded in full.
- devnonymous 11y agoSure looks interesting. It seems like a good idea for specific use cases, like for instance backup storage. The SQLite usage for the inode table is a neat hack. Implementation details here: http://www.rath.org/s3ql-docs/impl_details.html http://www.rath.org/s3ql-docs/impl_details.html
- pronoiac 11y agoAh, then I wouldn't expect concurrent connections to be on the menu, for a shared workspace.
- kordless 11y ago> Immutable Trees. Directory trees can be made immutable, so that their contents can no longer be changed in any way whatsoever. This can be used to ensure that backups can not be modified after they have been made. That would be perfect for things we don't want to ever change...like container images. Or configuration files.
- porker 11y agoWhat is the benefit of immutable trees over versioned backups - or to put it another way: why wouldn't you keep a change history in your backups?
- aidos 11y agoI might be mistaken but I believe that feature refers to the ability to make a bit of your file system read only. It's just there so you don't accidentally wipe out a backup you meant to keep. You would still do versioned backups and then make one a month immutable. Or whatever your strategy is.
- kordless 11y agoBackups are for data generated by people using applications running on services provided by infrastructure. What I'm referring to is the configuration portion of launching and running a service, not the data generated after a human uses it. The thing that pops out of this is TRUST. We need to be able to trust an application we are running is the one we want to run, who was responsible for writing it, who's responsible for running it, and all the bits in-between.
- stubish 11y agoThe closest thing I've seen to this is Jungle Disk, which is closed source, offers a less fully featured file system, but does have Windows and Mac clients.
- _delirium 11y agoThere was also ObjectiveFS discussed on HN a few days ago, also closed source: https://news.ycombinator.com/item?id=10117506 https://news.ycombinator.com/item?id=10117506 Main difference appears to be that it supports concurrent mounts, while S3QL can be mounted by one client at a time.
- LeoPanthera 11y agoSurely the "main" difference is that ObjectiveFS is a paid service with prices starting at $7/mount. S3QL is FOSS.
- porker 11y agoJungleDisk works, but the clients are buggy PoS that crash/hang whenever they choose. I've also had JungleDisk's server daemon silently quit working & lock up on some servers, which is scary when you go to restore a file and realise no backups have happened in a while. It doesn't engender trust. That said, I still use it as it's easier & less error-prone than setting up my own backup system with tarsnap.
- jaytagdamian 11y agoThe reporting is very poor also - if the backups don't run, you arent alerted to failure or success, they just dont appear on the emailed reports. Many a server has not been backed up because of this...
- bretpiatt 11y agoEach backup job reports when successfully completed to provide an audit trail for your successful backups. If you are scheduled to run daily backups and you do not see the daily email the backup did not successfully run. We're open to understanding if there is a better behavior for this process. My worry is if we email on both cases people see the message come through and ignore the details where it shows 'failed'.
- jerrac 11y agoHow well would this work for something like pointing OwnCloud storage at? Most of my cloud experience is with an internal vmware cluster, so I'm not sure how to evaluate something like this.
- btgeekboy 11y agoI'm not sure what the point would be - OwnCloud already knows how to work with S3 directly.
- jerrac 11y agoJust as an example app. Basically, use S3 the same way you use an in house SAN. Also, if I remember correctly, S3 integration is per account, and only if you enable the external file storage app. It wouldn't let you store all data on S3. So, mounting a bucket as the OwnCloud data folder would be the way I'd get that.
- gaul 11y agos3fs also layers a filesystem on top of S3 but preserves the native object format: https://github.com/s3fs-fuse/s3fs-fuse https://github.com/s3fs-fuse/s3fs-fuse This approach allows use of other tools like s3cmd and the Amazon web console but prevents advanced features like deduplication and snapshotting.
- aorth 11y agoI'm curious to see how this compares to tarsnap for encrypted backups. One cool thing that people might not know about tarsnap is that you can generate read-only subkeys of your main key! https://www.tarsnap.com/man-tarsnap-keymgmt.1.html https://www.tarsnap.com/man-tarsnap-keymgmt.1.html
- cperciva 11y agoI'm also curious. I hadn't seen S3QL before it appeared at the top of HN. Two things jump out at me from a quick glance at the site: 1. "S3QL splits files in blocks of a configurable size (default: 10 MB)" <-- this is quite a large block size and will result in significantly less deduplication than tarsnap's variable-length average 64kB blocks. (On the other hand, large blocks significantly reduce the amount of work and RAM needed; S3QL's tuning probably makes sense for a "live" filesystem.) 2. "all data can AES encrypted with a 256 bit key" <-- leaving aside the grammatical problem, I can't find anything beyond this about key management. At best this means that anyone who can decrypt data can encrypt it, and vice versa; but the usual rule about crypto documentation applies here: If people don't realize that the details matter, they're probably doing something wrong. (Unfortunately it's 1AM and I'm allergic to snakes, or else I would dig into the code to see exactly how their crypto works.)
- a3_nm 11y agoA difference with Tarsnap is that s3ql is free and open-source software (both the client and server), so you can also use it on your own server, your friends' servers, etc. I used s3ql to back up my stuff on servers where I had shell access and disk space but didn't trust root. By contrast, from what I understand, tarsnap requires you to use the official server.
- witten 11y agoHaving used both S3QL and Tarsnap.. In my experience, S3QL's use of remote mounted filesystems is fundamentally unsuited to unattended backups. On many occasions, S3QL's mounted filesystem would break mid-backup due to network issues, and stay mounted and unable to recover. Then, from that point forward, no backups would ever work again without manual intervention because the mount point was already open. Additionally, S3QL would periodically issue new releases that didn't support old versions of the file format, or only supported them a set number of releases back. So if you didn't upgrade frequently enough, you'd find yourself with a release that refused to read your existing gigabytes of backup data. And then at that point, you have to do a binary search to find and recompile old releases in the vain attempt to resurrect your data and avoid having to do a full backup from scratch. Bottom line: Stay far away from S3QL and instead use Attic, Borg, or Tarsnap. Also, if you select Attic or Borg, check out Atticmatic: https://torsion.org/atticmatic/ https://torsion.org/atticmatic/
- indiv0 11y agoI was recently looking into options for offsite, encrypted (at-rest & in-transit), deduplicated, snapshotted, backups to S3. S3QL was one of the options I investigated, but unfortunately their crypto is MAC-then-encrypt [0], which is no good. A small sample of the other options considered: * EncFS + rsnapshot/rdiff-backup * Weak cryptography in EncFS [1] * Duplicity * Uses forward snapshots, so recovery takes longer with each successive snapshot, unless you upload a "fresh" copy of the data * ext4 on LVM on LUKS on a loopback device provided by s3backer [2] * OK I'll admit this one is pretty far out there, but it actually works decently well (ignoring the massive FUSE overhead) * Downside: a slight hiccup in network connectivity guarantees data corruption * Attic * Known data corruption issues with large datasets [3] Does anyone know of anything that can provide the above requirements? At this point it's really more of a thought experiment, as I've decided to go with Tarsnap [4] for my use-case. [0]: http://www.rath.org/s3ql-docs/impl_details.html http://www.rath.org/s3ql-docs/impl_details.html [1]: https://defuse.ca/audits/encfs.htm https://defuse.ca/audits/encfs.htm [2]: https://github.com/archiecobbs/s3backer https://github.com/archiecobbs/s3backer [3]: http://librelist.com/browser/attic/2015/3/31/comparison-of-attic-vs-bup-vs-obnam/#cbbe599389a20c787a74b137dc78fb1a http://librelist.com/browser/attic/2015/3/31/comparison-of-a... [4]: https://www.tarsnap.com/ https://www.tarsnap.com/
- lfam 11y ago> offsite, encrypted (at-rest & in-transit), deduplicated, snapshotted, backups to S3 I think restic[1] is going for that. I don't think it's "ready" yet, though. [1] https://github.com/restic/restic https://github.com/restic/restic
- timf 11y agoIf this is interesting, recommend looking at Tahoe-LAFS too: https://www.tahoe-lafs.org/trac/tahoe-lafs https://www.tahoe-lafs.org/trac/tahoe-lafs (And https://leastauthority.com/how_it_works https://leastauthority.com/how_it_works )
- IgorPartola 11y agoThis looks cool and I am going to give this a try. The problem for me is, as is usually the case with such project, the packaging. If this thing is production-ready, then why must I check for installed dependencies by running random commands [1]? If it's a Python project, why isn't it distributed on PyPI? I don't want to download stuff from BitBucket manually and install it by executing setup.py. I understand that the project supports multiple OS's. That's great. But there are simple steps that can be taken to make installing this thing via automated tools (Puppet, Chef, Ansible, etc.) easier than how it's set up now. A Debian package would be so nice for Ubuntu/Debian. [1] http://www.rath.org/s3ql-docs/installation.html#dependencies http://www.rath.org/s3ql-docs/installation.html#dependencies
- repomies691 11y agoThere is plenty extensive packaging: https://bitbucket.org/nikratio/s3ql/wiki/Installation https://bitbucket.org/nikratio/s3ql/wiki/Installation The documentation is just somewhat messed up...
- IgorPartola 11y agoNevermind the things I said above. Thank you for pointing this out. This is exactly what I was looking for.
- DavideNL 11y agoSo would it be possible to mount this as a Volume in OS X and create (encrypted) Time Machine backups to Amazon S3?