5 ms·
Not sure why it isn't listed in the overview, but this is based on SQLite (hence the name). HVent looked into the details yet but it looks pretty fun
by spydum 11y ago
Not sure why it isn't listed in the overview, but this is based on SQLite (hence the name). HVent looked into the details yet but it looks pretty fun
- eloff 11y agoI think SQLite is only used for the metadata, which seems like a reasonable choice. The speed of metadata operations won't matter next to the slowness of the read/write operations (slow relative to local SSDs.)
- atmosx 11y agoIsn't that kind limited? How exactly are two different clients going to access the metadata at the same time, if SQLite3 is used?!
- hvidgaard 11y agoI don't think that's the aim of the project. They do state that it's "especially suitable for online backup and archival". We typically have one network location where we store data for backup and archival, so only one machine would be having a connection anyway.
- viraptor 11y agoThey won't: https://bitbucket.org/nikratio/s3ql/wiki/FAQ#!can-i-access-an-s3ql-file-system-on-multiple-computers-simultaneously https://bitbucket.org/nikratio/s3ql/wiki/FAQ#!can-i-access-a... """In principle, both VPN and NFS/CIFS-alike functionality could be integrated into S3QL to allow simultaneous mounting using mount.s3ql directly. However, consensus among the S3QL developers is that this is not worth the increased complexity."""
- nbevans 11y agoSQLite supports multiple readers when Write Ahead Logging (WAL) is enabled. This mode is so good and transforms the whole operation of SQLite that it is shame that 1) it isn't the default, 2) more people don't know about it.
- aidos 11y agoThe way s3ql works is that it copies the db from s3 to your local machine. You make changes and then write the new version back to s3 later with the data. So it won't help in this case.
- nbevans 11y agoI see. Out of interest how does it write back the new version of the SQLite database file with only the differences? Does S3 support binary patching?
- aidos 11y agoI believe it uploads the whole thing each time. I think it may even upload it with a new index counter in the name to version it (but I can't find that in the docs now). http://www.rath.org/s3ql-docs/impl_details.html#metadata-storage http://www.rath.org/s3ql-docs/impl_details.html#metadata-sto...
- tobias3 11y agoI think you found the Achilles heel. This won't scale, so it is only suitable for small file systems.
- nbevans 11y agoIndeed, I'm good at that ;) It could probably be improved (subject to the nuances of S3 which I'm not fully familiar with). One way to fix it would be to copy the concept of SQLite's WAL mode. Use an appended write operation on S3 (if it supports it) to append to an existing file that contains the transaction log. Then at certain intervals (say every few thousand transactions) one can finally flush that log to be stored in the main database file. This would substantially reduce the number of times the database would need to be re-uploaded in full.
- edwintorok 11y agoThere is also `PRAGMA synchronous = NORMAL` that helps a lot. Your database will still be consistent and cannot become corrupted, although you do have to give up transaction durabibility after a power loss: https://www.sqlite.org/pragma.html#pragma_synchronous https://www.sqlite.org/pragma.html#pragma_synchronous
- devnonymous 11y agoSure looks interesting. It seems like a good idea for specific use cases, like for instance backup storage. The SQLite usage for the inode table is a neat hack. Implementation details here: http://www.rath.org/s3ql-docs/impl_details.html http://www.rath.org/s3ql-docs/impl_details.html
- pronoiac 11y agoAh, then I wouldn't expect concurrent connections to be on the menu, for a shared workspace.