3 ms·
I agree with the thrust of this and suggest taking it further by using SQLite for the underlying file format. The advantages of SQLite are too numerous to list
by samatman 8y ago
I agree with the thrust of this and suggest taking it further by using SQLite for the underlying file format.
The advantages of SQLite are too numerous to list, and it has a built-in compressed format so there's no bloat to worry about vs. ZIP files.
As an example, this should make it practical, given a few iterations, to store multiple snapshots as deltas, deduplicating identical content. It also obviates having to base64 encode images and other binary assets.
- mysterydip 8y agoOne disadvantage is it introduces an application dependency. If the purpose of an archive is to preserve the data for future recall, years or decades later, having it in the most accessible format possible would be a priority.
- aloer 8y agoRemembered I read something about this some time ago, looked it up: SQLite is a recommended storage format (https://www.sqlite.org/locrsf.html https://www.sqlite.org/locrsf.html) by the library of congress > As of this writing (2018-05-29) the only other recommended storage formats for datasets are XML, JSON, and CSV
- greglindahl 8y agoThe Library of Congress stores its web archives as WARCs.
- mysterydip 8y agoGood to know, thanks!
- rakoo 8y agoVery true, but I'd say SQlite is the one application dependency that could have a pass: - they promise support until at least 2050 (https://sqlite.org/lts.html https://sqlite.org/lts.html) - if a promise isn't enough, SQlite is to be supported for the entire lifetime of the Airbus A350 airframe (https://mobile.twitter.com/copiousfreetime/status/675834543304081409?lang=en https://mobile.twitter.com/copiousfreetime/status/6758345433...). I would assume Airbus will be paying for it. A real problem with application dependency is that it makes it extremely hard to create a completely independent alternative. A good protocol should have multiple implementations with nothing in common otherwise you rely on specific implementation details. The infamous case of WebSQL should remind us of that.
- mysterydip 8y agoAlso excellent points I waan't aware of, thanks!