4 ms·

This is what Polar does but WARC is standardized... I think the argument against WARC though is that there isn't really much interchange. the point of standard

This is what Polar does but WARC is standardized... I think the argument against WARC though is that there isn't really much interchange. the point of standards is interchange IMO

krick 8y ago
I am always inclined to use standards, because of the "right tool for the job" mentality, and it seems like an even more obvious thing to do when the standard is widely adopted already. But let's be honest: there are lots of really crappy standards in use today, because at some point they were widely adopted, and now it seems like a too big job to pull off to replace them with something sane. Because it all depends on adoption by the end users, and the end users don't care about the formats, they care about the neat tools that make the life easier. If POLAR gets popular enough, it can potentially dictate what is the standard. So I think it is best to compare the formats from the purely technical perspective. ZIP + json with as simple inner structure as possible has a huge advantage of being easy to handle by anyone: I can open (and actually read/modify) such a file with tools I will have on any machine, anytime. It is so simple and obvious, that I can write a script that packs some data in a format probably (at least partially) readable by your software in a minute. So: 1. Can I (meaningfully) do the same with this WARC thing? 2. What are the technical benefits of WARC over this unnamed (yet) file format? 3. Can WARC be losslessly converted into *.polar? And the other way around? 4. Are there tools to do #3? Is it tricky to implement on a new platform? I mean, if you can actually propose something better than WARC (or whatever) you can potentially save the world from another one WebDAV (or name your favorite horrible standard which we cannot rid of because everyone uses it).
- samatman 8y ago
  I agree with the thrust of this and suggest taking it further by using SQLite for the underlying file format. The advantages of SQLite are too numerous to list, and it has a built-in compressed format so there's no bloat to worry about vs. ZIP files. As an example, this should make it practical, given a few iterations, to store multiple snapshots as deltas, deduplicating identical content. It also obviates having to base64 encode images and other binary assets.
  - mysterydip 8y ago
    One disadvantage is it introduces an application dependency. If the purpose of an archive is to preserve the data for future recall, years or decades later, having it in the most accessible format possible would be a priority.
    - aloer 8y ago
      Remembered I read something about this some time ago, looked it up: SQLite is a recommended storage format (https://www.sqlite.org/locrsf.html https://www.sqlite.org/locrsf.html) by the library of congress > As of this writing (2018-05-29) the only other recommended storage formats for datasets are XML, JSON, and CSV
      - greglindahl 8y ago
        The Library of Congress stores its web archives as WARCs.
      - mysterydip 8y ago
        Good to know, thanks!
    - rakoo 8y ago
      Very true, but I'd say SQlite is the one application dependency that could have a pass: - they promise support until at least 2050 (https://sqlite.org/lts.html https://sqlite.org/lts.html) - if a promise isn't enough, SQlite is to be supported for the entire lifetime of the Airbus A350 airframe (https://mobile.twitter.com/copiousfreetime/status/675834543304081409?lang=en https://mobile.twitter.com/copiousfreetime/status/6758345433...). I would assume Airbus will be paying for it. A real problem with application dependency is that it makes it extremely hard to create a completely independent alternative. A good protocol should have multiple implementations with nothing in common otherwise you rely on specific implementation details. The infamous case of WebSQL should remind us of that.
      - mysterydip 8y ago
greglindahl 8y ago
There actually is interchange of WARCs in the web archiving community.

This is what Polar does but WARC is standardized... I think the argument against WARC though is that there isn't really much interchange. the point of standards is interchange IMO

krick 8y ago

I am always inclined to use standards, because of the "right tool for the job" mentality, and it seems like an even more obvious thing to do when the standard is widely adopted already. But let's be honest: there are lots of really crappy standards in use today, because at some point they were widely adopted, and now it seems like a too big job to pull off to replace them with something sane. Because it all depends on adoption by the end users, and the end users don't care about the formats, they care about the neat tools that make the life easier. If POLAR gets popular enough, it can potentially dictate what is the standard. So I think it is best to compare the formats from the purely technical perspective. ZIP + json with as simple inner structure as possible has a huge advantage of being easy to handle by anyone: I can open (and actually read/modify) such a file with tools I will have on any machine, anytime. It is so simple and obvious, that I can write a script that packs some data in a format probably (at least partially) readable by your software in a minute. So: 1. Can I (meaningfully) do the same with this WARC thing? 2. What are the technical benefits of WARC over this unnamed (yet) file format? 3. Can WARC be losslessly converted into *.polar? And the other way around? 4. Are there tools to do #3? Is it tricky to implement on a new platform? I mean, if you can actually propose something better than WARC (or whatever) you can potentially save the world from another one WebDAV (or name your favorite horrible standard which we cannot rid of because everyone uses it).

samatman 8y ago
I agree with the thrust of this and suggest taking it further by using SQLite for the underlying file format. The advantages of SQLite are too numerous to list, and it has a built-in compressed format so there's no bloat to worry about vs. ZIP files. As an example, this should make it practical, given a few iterations, to store multiple snapshots as deltas, deduplicating identical content. It also obviates having to base64 encode images and other binary assets.
- mysterydip 8y ago
  One disadvantage is it introduces an application dependency. If the purpose of an archive is to preserve the data for future recall, years or decades later, having it in the most accessible format possible would be a priority.
  - aloer 8y ago
    Remembered I read something about this some time ago, looked it up: SQLite is a recommended storage format (https://www.sqlite.org/locrsf.html https://www.sqlite.org/locrsf.html) by the library of congress > As of this writing (2018-05-29) the only other recommended storage formats for datasets are XML, JSON, and CSV
    - greglindahl 8y ago
      The Library of Congress stores its web archives as WARCs.
    - mysterydip 8y ago
      Good to know, thanks!
  - rakoo 8y ago
    Very true, but I'd say SQlite is the one application dependency that could have a pass: - they promise support until at least 2050 (https://sqlite.org/lts.html https://sqlite.org/lts.html) - if a promise isn't enough, SQlite is to be supported for the entire lifetime of the Airbus A350 airframe (https://mobile.twitter.com/copiousfreetime/status/675834543304081409?lang=en https://mobile.twitter.com/copiousfreetime/status/6758345433...). I would assume Airbus will be paying for it. A real problem with application dependency is that it makes it extremely hard to create a completely independent alternative. A good protocol should have multiple implementations with nothing in common otherwise you rely on specific implementation details. The infamous case of WebSQL should remind us of that.
    - mysterydip 8y ago

greglindahl 8y ago

There actually is interchange of WARCs in the web archiving community.