4 ms·

I actually talked to the author of ArchiveBox about 1-2 weeks ago. Nice guy and it's good that he's also friendly with the Internet Archive. ArchiveBox uses WA

I actually talked to the author of ArchiveBox about 1-2 weeks ago. Nice guy and it's good that he's also friendly with the Internet Archive. ArchiveBox uses WARC as it's backing store: https://en.wikipedia.org/wiki/Web_ARChive https://en.wikipedia.org/wiki/Web_ARChive which is nice because it's standardized. We were discussing integrating Polar web archives along with ArchiveBox and maybe having some sort of standard to automatically submit these WARCs to the Internet Archive as part of your normal browsing activity. Polar has a similar web capture feature but it's not WARC https://getpolarized.io/ https://getpolarized.io/ (yet)... WARC is probably the easiest standard for Polar to adopt. Right now we use HTML encoded in JSON objects. When the user captures a web page we save all resources and store them in a PHZ file which you can keep as your own personal web archive. What I'd like to eventually do is update our extension to auto-capture web pages so you could use Polar's cloud storage feature to basically store every page you've ever visited. It really wouldn't be that much money per year. I did the math and it's about $50 per year to store your entire web history. If I can get Polar over to WARC that would mean that tools like ArchiveBox and Polar could interop but we could do things like automatically send your documents you browse to the Internet Archive. There's one huge problem though. What do we do about cookies and private data. I'm really not sure what to do there. It might be possible to strip this data for certain sites (news) without any risk of violating the users privacy.

peterwwillis 8y ago
Why don't they store everything as plain documents in a zip file and keep metadata in a json? Seems more future proof/easier for users to manipulate than WARC
- deleted 8y ago
  [deleted]
- burtonator 8y ago
  This is what Polar does but WARC is standardized... I think the argument against WARC though is that there isn't really much interchange. the point of standards is interchange IMO
  - krick 8y ago
    I am always inclined to use standards, because of the "right tool for the job" mentality, and it seems like an even more obvious thing to do when the standard is widely adopted already. But let's be honest: there are lots of really crappy standards in use today, because at some point they were widely adopted, and now it seems like a too big job to pull off to replace them with something sane. Because it all depends on adoption by the end users, and the end users don't care about the formats, they care about the neat tools that make the life easier. If POLAR gets popular enough, it can potentially dictate what is the standard. So I think it is best to compare the formats from the purely technical perspective. ZIP + json with as simple inner structure as possible has a huge advantage of being easy to handle by anyone: I can open (and actually read/modify) such a file with tools I will have on any machine, anytime. It is so simple and obvious, that I can write a script that packs some data in a format probably (at least partially) readable by your software in a minute. So: 1. Can I (meaningfully) do the same with this WARC thing? 2. What are the technical benefits of WARC over this unnamed (yet) file format? 3. Can WARC be losslessly converted into *.polar? And the other way around? 4. Are there tools to do #3? Is it tricky to implement on a new platform? I mean, if you can actually propose something better than WARC (or whatever) you can potentially save the world from another one WebDAV (or name your favorite horrible standard which we cannot rid of because everyone uses it).
    - samatman 8y ago
      I agree with the thrust of this and suggest taking it further by using SQLite for the underlying file format. The advantages of SQLite are too numerous to list, and it has a built-in compressed format so there's no bloat to worry about vs. ZIP files. As an example, this should make it practical, given a few iterations, to store multiple snapshots as deltas, deduplicating identical content. It also obviates having to base64 encode images and other binary assets.
      - mysterydip 8y ago
        One disadvantage is it introduces an application dependency. If the purpose of an archive is to preserve the data for future recall, years or decades later, having it in the most accessible format possible would be a priority.
- bnewbold 8y ago
  Have you looked at the WARC format? It's ridiculously simple, basically concatenated raw HTTP requests and responses, with some extra HTTP metadata headers mixed in (a la extra JSON metadata keys). You can open it with a text editor. Very simple and efficient to manipulate, and very efficient to iterate over or generate. https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/#file-and-record-model https://iipc.github.io/warc-specifications/specifications/wa... Arguably the biggest problem is that it isn't complex enough: there is no index of contents built-in (the standard .csv-like index format of URL/timestamp/hash/offset is called CDX). There aren't a ton of tools in the web archiving space in general, but almost all of the ones that do exist work with WARC. Existing tools (for interchange) include bulk indexing (for search, graph analysis, etc) and "replay" via web interface or browser-like application. Apart from specific centralized web archiving services that use WARC, there are several large public datasets, like Common Crawl, that are released in WARC format.
  - deleted 8y ago
    [deleted]
  - peterwwillis 8y ago
    Yeah, the fact that I have to convert them after is just an unnecessary extra step for me, so I'll stick to wget and httrack for my archives. Once I mirror them I can just copy the files anywhere and browse them on any browser/device.
nikisweeting 8y ago
Hi Kevin! Thanks for chiming in here! I only just saw ArchiveBox hit HN while I was away for the weekend. Just responding to everything now... (@everyone in this thread, go check out Polar, it's awesome!)

peterwwillis 8y ago

Why don't they store everything as plain documents in a zip file and keep metadata in a json? Seems more future proof/easier for users to manipulate than WARC

deleted 8y ago
[deleted]
burtonator 8y ago
This is what Polar does but WARC is standardized... I think the argument against WARC though is that there isn't really much interchange. the point of standards is interchange IMO
- krick 8y ago
  I am always inclined to use standards, because of the "right tool for the job" mentality, and it seems like an even more obvious thing to do when the standard is widely adopted already. But let's be honest: there are lots of really crappy standards in use today, because at some point they were widely adopted, and now it seems like a too big job to pull off to replace them with something sane. Because it all depends on adoption by the end users, and the end users don't care about the formats, they care about the neat tools that make the life easier. If POLAR gets popular enough, it can potentially dictate what is the standard. So I think it is best to compare the formats from the purely technical perspective. ZIP + json with as simple inner structure as possible has a huge advantage of being easy to handle by anyone: I can open (and actually read/modify) such a file with tools I will have on any machine, anytime. It is so simple and obvious, that I can write a script that packs some data in a format probably (at least partially) readable by your software in a minute. So: 1. Can I (meaningfully) do the same with this WARC thing? 2. What are the technical benefits of WARC over this unnamed (yet) file format? 3. Can WARC be losslessly converted into *.polar? And the other way around? 4. Are there tools to do #3? Is it tricky to implement on a new platform? I mean, if you can actually propose something better than WARC (or whatever) you can potentially save the world from another one WebDAV (or name your favorite horrible standard which we cannot rid of because everyone uses it).
  - samatman 8y ago
    I agree with the thrust of this and suggest taking it further by using SQLite for the underlying file format. The advantages of SQLite are too numerous to list, and it has a built-in compressed format so there's no bloat to worry about vs. ZIP files. As an example, this should make it practical, given a few iterations, to store multiple snapshots as deltas, deduplicating identical content. It also obviates having to base64 encode images and other binary assets.
    - mysterydip 8y ago
      One disadvantage is it introduces an application dependency. If the purpose of an archive is to preserve the data for future recall, years or decades later, having it in the most accessible format possible would be a priority.
bnewbold 8y ago
Have you looked at the WARC format? It's ridiculously simple, basically concatenated raw HTTP requests and responses, with some extra HTTP metadata headers mixed in (a la extra JSON metadata keys). You can open it with a text editor. Very simple and efficient to manipulate, and very efficient to iterate over or generate. https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/#file-and-record-model https://iipc.github.io/warc-specifications/specifications/wa... Arguably the biggest problem is that it isn't complex enough: there is no index of contents built-in (the standard .csv-like index format of URL/timestamp/hash/offset is called CDX). There aren't a ton of tools in the web archiving space in general, but almost all of the ones that do exist work with WARC. Existing tools (for interchange) include bulk indexing (for search, graph analysis, etc) and "replay" via web interface or browser-like application. Apart from specific centralized web archiving services that use WARC, there are several large public datasets, like Common Crawl, that are released in WARC format.
- deleted 8y ago
  [deleted]
- peterwwillis 8y ago
  Yeah, the fact that I have to convert them after is just an unnecessary extra step for me, so I'll stick to wget and httrack for my archives. Once I mirror them I can just copy the files anywhere and browse them on any browser/device.

nikisweeting 8y ago

Hi Kevin! Thanks for chiming in here! I only just saw ArchiveBox hit HN while I was away for the weekend. Just responding to everything now... (@everyone in this thread, go check out Polar, it's awesome!)