2 ms·

For personal archiving, how does WARC compare to tools that can save pages as single HTML that can be directly opened in browser, like SingleFile, monolith? [0

For personal archiving, how does WARC compare to tools that can save pages as single HTML that can be directly opened in browser, like SingleFile, monolith? [0] https://chromewebstore.google.com/detail/singlefile/mpiodijhokgodhhofbcjdecpffjipkle https://chromewebstore.google.com/detail/singlefile/mpiodijh... [1] https://github.com/Y2Z/monolith https://github.com/Y2Z/monolith

brnt 3y ago
And here is was printing to PDF. How do these tools deal with content rendered by JavaScript, lazy loading, etc?
- nikisweeting 3y ago
  Singlefile uses the browser it's running in to execute all the JS before saving. The Singlefile CLI runs a headless browser to do the same. Most tools that can handle JS use puppeteer, playwright, or chrome headless directly. ArchiveBox uses Singlefile and Chrome headless directly for screenshot, PDF, and HTML saving, though we may switch to playwright soon. The best currently for high-fidelity with JS support is ArchiveWeb.page/Browsertrix though, they use puppeteer under the hood with a lot of magic to even make embedded YouTube videos work in the native player on replay.
mdaniel 3y ago
I have never used monolith to say with any certainty, but two things in your description are worth highlighting between the goals of WARC versus the umpteen bazillion "save this one page I'm looking at as a single file" type projects: 1. WARC is designed, as a goal, to archive the request-response handshake. It does not get into the business of trying to make it easy for a browser to subsequently display that content, since that's a browser's problem 2. Using your cited project specifically, observe the number of "well, save it but ..." options <https://github.com/Y2Z/monolith#options https://github.com/Y2Z/monolith#options> which is in stark contrast to the archiving goals I just spoke about. It's not a good snapshot of history if the server responded with `content-type: text/html;charset=iso-8859-1` back in the 90s but "modern tools" want everything to be UTF-8 so we'll just convert it, shall we? Bah, I don't like JavaScript, so we'll just toss that out, shall we? And so on For 100% clarity: monolith, and similar, may work fantastic for any individual's workflow, and I'm not here to yuck anyone's yum; but I do want to highlight that all things being equal it should always be possible to derive monolith files from warc files because the warc files are (or at least have the goal of) perfect fidelity of what the exchange was. I would guess only pcap files would be of higher fidelity, but also a lot more extraneous or potentially privacy violating details
- eviks 3y ago
  What privacy violating details does WARC retain?
  - Brian_K_White 3y ago
    They said that a pcap could contain unwanted private info. I don't know what pcap is though, tcpdump?
    - eviks 3y ago
      they said pcap could contain "a lot more", so I was wondering what "lesser" private info WARC has
      - DrewADesign 3y ago
        Depends on what you’re archiving. Are you archiving a site with poorly protected authentication mechanisms? Something with personal data? In any case, it won’t have a bunch of internal network traffic that could be sensitive regardless of what you’re archiving.
      - Brian_K_White 3y ago
        Anything that could ever appear in an http transaction. Which is to say, anything. This is like asking what possible sensitive information a piece of paper might have. If you just mean unseen info that wasn't obviously displayed right on the screen where the original user will know not to be careless with the archive of that page, the answer is still anything.
  - TheTechRobo 3y ago
    Everything transmitted over HTTP. That includes cookies, passwords, etc. You need to be careful when writing WARCs.

brnt 3y ago

And here is was printing to PDF. How do these tools deal with content rendered by JavaScript, lazy loading, etc?

nikisweeting 3y ago
Singlefile uses the browser it's running in to execute all the JS before saving. The Singlefile CLI runs a headless browser to do the same. Most tools that can handle JS use puppeteer, playwright, or chrome headless directly. ArchiveBox uses Singlefile and Chrome headless directly for screenshot, PDF, and HTML saving, though we may switch to playwright soon. The best currently for high-fidelity with JS support is ArchiveWeb.page/Browsertrix though, they use puppeteer under the hood with a lot of magic to even make embedded YouTube videos work in the native player on replay.

mdaniel 3y ago

I have never used monolith to say with any certainty, but two things in your description are worth highlighting between the goals of WARC versus the umpteen bazillion "save this one page I'm looking at as a single file" type projects: 1. WARC is designed, as a goal, to archive the request-response handshake. It does not get into the business of trying to make it easy for a browser to subsequently display that content, since that's a browser's problem 2. Using your cited project specifically, observe the number of "well, save it but ..." options <https://github.com/Y2Z/monolith#options https://github.com/Y2Z/monolith#options> which is in stark contrast to the archiving goals I just spoke about. It's not a good snapshot of history if the server responded with `content-type: text/html;charset=iso-8859-1` back in the 90s but "modern tools" want everything to be UTF-8 so we'll just convert it, shall we? Bah, I don't like JavaScript, so we'll just toss that out, shall we? And so on For 100% clarity: monolith, and similar, may work fantastic for any individual's workflow, and I'm not here to yuck anyone's yum; but I do want to highlight that all things being equal it should always be possible to derive monolith files from warc files because the warc files are (or at least have the goal of) perfect fidelity of what the exchange was. I would guess only pcap files would be of higher fidelity, but also a lot more extraneous or potentially privacy violating details

eviks 3y ago
What privacy violating details does WARC retain?
- Brian_K_White 3y ago
  They said that a pcap could contain unwanted private info. I don't know what pcap is though, tcpdump?
  - eviks 3y ago
    they said pcap could contain "a lot more", so I was wondering what "lesser" private info WARC has
    - DrewADesign 3y ago
      Depends on what you’re archiving. Are you archiving a site with poorly protected authentication mechanisms? Something with personal data? In any case, it won’t have a bunch of internal network traffic that could be sensitive regardless of what you’re archiving.
    - Brian_K_White 3y ago
      Anything that could ever appear in an http transaction. Which is to say, anything. This is like asking what possible sensitive information a piece of paper might have. If you just mean unseen info that wasn't obviously displayed right on the screen where the original user will know not to be careless with the archive of that page, the answer is still anything.
- TheTechRobo 3y ago
  Everything transmitted over HTTP. That includes cookies, passwords, etc. You need to be careful when writing WARCs.