3 ms·
I have never used monolith to say with any certainty, but two things in your description are worth highlighting between the goals of WARC versus the umpteen baz
by mdaniel 3y ago
I have never used monolith to say with any certainty, but two things in your description are worth highlighting between the goals of WARC versus the umpteen bazillion "save this one page I'm looking at as a single file" type projects:
1. WARC is designed, as a goal, to archive the request-response handshake. It does not get into the business of trying to make it easy for a browser to subsequently display that content, since that's a browser's problem
2. Using your cited project specifically, observe the number of "well, save it but ..." options <https://github.com/Y2Z/monolith#options https://github.com/Y2Z/monolith#options> which is in stark contrast to the archiving goals I just spoke about. It's not a good snapshot of history if the server responded with `content-type: text/html;charset=iso-8859-1` back in the 90s but "modern tools" want everything to be UTF-8 so we'll just convert it, shall we? Bah, I don't like JavaScript, so we'll just toss that out, shall we? And so on
For 100% clarity: monolith, and similar, may work fantastic for any individual's workflow, and I'm not here to yuck anyone's yum; but I do want to highlight that all things being equal it should always be possible to derive monolith files from warc files because the warc files are (or at least have the goal of) perfect fidelity of what the exchange was. I would guess only pcap files would be of higher fidelity, but also a lot more extraneous or potentially privacy violating details
- eviks 3y agoWhat privacy violating details does WARC retain?
- Brian_K_White 3y agoThey said that a pcap could contain unwanted private info. I don't know what pcap is though, tcpdump?
- eviks 3y agothey said pcap could contain "a lot more", so I was wondering what "lesser" private info WARC has
- DrewADesign 3y agoDepends on what you’re archiving. Are you archiving a site with poorly protected authentication mechanisms? Something with personal data? In any case, it won’t have a bunch of internal network traffic that could be sensitive regardless of what you’re archiving.
- Brian_K_White 3y agoAnything that could ever appear in an http transaction. Which is to say, anything. This is like asking what possible sensitive information a piece of paper might have. If you just mean unseen info that wasn't obviously displayed right on the screen where the original user will know not to be careless with the archive of that page, the answer is still anything.
- TheTechRobo 3y agoEverything transmitted over HTTP. That includes cookies, passwords, etc. You need to be careful when writing WARCs.