4 ms·
Since this is pretty new, some background: WARC is a file format written and read by a rather small but specialized set of web crawler tools, most notably the
by uniqueuid 4y ago
Since this is pretty new, some background:
WARC is a file format written and read by a rather small but specialized set of web crawler tools, most notably the internet archive's tooling. For example, its java-based crawler heretrix produces warc files.
There are a couple of other very cool tools, such as warcprox, which can create web archives from web browser activity by acting as a proxy, pywb which play back the same to make archived versions browsable, and related libraries [1](shoutout to Noah Levitt, Ilya Kreymer and collaborators for building all this).
The file format itself is an iso standard, and it's very simple: Capture all http headers sent and received, and all http bodies sent and received, and do simple de-duplication based on digests of the body.
There is a companion format, CDX, which builds indexes from warc files (which in turn are just concatenated records, so rather robust).
Although all of this is great, I worry a bit about where we're heading with QUIC / udp-based protocols, websockets and other very involved protocols which ultimately make archival much harder.
If there's anything you can do to help these (or other) tools to keep our web's archival records alive and flowing, please do so.
[1] https://github.com/internetarchive/warcprox https://github.com/internetarchive/warcprox
- fforflo 4y agoThe web archiving community is surprisingly small and fragmented (in terms of tools) given its impact. Thankfully the .warc format looks pretty powerful and standard for the web we have so far (which is a lot! ). Now with the new protocols, dunno maybe its too soon to worry? Then again, maybe its an IPv4 / IPv6 analogy. > There is a companion format, CDX, which builds indexes from warc files (which in turn are just concatenated records, so rather robust). Good point. I's planning of combining this fact with the ATTACH option that SQLite has - allowing to query multiple database files [0] [0] https://www.sqlite.org/lang_attach.html https://www.sqlite.org/lang_attach.html
- uniqueuid 4y agoOh hi, thanks for building this! I haven't had the chance to play with it, but my hunch is that sqlite for warc can fill a great niche and would be much more portable (and probably performant). Allowing multiple DB files is a great idea, since that fundamentally enables large archives, cold-storage and so on.
- pstuart 4y agoSwarm and Union should be of interest to you: https://www.sqlite.org/swarmvtab.html https://www.sqlite.org/unionvtab.html
- ma2rten 4y agoCommon Crawl is also in WARC format.
- lijogdfljk 4y agoWould a WARC format reduce effort needed to make Reader-like programs? Ie strip pages of HTML cruft, leaving you with text, images, etc - the content?
- uniqueuid 4y agoNot at all. A WARC file gives you exactly what you would see on the wire, or in the network inspector tab of your browser. It does nothing to the content, and that's the point. The only thing you gain (and that's very important for other reasons as well) is an immutable ground truth to work from when creating the reader view of a given article.
- lijogdfljk 4y agoGotcha - yea i was hoping maybe it snapshotted the HTML or some such, side stepping some issues long dynamic text or JS shenanigans
- simonw 4y agoI've found the Readability.js library to be really good for that - here's my recipe for running it as a CLI: https://til.simonwillison.net/shot-scraper/readability https://til.simonwillison.net/shot-scraper/readability
- deleted 4y ago[deleted]