5 ms·
Fun fact, modern wget binaries can emit warc natively, if one wanted to kick the tires on such a thing $ wget --version GNU Wget 1.21.2 built on linux-gnu.
by mdaniel 3y ago
Fun fact, modern wget binaries can emit warc natively, if one wanted to kick the tires on such a thing
$ wget --version
GNU Wget 1.21.2 built on linux-gnu.
-cares +digest -gpgme +https +ipv6 +iri +large-file -metalink +nls
+ntlm +opie +psl +ssl/openssl
$ wget --help | grep warc
--warc-file=FILENAME save request/response data to a .warc.gz file
--warc-header=STRING insert STRING into the warcinfo record
--warc-max-size=NUMBER set maximum size of WARC files to NUMBER
--warc-cdx write CDX index files
--warc-dedup=FILENAME do not store records listed in this CDX file
--no-warc-compression do not compress WARC files with GZIP
--no-warc-digests do not calculate SHA1 digests
--no-warc-keep-log do not store the log file in a WARC record
--warc-tempdir=DIRECTORY location for temporary files created by the
The ArchiveBox project (which gets reposted on the regular: e.g. https://news.ycombinator.com/item?id=38954189 https://news.ycombinator.com/item?id=38954189 ) also saves in WARC https://github.com/ArchiveBox/ArchiveBox#output-formats https://github.com/ArchiveBox/ArchiveBox#output-formats although I've personally not used it to comment further
- iansinnott 3y agoI've used it for about a year. ArchiveBox works well. My only complaint is that there's no JSON API to trigger archiving a page--it's primarily a CLI tool despite being built with Django. This can be worked around of course, and there's a browser plug-in to auto-archive sites you visit based on a whitelist.
- nikisweeting 3y agoAPI is coming soon (relatively, it's still a one-man project)! Stay tuned https://github.com/ArchiveBox/ArchiveBox/issues/496 https://github.com/ArchiveBox/ArchiveBox/issues/496 I have an event-sourcing refactor in progress now to allow us to pluginize functionality like the API (similar to Home Assistant with a plugin app store), it will take a month or two. Next up is the REST API using the new plugin system.
- iansinnott 3y agoYeah, thanks for linking to the issue. I'm one of the upvotes on that. However, the age of that issue suggests to me that current contributors are (or have historically been) uninterested in a REST API. Still, would be great to see such an update, and I'm still using ArchiveBox either way. (Just read your bio, thanks for creating this)
- nikisweeting 3y agoIt's not so much that the team isn't interested in building it, more that there there were some volunteers that indicated they started work on APIs already so I moved on to work on other areas of the codebase. Those draft PRs have since been closed as stale... so I'm planning to come back to doing the API myself in the next major version cycle.
- seabass-labrax 3y agoVery helpful to know that! Zimit[1] also uses warc files as an intermediate step to producing Zim files. You can use these Zim files to read and search websites offline with the excellent app Kiwix[2]. I think 'Kiwix for Android' and the Kiwix PWA support Zim files made with Zimit, with support by the desktop Kiwix application currently work-in-progress. Other useful information about archiving websites is available from Webrecorder[3]. [1]: https://youzim.it/ https://youzim.it/ [2]: https://kiwix.org/ https://kiwix.org/ [3]: https://webrecorder.net/ https://webrecorder.net/
- linuxandrew 3y agoAre there any reasons why Kiwix doesn't use WARC/WACZ directly?
- nikisweeting 3y agoI think ZIM files contain way more data (like 80gb for Wikipedia) than a typical WARC, they're optimized for different things.
- pjot 3y agoYou can also generate warc natively in iOS! On a website, tap the share sheet icon, at the top of the tray tap “options”, and finally “web archive”. From there you can save/send/copy the file to your desired location.
- walterbell 3y agoThanks, that's much better than iOS PDFs which don't preserve links.
- mdaniel 3y agoI was positively stunned to learn such a thing, so while I don't have iOS I was able to find some references to that behavior in discussions.apple.com which led me to the fact that Safari on macOS can do it, too But Apple gonna apple about using the plist hammer for every file format nail in their world, so no: definitely not the same thing we are discussing here $ xxd -l 512 'You can also generate warc natively in iOS! On a website, tap the share sheet ic... | Hacker News.webarchive' 00000000: 6270 6c69 7374 3030 d201 0203 205f 100f bplist00.... _.. 00000010: 5765 6253 7562 7265 736f 7572 6365 735f WebSubresources_ 00000020: 100f 5765 624d 6169 6e52 6573 6f75 7263 ..WebMainResourc 00000030: 65a5 040d 1216 1bd4 0506 0708 090a 0b0c e............... 00000040: 5f10 0f57 6562 5265 736f 7572 6365 4461 _..WebResourceDa 00000050: 7461 5f10 1357 6562 5265 736f 7572 6365 ta_..WebResource 00000060: 4d49 4d45 5479 7065 5f10 1357 6562 5265 MIMEType_..WebRe 00000070: 736f 7572 6365 5265 7370 6f6e 7365 5e57 sourceResponse^W 00000080: 6562 5265 736f 7572 6365 5552 4c4f 111c ebResourceURLO.. 00000090: dd62 6f64 7920 207b 2066 6f6e 742d 6661 .body { font-fa 000000a0: 6d69 6c79 3a56 6572 6461 6e61 2c20 4765 mily:Verdana, Ge 000000b0: 6e65 7661 2c20 7361 6e73 2d73 6572 6966 neva, sans-serif $ plutil -p - < 'You can also generate warc natively in iOS! On a website, tap the share sheet ic... | Hacker News.webarchive' { "WebMainResource" => { "WebResourceData" => {length = 5725, bytes = 0x3c68746d 6c206c61 6e673d22 656e2220 ... 3e3c2f68 746d6c3e } "WebResourceFrameName" => "" "WebResourceMIMEType" => "text/html" "WebResourceTextEncodingName" => "UTF-8" "WebResourceURL" => "https://news.ycombinator.com/item?id=39185542" } "WebSubresources" => [ 0 => { "WebResourceData" => {length = 7389, bytes = 0x626f6479 20207b20 666f6e74 2d66616d ... 64656e20 7d0a7d0a } "WebResourceMIMEType" => "text/css" "WebResourceResponse" => {length = 1880, bytes = 0x62706c69 73743030 d4010203 04050607 ... 00000000 0000065c } "WebResourceURL" => "https://news.ycombinator.com/news.css?15bzD7gVH5AOktnbu3y0" } 1 => { "WebResourceData" => {length = 131, bytes = 0x3c737667 20686569 6768743d 22333222 ... 2f3e3c2f 7376673e } "WebResourceMIMEType" => "image/svg+xml" "WebResourceResponse" => {length = 1842, bytes = 0x62706c69 73743030 d4010203 04050607 ... 00000000 00000636 } "WebResourceURL" => "https://news.ycombinator.com/triangle.svg" } 2 => { "WebResourceData" => {length = 315, bytes = 0x3c737667 20686569 6768743d 22313822 ... 2f3e3c2f 7376673e } "WebResourceMIMEType" => "image/svg+xml" "WebResourceResponse" => {length = 1842, bytes = 0x62706c69 73743030 d4010203 04050607 ... 00000000 00000636 } "WebResourceURL" => "https://news.ycombinator.com/y18.svg" } 3 => { "WebResourceData" => {length = 43, bytes = 0x47494638 39610100 010080ff 00c0c0c0 ... 00000202 4401003b } "WebResourceMIMEType" => "image/gif" "WebResourceResponse" => {length = 1826, bytes = 0x62706c69 73743030 d4010203 04050607 ... 00000000 00000626 } "WebResourceURL" => "https://news.ycombinator.com/s.gif" } 4 => { "WebResourceData" => {length = 5224, bytes = 0x66756e63 74696f6e 20242028 69642920 ... 636c6963 6b293b0a } "WebResourceMIMEType" => "application/javascript" "WebResourceResponse" => {length = 1945, bytes = 0x62706c69 73743030 d4010203 04050607 ... 00000000 0000069b } "WebResourceURL" => "https://news.ycombinator.com/hn.js?15bzD7gVH5AOktnbu3y0" } ] }