4 ms·
That's a really great idea. Here's a one-line wget that will grab the provided URL and all of the data necessary to render, to one level of recursion, and dump
by zrail 11y ago
That's a really great idea. Here's a one-line wget that will grab the provided URL and all of the data necessary to render, to one level of recursion, and dump it to a WARC:
wget -e robots=off \
--user-agent="Mozilla" \
-r 1 -p -E -H -k -K \
--warc-file=/path/to/your/warc/file/without/warc/extension \
'http://www.example.com'
I think I might start capturing these. Shouldn't take up too much additional disk space.
edit: previously it was `-r 2` which is two levels of recursion.
- scrollaway 11y agoYou can shorten that to `-pEHkKr 1`, so you know.
- aw3c2 11y agoYou might have more fun with wpull, it has more sensible options to grab resources on referenced urls. https://github.com/chfoo/wpull https://github.com/chfoo/wpull