3 ms·

Anybody know of something like this that I can use for personal archiving?

by bane 8y ago

xj9 8y ago
https://www.archiveteam.org/index.php?title=Wget_with_WARC_output https://www.archiveteam.org/index.php?title=Wget_with_WARC_o... https://github.com/oduwsdl/ipwb https://github.com/oduwsdl/ipwb #!/bin/sh cd ${HOME}/data/archive user_agent="Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27" domain="${1}" wget \ --mirror \ --page-requisites \ --restrict-file-names=nocontrol \ --timeout 60 \ --tries 5 \ --wait 1 \ --waitretry 5 \ --warc-cdx \ --warc-file="${domain}" \ --warc-header "operator: Wayback Mesh" \ -U "${user_agent}" \ -e robots=off \ "http://${domain}" ipwb index "${domain}.warc" > "${domain}.ipfs.cdxj"
PeterMikhailov 8y ago
https://wallabag.org/en https://wallabag.org/en If you run your own, make sure you turn on "Download images"
- zaarn 8y ago
  Wallabad isn't quite suitable for Archival operation last I checked since it doesn't present or save the original webpage. That's one of the major points why archive.is/wayback is popular; you get the exact same page as you get in the browser. Perfectly for archival.
fake-name 8y ago
https://github.com/fake-name/ReadableWebProxy https://github.com/fake-name/ReadableWebProxy Disclaimer - It's a project of mine, specifically written for my interests, and is very much not production ready. It does have a full blown spider in it, which supports distributed fetching and stuff via another project [1]. It does both rewritten archiving (basically overwriting the page style) and raw archiving. I also have a bunch of miscellaneous stuff like a custom python interface for chromium so I can handle jerberscript shitshows (cloudflare "protection" [2], reCaptcha [3], etc...). People seem really dead-set on ruining the internet. I keep meaning to implement WARC proxying to feed into the IA as a parallel output stream for the spider, but it's a lot of work and I have too many projects already. 1: https://github.com/fake-name/AutoTriever https://github.com/fake-name/AutoTriever 2: https://github.com/fake-name/ChromeController https://github.com/fake-name/ChromeController 3: https://github.com/fake-name/WebRequest https://github.com/fake-name/WebRequest
nikisweeting 8y ago
The self-hosted way-back machine! Bookmark Archiver: https://github.com/pirate/bookmark-archiver https://github.com/pirate/bookmark-archiver It's modeled on the internet archive except it uses Chrome Headless in addition to wget so you get snapshots of the page after JS executes too.
telotortium 8y ago
https://www.gwern.net/Archiving-URLs https://www.gwern.net/Archiving-URLs

Anybody know of something like this that I can use for personal archiving?

by bane 8y ago

Anybody know of something like this that I can use for personal archiving?

xj9 8y ago

https://www.archiveteam.org/index.php?title=Wget_with_WARC_output https://www.archiveteam.org/index.php?title=Wget_with_WARC_o... https://github.com/oduwsdl/ipwb https://github.com/oduwsdl/ipwb #!/bin/sh cd ${HOME}/data/archive user_agent="Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27" domain="${1}" wget \ --mirror \ --page-requisites \ --restrict-file-names=nocontrol \ --timeout 60 \ --tries 5 \ --wait 1 \ --waitretry 5 \ --warc-cdx \ --warc-file="${domain}" \ --warc-header "operator: Wayback Mesh" \ -U "${user_agent}" \ -e robots=off \ "http://${domain}" ipwb index "${domain}.warc" > "${domain}.ipfs.cdxj"

PeterMikhailov 8y ago

https://wallabag.org/en https://wallabag.org/en If you run your own, make sure you turn on "Download images"

zaarn 8y ago
Wallabad isn't quite suitable for Archival operation last I checked since it doesn't present or save the original webpage. That's one of the major points why archive.is/wayback is popular; you get the exact same page as you get in the browser. Perfectly for archival.

fake-name 8y ago

https://github.com/fake-name/ReadableWebProxy https://github.com/fake-name/ReadableWebProxy Disclaimer - It's a project of mine, specifically written for my interests, and is very much not production ready. It does have a full blown spider in it, which supports distributed fetching and stuff via another project [1]. It does both rewritten archiving (basically overwriting the page style) and raw archiving. I also have a bunch of miscellaneous stuff like a custom python interface for chromium so I can handle jerberscript shitshows (cloudflare "protection" [2], reCaptcha [3], etc...). People seem really dead-set on ruining the internet. I keep meaning to implement WARC proxying to feed into the IA as a parallel output stream for the spider, but it's a lot of work and I have too many projects already. 1: https://github.com/fake-name/AutoTriever https://github.com/fake-name/AutoTriever 2: https://github.com/fake-name/ChromeController https://github.com/fake-name/ChromeController 3: https://github.com/fake-name/WebRequest https://github.com/fake-name/WebRequest

nikisweeting 8y ago

The self-hosted way-back machine! Bookmark Archiver: https://github.com/pirate/bookmark-archiver https://github.com/pirate/bookmark-archiver It's modeled on the internet archive except it uses Chrome Headless in addition to wget so you get snapshots of the page after JS executes too.

telotortium 8y ago

https://www.gwern.net/Archiving-URLs https://www.gwern.net/Archiving-URLs