4 ms·

I used to solely depend on Wayback machine to automate archiving pages. Now, I am archiving webpages using selenium python package on https://archive.ph/ https:

I used to solely depend on Wayback machine to automate archiving pages. Now, I am archiving webpages using selenium python package on https://archive.ph/ https://archive.ph/ and https://ghostarchive.org/ https://ghostarchive.org/. This told me not to depend on 3rd party services. Might self-host https://archivebox.io/ https://archivebox.io/.

keyle 2y ago
I was just fantasising earlier, daydreaming, about what a distributed warc or similar solution would look like, with peering and user or distributed server archiving. Either by browser plugin submission or passively sending the urls to servers to do the fetching and archiving (removes some of the privacy issues). I think it's everyone's responsibility to make sure the web gets cached, not one org... and since Google has canned the Google cache.......
- aspenmayer 2y ago
  Here's some prior art: https://wiki.archiveteam.org/index.php/Main_Page https://wiki.archiveteam.org/index.php/Main_Page
- nikisweeting 2y ago
  ArchiveBox v0.8 is adding the beginnings of a content addressable store for P2P sharing! Stay tuned :)
Cthulhu_ 2y ago
There should be more internet archives, for various reasons, but it doesn't seem like anyone is willing to put in the effort and money involved, let alone the legal headaches.
- jfil 2y ago
  I agree. And I am dismayed that government and academic institutions like to dance around the legal issues of archiving (outsourcing the legal risk to Internet Archive), instead of pushing for legal protections/exemptions for the act of archiving.
  - razakel 2y ago
    The UK and Portugal are both doing it for domestically published websites.
yamrzou 2y ago
Would you mind sharing your script?
- yasser_kaddoura 2y ago
  https://www.reddit.com/r/qutebrowser/comments/1g1zbel/userscript_archive_web_pages_to_waybackmachine/ https://www.reddit.com/r/qutebrowser/comments/1g1zbel/usersc...
  - yamrzou 2y ago
    Thank you. It says: "Sorry, this post was removed by Reddit’s filters."
    - yasser_kaddoura 2y ago
      That's very odd. I can see it on my end while logged in. I tested it without being logged in, and I am seeing it being removed. Gotta be a bug. You can find it here: https://gist.github.com/YasserKa/9a02bc50e75e7239f6f0c8f04fe4cfb1 https://gist.github.com/YasserKa/9a02bc50e75e7239f6f0c8f04fe...

keyle 2y ago

I was just fantasising earlier, daydreaming, about what a distributed warc or similar solution would look like, with peering and user or distributed server archiving. Either by browser plugin submission or passively sending the urls to servers to do the fetching and archiving (removes some of the privacy issues). I think it's everyone's responsibility to make sure the web gets cached, not one org... and since Google has canned the Google cache.......

aspenmayer 2y ago
Here's some prior art: https://wiki.archiveteam.org/index.php/Main_Page https://wiki.archiveteam.org/index.php/Main_Page
nikisweeting 2y ago
ArchiveBox v0.8 is adding the beginnings of a content addressable store for P2P sharing! Stay tuned :)

Cthulhu_ 2y ago

There should be more internet archives, for various reasons, but it doesn't seem like anyone is willing to put in the effort and money involved, let alone the legal headaches.

jfil 2y ago
I agree. And I am dismayed that government and academic institutions like to dance around the legal issues of archiving (outsourcing the legal risk to Internet Archive), instead of pushing for legal protections/exemptions for the act of archiving.
- razakel 2y ago
  The UK and Portugal are both doing it for domestically published websites.

yamrzou 2y ago

Would you mind sharing your script?

yasser_kaddoura 2y ago
https://www.reddit.com/r/qutebrowser/comments/1g1zbel/userscript_archive_web_pages_to_waybackmachine/ https://www.reddit.com/r/qutebrowser/comments/1g1zbel/usersc...
- yamrzou 2y ago
  Thank you. It says: "Sorry, this post was removed by Reddit’s filters."
  - yasser_kaddoura 2y ago
    That's very odd. I can see it on my end while logged in. I tested it without being logged in, and I am seeing it being removed. Gotta be a bug. You can find it here: https://gist.github.com/YasserKa/9a02bc50e75e7239f6f0c8f04fe4cfb1 https://gist.github.com/YasserKa/9a02bc50e75e7239f6f0c8f04fe...