3 ms·

Archive.org is the archiver, rotted links are replaced by Archive.org links with a bot. https://meta.wikimedia.org/wiki/InternetArchiveBot https://meta.wikimed

Archive.org is the archiver, rotted links are replaced by Archive.org links with a bot. https://meta.wikimedia.org/wiki/InternetArchiveBot https://meta.wikimedia.org/wiki/InternetArchiveBot https://github.com/internetarchive/internetarchivebot https://github.com/internetarchive/internetarchivebot

jsheard 9mo ago
Yeah for historical links it makes sense to fall back on IAs existing archives, but going forward Wikipedia could take their own snapshots of cited pages and substitute them in if/when the original rots. It would be more reliable than hoping IA grabbed it.
- toomuchtodo 9mo ago
  Not opposed, Wikimedia tech folks are very accessible in my experience, ask them to make a GET or POST to https://web.archive.org/save https://web.archive.org/save whenever a link is added via the Wiki editing mechanism. Easy peasy. Example CLI tools are https://github.com/palewire/savepagenow https://github.com/palewire/savepagenow and https://github.com/akamhy/waybackpy https://github.com/akamhy/waybackpy Shortcut is to consume the Wikimedia changelog firehose and make these http requests yourself, performing a CDX lookup request to see if a recent snapshot was already taken before issuing a capture request (to be polite to the capture worker queue).
  - jsheard 9mo ago
    I didn't know you can just ask IA to grab a page before their crawler gets to it. In that case yeah it would make sense for Wikipedia to ping them automatically.
    - extraduder_ire 9mo ago
      There's a /save/<url> endpoint that archives the page you point it at. You can see a text box for it on the right, if you go on the waybackmachine's homepage. I used it yesterday.
  - RupertSalt 9mo ago
    Spammers and pirates just got super excited at that plan!
    - toomuchtodo 9mo ago
      There are various systems in place to defend against them, I recommend against this, poor form against a public good is not welcome.
  - ferngodfather 9mo ago
    Why wouldn't Wikipedia just capture and host this themselves? Surely it makes more sense to DIY than to rely on a third party.
    - deleted 9mo ago
      [deleted]
    - huslage 9mo ago
      Why would they need to own the archive at all? The archive.org infrastructure is built to do this work already. It's outside of WMF's remit to internally archive all of the data it has links to.
  - Gander5739 9mo ago
    This already happens. Every link added to Wikipedia is automatically archived on the wayback machine.
    - deleted 9mo ago
      [deleted]
    - RupertSalt 9mo ago
      [citation needed]
      - Gander5739 9mo ago
        Ironic, I know. I couldn't find where I originally heard this years ago, but the InternetArchiveBot page linked above says "InternetArchiveBot monitors every Wikimedia wiki for new outgoing links" which is probably referring to what I said.
snigsnog 9mo ago
Archive.org are left wing activists that will agree to censor anything other left wing activists or large companies don't want online.
- Maken 9mo ago
  Like what?
  - snigsnog 9mo ago
    Kiwifarms is an example: https://old.reddit.com/r/DataHoarder/comments/x95gd5/internet_archive_breaks_from_previous_policies_on/ https://old.reddit.com/r/DataHoarder/comments/x95gd5/interne... Anyone can request anything be removed and they may honor the request: https://help.archive.org/help/how-do-i-request-to-remove-something-from-archive-org/ https://help.archive.org/help/how-do-i-request-to-remove-som... they say nothing about only removing things illegal in the US or anything like that, meaning they can and will remove things based on personal judgements about whether it should be archived.
    - Maken 9mo ago
      Unlisting (not even removing) doxing information about living people is being a left wing activist? Is there where the Oberton window lies now?
- AlexeyBelov 9mo ago
  And you're another disruptive "N days old" account. Troll somewhere else.

jsheard 9mo ago

Yeah for historical links it makes sense to fall back on IAs existing archives, but going forward Wikipedia could take their own snapshots of cited pages and substitute them in if/when the original rots. It would be more reliable than hoping IA grabbed it.

toomuchtodo 9mo ago
Not opposed, Wikimedia tech folks are very accessible in my experience, ask them to make a GET or POST to https://web.archive.org/save https://web.archive.org/save whenever a link is added via the Wiki editing mechanism. Easy peasy. Example CLI tools are https://github.com/palewire/savepagenow https://github.com/palewire/savepagenow and https://github.com/akamhy/waybackpy https://github.com/akamhy/waybackpy Shortcut is to consume the Wikimedia changelog firehose and make these http requests yourself, performing a CDX lookup request to see if a recent snapshot was already taken before issuing a capture request (to be polite to the capture worker queue).
- jsheard 9mo ago
  I didn't know you can just ask IA to grab a page before their crawler gets to it. In that case yeah it would make sense for Wikipedia to ping them automatically.
  - extraduder_ire 9mo ago
    There's a /save/<url> endpoint that archives the page you point it at. You can see a text box for it on the right, if you go on the waybackmachine's homepage. I used it yesterday.
- RupertSalt 9mo ago
  Spammers and pirates just got super excited at that plan!
  - toomuchtodo 9mo ago
    There are various systems in place to defend against them, I recommend against this, poor form against a public good is not welcome.
- ferngodfather 9mo ago
  Why wouldn't Wikipedia just capture and host this themselves? Surely it makes more sense to DIY than to rely on a third party.
  - deleted 9mo ago
    [deleted]
  - huslage 9mo ago
    Why would they need to own the archive at all? The archive.org infrastructure is built to do this work already. It's outside of WMF's remit to internally archive all of the data it has links to.
- Gander5739 9mo ago
  This already happens. Every link added to Wikipedia is automatically archived on the wayback machine.
  - deleted 9mo ago
    [deleted]
  - RupertSalt 9mo ago
    [citation needed]
    - Gander5739 9mo ago
      Ironic, I know. I couldn't find where I originally heard this years ago, but the InternetArchiveBot page linked above says "InternetArchiveBot monitors every Wikimedia wiki for new outgoing links" which is probably referring to what I said.

snigsnog 9mo ago

Archive.org are left wing activists that will agree to censor anything other left wing activists or large companies don't want online.

Maken 9mo ago
Like what?
- snigsnog 9mo ago
  Kiwifarms is an example: https://old.reddit.com/r/DataHoarder/comments/x95gd5/internet_archive_breaks_from_previous_policies_on/ https://old.reddit.com/r/DataHoarder/comments/x95gd5/interne... Anyone can request anything be removed and they may honor the request: https://help.archive.org/help/how-do-i-request-to-remove-something-from-archive-org/ https://help.archive.org/help/how-do-i-request-to-remove-som... they say nothing about only removing things illegal in the US or anything like that, meaning they can and will remove things based on personal judgements about whether it should be archived.
  - Maken 9mo ago
    Unlisting (not even removing) doxing information about living people is being a left wing activist? Is there where the Oberton window lies now?
AlexeyBelov 9mo ago
And you're another disruptive "N days old" account. Troll somewhere else.