3 ms·

Save Page Now seems huge. It's a real bummer going to old forums for an obscure hobby or fandom only to find all the text and none of the images, music, etc.

deleted 7y ago
[deleted]
deleted 7y ago
[deleted]
btrettel 7y ago
I've had the exact same problem before. I used to run an old forum that's now just an archive and I've been really meaning to download all the images linked in the posts in case they go offline. At some point during the forum's run I added a file upload feature which seems to have helped a lot (by avoiding external dependencies), but did not solve the problem. Fortunately I believe I have many of the missing images saved, but there very likely are important things missing. I'm planning to launch a new forum next year and I think I'm going to write a script to periodically archive all images and links posted to the forum. I might make external images not allowed, though that seems rather extreme and might just make people post a link rather than use the upload feature.
- toomuchtodo 7y ago
  Run grab-site [1] periodically (with the --no-offsite-links flag) and upload the resulting WARC files into an item in the Internet Archive. They can then be ingested by Wayback. If you prefer, I can do this for you as part of my existing archival operations. [1] https://github.com/ArchiveTeam/grab-site https://github.com/ArchiveTeam/grab-site
  - btrettel 7y ago
    Thanks, I wasn't aware of this bot. I'll keep your offer in mind.
- duskwuff 7y ago
  Tangent: There's an incredible amount of information stored in obscure web forums, often in posts with photos. The damage that services like Photobucket have done by deleting old files, or by restricting hotlinking, has been incalculable. I worry that Imgur has the potential to do even worse damage, as so many forum users have converged on their service after others became unavailable. (Imgur's popularity with Reddit users leaves Reddit highly vulnerable as well.)
  - jborichevskiy 7y ago
    Crazy idea: an browser extension a user can install which downloads images as they come across them in their browser and uploads them to something distributed - perhaps built on top of IPFS? Users could choose which domains it would be active on. The network could be split up by either domain or topic (say, people interested in diagrams of space which might include several domains/sites). Just thinking out loud here.
    - duskwuff 7y ago
      The problems with putting that kind of data in any sort of distributed service are that: 1) It depends upon enough users being able to consistently contribute a lot of storage to the system. It turns out that this is hard. Casual users are actually a hindrance, because they'll suck up a bunch of bandwidth trying to replicate data, then drop out of the swarm forever. 2) The service will inevitably be used to host illegal pornographic content. Without some sort of centralized control, there's no way to stop this, making participation legally problematic.
dredmorbius 7y ago
SPN can be run from a script using wget, curl, or any other HTTP GET request generator. If you can come up with a set of URLs for your content, you can archive it by prefixing it with "https://web.archive.org/save" https://web.archive.org/save" So if you have: https://www.example.com/page-1 https://www.example.com/page-2 https://www.example.com/page-3 You'd generate (and request): https://web.archive.com/save/https://www.example.com/page-1 https://web.archive.com/save/https://www.example.com/page-2 https://web.archive.com/save/https://www.example.com/page-3 This is trivially scripted, or there are a few existing generators. I've created archives of ~12,000 or so posts, from an old desktop Linux system over modest residential broadband, in less than an hour, running up to 20 parallel requests via xargs or GNU parallel. For a basic curl-based URL archiver (call once per URL on your list): curl -s -I -H "Accept: application/json" "https://web.archive.org/save/${1}" | grep '^x-cache-key:' | sed "s,https,&://,; s,$${1}$.*$,\1,"
- toomuchtodo 7y ago
  https://github.com/pastpages/savepagenow https://github.com/pastpages/savepagenow is useful for this as a packaged utility.

deleted 7y ago

[deleted]

deleted 7y ago

[deleted]

btrettel 7y ago

I've had the exact same problem before. I used to run an old forum that's now just an archive and I've been really meaning to download all the images linked in the posts in case they go offline. At some point during the forum's run I added a file upload feature which seems to have helped a lot (by avoiding external dependencies), but did not solve the problem. Fortunately I believe I have many of the missing images saved, but there very likely are important things missing. I'm planning to launch a new forum next year and I think I'm going to write a script to periodically archive all images and links posted to the forum. I might make external images not allowed, though that seems rather extreme and might just make people post a link rather than use the upload feature.

toomuchtodo 7y ago
Run grab-site [1] periodically (with the --no-offsite-links flag) and upload the resulting WARC files into an item in the Internet Archive. They can then be ingested by Wayback. If you prefer, I can do this for you as part of my existing archival operations. [1] https://github.com/ArchiveTeam/grab-site https://github.com/ArchiveTeam/grab-site
- btrettel 7y ago
  Thanks, I wasn't aware of this bot. I'll keep your offer in mind.
duskwuff 7y ago
Tangent: There's an incredible amount of information stored in obscure web forums, often in posts with photos. The damage that services like Photobucket have done by deleting old files, or by restricting hotlinking, has been incalculable. I worry that Imgur has the potential to do even worse damage, as so many forum users have converged on their service after others became unavailable. (Imgur's popularity with Reddit users leaves Reddit highly vulnerable as well.)
- jborichevskiy 7y ago
  Crazy idea: an browser extension a user can install which downloads images as they come across them in their browser and uploads them to something distributed - perhaps built on top of IPFS? Users could choose which domains it would be active on. The network could be split up by either domain or topic (say, people interested in diagrams of space which might include several domains/sites). Just thinking out loud here.
  - duskwuff 7y ago
    The problems with putting that kind of data in any sort of distributed service are that: 1) It depends upon enough users being able to consistently contribute a lot of storage to the system. It turns out that this is hard. Casual users are actually a hindrance, because they'll suck up a bunch of bandwidth trying to replicate data, then drop out of the swarm forever. 2) The service will inevitably be used to host illegal pornographic content. Without some sort of centralized control, there's no way to stop this, making participation legally problematic.

dredmorbius 7y ago

SPN can be run from a script using wget, curl, or any other HTTP GET request generator. If you can come up with a set of URLs for your content, you can archive it by prefixing it with "https://web.archive.org/save" https://web.archive.org/save" So if you have: https://www.example.com/page-1 https://www.example.com/page-2 https://www.example.com/page-3 You'd generate (and request): https://web.archive.com/save/https://www.example.com/page-1 https://web.archive.com/save/https://www.example.com/page-2 https://web.archive.com/save/https://www.example.com/page-3 This is trivially scripted, or there are a few existing generators. I've created archives of ~12,000 or so posts, from an old desktop Linux system over modest residential broadband, in less than an hour, running up to 20 parallel requests via xargs or GNU parallel. For a basic curl-based URL archiver (call once per URL on your list): curl -s -I -H "Accept: application/json" "https://web.archive.org/save/${1}" | grep '^x-cache-key:' | sed "s,https,&://,; s,$${1}$.*$,\1,"

toomuchtodo 7y ago
https://github.com/pastpages/savepagenow https://github.com/pastpages/savepagenow is useful for this as a packaged utility.