12 ms·

I worked with y'all as a volunteer back when trump was pushing to drop global warming sites from gov websites. I also use the wayback machine professionally on

by chaps 5y ago

I worked with y'all as a volunteer back when trump was pushing to drop global warming sites from gov websites. I also use the wayback machine professionally on a regular basis. The work y'all do is genuinely appreciated to say the least. That said.. damn I really wish y'all would revisit some of your fundamentals like recursive scraping and making sure your scraping is whole and complete before working on filecoin and other needlessly flashy systems. I'm genuinely worried y'all are digging yourselves into a technical pit that you can't get out of and it will hurt or even kill your goals. Cheers.

gjs278 5y ago
bcherny 5y ago
What’s recursive scraping (presumably, following internal links), and what’s broken about it?
- chaps 5y ago
  Yep, that's exactly it. Recursive scraping, in theory, would remove a lot gaps on IA. It's really hard to do generically and "right" though. Lots of one-off.. strangeness.. especially from older sites and sites that auto-generate URLs at render time.
smarx007 5y ago
Recursive scraping: 1) can consume a lot of storage really fast 2) makes IA a bot rather than a user service 3) is already done on a case by case basis by ArchiveTeam allowing IA to stay away from the previous two problems.
- eslaught 5y ago
  There's an old school website I wanted to access that's long gone. It's only partially preserved in the Wayback Machine. I'm glad it's there at all, but some of the specific information I was looking for was never captured, and now there's no way to recover it. I realize it's a hard problem, but I really wish there were a way to automate more of it. Some of these communities are too small for anyone to bother preserving the pages manually, and I don't imagine we'd even show up on ArchiveTeam's radar. But they're not large pages, and basically static. I don't think they'd be a huge burden to store and maintain. It seems like some sort of a coverage + size metric would be pretty effective at guiding an automated scan such that you'd be able to preserve things liked this without needing humans to go and manually archive each and every page.
  - chaps 5y ago
    Pretty much. Your idea of getting coverage could probably work well to make sure IA doesn't "turn into a bot" (though it's practically the same, just without the permanent downloading).
  - ivank 5y ago
    https://wiki.archiveteam.org/index.php/ArchiveBot https://wiki.archiveteam.org/index.php/ArchiveBot is available if you would like to help save things that are still up.
  - smarx007 5y ago
    I never got turned down sending an IRC message "x is going down in a few days" to the AT channels. https://wiki.archiveteam.org/index.php/Archiveteam:IRC https://wiki.archiveteam.org/index.php/Archiveteam:IRC AT just asks to be given a few weeks/months of notice depending on how many TBs (MBs?) the crawl needs to be and not to throttle/ban their clients.

by chaps 5y ago

gjs278 5y ago

bcherny 5y ago

What’s recursive scraping (presumably, following internal links), and what’s broken about it?

chaps 5y ago
Yep, that's exactly it. Recursive scraping, in theory, would remove a lot gaps on IA. It's really hard to do generically and "right" though. Lots of one-off.. strangeness.. especially from older sites and sites that auto-generate URLs at render time.

smarx007 5y ago

Recursive scraping: 1) can consume a lot of storage really fast 2) makes IA a bot rather than a user service 3) is already done on a case by case basis by ArchiveTeam allowing IA to stay away from the previous two problems.

eslaught 5y ago
There's an old school website I wanted to access that's long gone. It's only partially preserved in the Wayback Machine. I'm glad it's there at all, but some of the specific information I was looking for was never captured, and now there's no way to recover it. I realize it's a hard problem, but I really wish there were a way to automate more of it. Some of these communities are too small for anyone to bother preserving the pages manually, and I don't imagine we'd even show up on ArchiveTeam's radar. But they're not large pages, and basically static. I don't think they'd be a huge burden to store and maintain. It seems like some sort of a coverage + size metric would be pretty effective at guiding an automated scan such that you'd be able to preserve things liked this without needing humans to go and manually archive each and every page.
- chaps 5y ago
  Pretty much. Your idea of getting coverage could probably work well to make sure IA doesn't "turn into a bot" (though it's practically the same, just without the permanent downloading).
- ivank 5y ago
  https://wiki.archiveteam.org/index.php/ArchiveBot https://wiki.archiveteam.org/index.php/ArchiveBot is available if you would like to help save things that are still up.
- smarx007 5y ago
  I never got turned down sending an IRC message "x is going down in a few days" to the AT channels. https://wiki.archiveteam.org/index.php/Archiveteam:IRC https://wiki.archiveteam.org/index.php/Archiveteam:IRC AT just asks to be given a few weeks/months of notice depending on how many TBs (MBs?) the crawl needs to be and not to throttle/ban their clients.