4 ms·

You might be interested in https://archive.org/ https://archive.org/.

by bcherny 10y ago

pmoriarty 10y ago
Does it actually archive all of HN, Reddit, and Stack Overflow (and related sites)? I've seen lots of omissions in archive.org. Not that it's not great. I love it. But there's a lot more that needs to be archived than what's in there.
- icebraining 10y ago
  The Archive Team[1] tries to scrape such records when the sites are in danger of closing down. Currently my Warrior[2] is pulling pictures from Panoramio[3]. Unfortunately, resources are often scarce, particularly storage. [1] http://www.archiveteam.org/index.php?title=Main_Page http://www.archiveteam.org/index.php?title=Main_Page [2] http://www.archiveteam.org/index.php?title=ArchiveTeam_Warrior http://www.archiveteam.org/index.php?title=ArchiveTeam_Warri... [3] https://news.ycombinator.com/item?id=12732692 https://news.ycombinator.com/item?id=12732692
  - r3bl 10y ago
    I believe they're currently preparing on working on Vine, and, from what they wrote on Twitter just today (can't search for it right now, don't have a stable connection right now), they don't have any problems with the storage at the moment thanks to Archive.org.
- nickpsecurity 10y ago
  I suppose we could email them with a request to make sure they scrape everything off of first, few pages of HN or key subreddits. I'm not sure about Stack Overflow where they or Google would know best based on hits certain questions get which are best to archive. Maybe just archive whole thing. Sites might even pay them to as a high-availability or backup option. Low odds there but asking doesn't hurt. ;)
  - duskwuff 10y ago
    The Internet Archive (and Stack Overflow) are way ahead of you here. SO is working directly with the Archive to publish anonymized database dumps. https://archive.org/details/stackexchange https://archive.org/details/stackexchange
    - nickpsecurity 10y ago
      That's awesome. Thanks for the link!

You might be interested in https://archive.org/ https://archive.org/.

by bcherny 10y ago

You might be interested in https://archive.org/ https://archive.org/.

Does it actually archive all of HN, Reddit, and Stack Overflow (and related sites)? I've seen lots of omissions in archive.org. Not that it's not great. I love it. But there's a lot more that needs to be archived than what's in there.

icebraining 10y ago
The Archive Team[1] tries to scrape such records when the sites are in danger of closing down. Currently my Warrior[2] is pulling pictures from Panoramio[3]. Unfortunately, resources are often scarce, particularly storage. [1] http://www.archiveteam.org/index.php?title=Main_Page http://www.archiveteam.org/index.php?title=Main_Page [2] http://www.archiveteam.org/index.php?title=ArchiveTeam_Warrior http://www.archiveteam.org/index.php?title=ArchiveTeam_Warri... [3] https://news.ycombinator.com/item?id=12732692 https://news.ycombinator.com/item?id=12732692
- r3bl 10y ago
  I believe they're currently preparing on working on Vine, and, from what they wrote on Twitter just today (can't search for it right now, don't have a stable connection right now), they don't have any problems with the storage at the moment thanks to Archive.org.
nickpsecurity 10y ago
I suppose we could email them with a request to make sure they scrape everything off of first, few pages of HN or key subreddits. I'm not sure about Stack Overflow where they or Google would know best based on hits certain questions get which are best to archive. Maybe just archive whole thing. Sites might even pay them to as a high-availability or backup option. Low odds there but asking doesn't hurt. ;)
- duskwuff 10y ago
  The Internet Archive (and Stack Overflow) are way ahead of you here. SO is working directly with the Archive to publish anonymized database dumps. https://archive.org/details/stackexchange https://archive.org/details/stackexchange
  - nickpsecurity 10y ago
    That's awesome. Thanks for the link!