6 ms·

We should be mirroring fucking everything. Where's a good place to put data like this? Is there even a place where people can start to inventory what's at risk

We should be mirroring fucking everything. Where's a good place to put data like this? Is there even a place where people can start to inventory what's at risk? The biggest problem is not knowing what's out there, what records need to be protected, as there's so many people doing important work.

kalleboo 10y ago
The internet archive has made this their mission. Archive Team keeps a constant eye out for major content sites that are at risk of being shut down and proactively mirroring their content. http://www.archiveteam.org/index.php?title=Main_Page http://www.archiveteam.org/index.php?title=Main_Page On EPA data https://twitter.com/textfiles/status/824110893034311680 https://twitter.com/textfiles/status/824110893034311680 They don't just archive online content either. They spend many resources scanning magazines, ingesting (and cataloging) CDs and floppies before they all rot away.
- astrodust 10y ago
  As much as it's probably a logistical nightmare, they basically need to back up everything now. This is triage time because everything is at risk of being shut down. You know when the Internet Archive itself is making a contingency plan to move out of the US that things are truly fucked. The most difficult thing here is any prirotization they end up doing must be kept absolutely secret or you're going to be providing a hit-list to the administration should they ever want one.
  - barney54 10y ago
    Where is the evidence that any of the data sites are at risk of being shutdown?
    - lazerwalker 10y ago
      I don't think there's direct concrete evidence, but that the Internet Archive themselves just ran a crowdfunding campaign to open a redundant data center in Canada suggests it is something they are worried about.
      - ianai 10y ago
        Whole departments of the US government being told to hush up, including in peer reviewed journals, is direct concrete evidence.
- closeparen 10y ago
  Won't they delete it all when epa.gov puts up a robots.txt?
  - af16090 10y ago
    My understanding is that they will disallow access to a webpage if a robots.txt goes up but they will keep the copy of the webpage in case the robots.txt is ever changed[1]. So a future administration could change the robots.txt and the webpages would be accessible again. [1]: Seems like that happened here: https://en.wikipedia.org/wiki/Wayback_Machine#Netbula_LLC_v._Chordiant_Software_Inc https://en.wikipedia.org/wiki/Wayback_Machine#Netbula_LLC_v.....
    - hunter2_ 10y ago
      I know robots.txt is all about respect in the first place, but is it really expected that obtained-via-robot mirrors will be made inaccessible to humans during periods of time when the upstream prohibits robotic fetching following that original fetch? It seems much more reasonable that the mirror would merely prohibit robots, not humans, from fetching its copy during periods of time when the upstream prohibits robots.
      - CM30 10y ago
        I suspect the original thought was that a website owner might later decide they don't want their site publically viewable in the Internet Archive, and would sue/complain if it was still visible there. Like say, if they accidentally allowed robots to archive a 'hidden' directory. Problem is, those use cases are greatly outweighed by ones where either: 1. The site has changed ownership to a different (legitimate) company or organisation, and they don't want their new site archived. 2. A domain squatter/seller has bought the domain, blocked all bots to stop the holding page being indexed in Google and accidentally blocked the archive in the process. 3. A technician or developer has accidentally blocked the archive/all robots due to a personal mistake/copying code from the internet. 4. The site owner doesn't know the archive has a bot, and has blocked all non Google/Bing bots to 'reduce strain' on the server. That last one is depressingly common: http://webmasters.stackexchange.com/questions/75993/only-allow-google-and-bing-bots-to-crawl-a-site http://webmasters.stackexchange.com/questions/75993/only-all...
toomanybeersies 10y ago
Bittorrent is a good place to put things, obviously you still need seeds, but it ensures anyone can access it. Otherwise, I guess AWS Glacier if you want to pay for it.
- astrodust 10y ago
  Torrents are a good stepping stone to a more permanent archive but they fundamentally depend on seeders. Likewise Glacier depends on Amazon not being raided by the Feds. I know that sounds totally insane, but we're living in totally insane times. It needs to be on hardware an organization like the Internet Archive controls or at least ordains. We're talking bunkers in Sweden or Iceland or some country with the political backbone to defend this sort of scientific data.
- steveax 10y ago
  usenet.
sandworm101 10y ago
Canada. We went through much of the same gag-ordering climate-denying idiocy under Steven Harper (a personal friend of trump by the way). We learned those lessons and wont be deleting anything anytime soon.
philipashlock 10y ago
Data.gov is also coordinating with Archive.org and similar efforts to have a more systematic comprehensive archiving approach. More info should be available soon at http://www.data.gov/developers/harvesting http://www.data.gov/developers/harvesting and https://archive.org/details/data_gov https://archive.org/details/data_gov
zachrose 10y ago
Seems like this is one of the many things that IPFS would be great for. In practice, I'm curious if the user base is wide enough to be as durable as a real archive. (Although I guess if a few people commit to pinning* the research it might be?) *Not sure if this is the right term.
- JumpCrisscross 10y ago
  Wouldn't IPFS just make finding mirrored content easier? It still needs to live somewhere.
- Dylan16807 10y ago
  A huge amount of rarely accessed data? IPFS isn't going to guarantee any copies actually exist in that case, and could provide a false sense of security.
  - rakoo 10y ago
    It doesn't make content magically more available, for sure, but it makes backuping a web site trivial, to the point where you can't have content disappear out of sheer will (the only way to have this is to make people uninterested in the content)

kalleboo 10y ago

The internet archive has made this their mission. Archive Team keeps a constant eye out for major content sites that are at risk of being shut down and proactively mirroring their content. http://www.archiveteam.org/index.php?title=Main_Page http://www.archiveteam.org/index.php?title=Main_Page On EPA data https://twitter.com/textfiles/status/824110893034311680 https://twitter.com/textfiles/status/824110893034311680 They don't just archive online content either. They spend many resources scanning magazines, ingesting (and cataloging) CDs and floppies before they all rot away.

astrodust 10y ago
As much as it's probably a logistical nightmare, they basically need to back up everything now. This is triage time because everything is at risk of being shut down. You know when the Internet Archive itself is making a contingency plan to move out of the US that things are truly fucked. The most difficult thing here is any prirotization they end up doing must be kept absolutely secret or you're going to be providing a hit-list to the administration should they ever want one.
- barney54 10y ago
  Where is the evidence that any of the data sites are at risk of being shutdown?
  - lazerwalker 10y ago
    I don't think there's direct concrete evidence, but that the Internet Archive themselves just ran a crowdfunding campaign to open a redundant data center in Canada suggests it is something they are worried about.
    - ianai 10y ago
      Whole departments of the US government being told to hush up, including in peer reviewed journals, is direct concrete evidence.
closeparen 10y ago
Won't they delete it all when epa.gov puts up a robots.txt?
- af16090 10y ago
  My understanding is that they will disallow access to a webpage if a robots.txt goes up but they will keep the copy of the webpage in case the robots.txt is ever changed[1]. So a future administration could change the robots.txt and the webpages would be accessible again. [1]: Seems like that happened here: https://en.wikipedia.org/wiki/Wayback_Machine#Netbula_LLC_v._Chordiant_Software_Inc https://en.wikipedia.org/wiki/Wayback_Machine#Netbula_LLC_v.....
  - hunter2_ 10y ago
    I know robots.txt is all about respect in the first place, but is it really expected that obtained-via-robot mirrors will be made inaccessible to humans during periods of time when the upstream prohibits robotic fetching following that original fetch? It seems much more reasonable that the mirror would merely prohibit robots, not humans, from fetching its copy during periods of time when the upstream prohibits robots.
    - CM30 10y ago
      I suspect the original thought was that a website owner might later decide they don't want their site publically viewable in the Internet Archive, and would sue/complain if it was still visible there. Like say, if they accidentally allowed robots to archive a 'hidden' directory. Problem is, those use cases are greatly outweighed by ones where either: 1. The site has changed ownership to a different (legitimate) company or organisation, and they don't want their new site archived. 2. A domain squatter/seller has bought the domain, blocked all bots to stop the holding page being indexed in Google and accidentally blocked the archive in the process. 3. A technician or developer has accidentally blocked the archive/all robots due to a personal mistake/copying code from the internet. 4. The site owner doesn't know the archive has a bot, and has blocked all non Google/Bing bots to 'reduce strain' on the server. That last one is depressingly common: http://webmasters.stackexchange.com/questions/75993/only-allow-google-and-bing-bots-to-crawl-a-site http://webmasters.stackexchange.com/questions/75993/only-all...

toomanybeersies 10y ago

Bittorrent is a good place to put things, obviously you still need seeds, but it ensures anyone can access it. Otherwise, I guess AWS Glacier if you want to pay for it.

astrodust 10y ago
Torrents are a good stepping stone to a more permanent archive but they fundamentally depend on seeders. Likewise Glacier depends on Amazon not being raided by the Feds. I know that sounds totally insane, but we're living in totally insane times. It needs to be on hardware an organization like the Internet Archive controls or at least ordains. We're talking bunkers in Sweden or Iceland or some country with the political backbone to defend this sort of scientific data.
steveax 10y ago
usenet.

sandworm101 10y ago

Canada. We went through much of the same gag-ordering climate-denying idiocy under Steven Harper (a personal friend of trump by the way). We learned those lessons and wont be deleting anything anytime soon.

philipashlock 10y ago

Data.gov is also coordinating with Archive.org and similar efforts to have a more systematic comprehensive archiving approach. More info should be available soon at http://www.data.gov/developers/harvesting http://www.data.gov/developers/harvesting and https://archive.org/details/data_gov https://archive.org/details/data_gov

zachrose 10y ago

Seems like this is one of the many things that IPFS would be great for. In practice, I'm curious if the user base is wide enough to be as durable as a real archive. (Although I guess if a few people commit to pinning* the research it might be?) *Not sure if this is the right term.

JumpCrisscross 10y ago
Wouldn't IPFS just make finding mirrored content easier? It still needs to live somewhere.
Dylan16807 10y ago
A huge amount of rarely accessed data? IPFS isn't going to guarantee any copies actually exist in that case, and could provide a false sense of security.
- rakoo 10y ago
  It doesn't make content magically more available, for sure, but it makes backuping a web site trivial, to the point where you can't have content disappear out of sheer will (the only way to have this is to make people uninterested in the content)