6 ms·

Public appeal to backup Climate Monitoring and Diagnostics Laboratory's FTP repo

85 points by bane 10y ago

tsomctl 10y ago
Also note https://github.com/climate-mirror/datasets/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc https://github.com/climate-mirror/datasets/issues?q=is%3Aiss... Seems like archive.org should be archiving this stuff, so it's all in one place and future researchers don't have to go hunting all over the internet for it.
- pimlottc 10y ago
  Jason Scott (twitter handle @textfiles) is already on it: https://twitter.com/textfiles/status/824104225688981504 https://twitter.com/textfiles/status/824104225688981504
  - bane 10y ago
    Jason Scott et al may find themselves in line for a medal of freedom at some point in the future.
barney54 10y ago
Why is there any threat of these databases going away?
- astrodust 10y ago
  If you're asking that question you haven't been paying attention. Science is the enemy. It will be destroyed if it continues to get in the way of the administration. It happened here (Canada) http://www.macleans.ca/news/canada/vanishing-canada-why-were-all-losers-in-ottawas-war-on-data/ http://www.macleans.ca/news/canada/vanishing-canada-why-were... and it could happen on an even bigger scale in America. The Conservative party went about cutting funding, slashing archives, burning everything to the ground, sometimes literally. They'd do it with little warning, zero fanfare, and an aggressive timeline. One day you had a climate archive, the next the shredding company had taken care of it. Those servers aren't free. They depend on budgets and grants to stay running. If that money is cut, those files are gone.
  - deleted 10y ago
    [deleted]
- rhizome 10y ago
  Because science is a source of truth.
- jakeogh 10y ago
  Won't happen.
- trhway 10y ago
  because they most probably contradict alternative facts.
astrodust 10y ago
We should be mirroring fucking everything. Where's a good place to put data like this? Is there even a place where people can start to inventory what's at risk? The biggest problem is not knowing what's out there, what records need to be protected, as there's so many people doing important work.
- kalleboo 10y ago
  The internet archive has made this their mission. Archive Team keeps a constant eye out for major content sites that are at risk of being shut down and proactively mirroring their content. http://www.archiveteam.org/index.php?title=Main_Page http://www.archiveteam.org/index.php?title=Main_Page On EPA data https://twitter.com/textfiles/status/824110893034311680 https://twitter.com/textfiles/status/824110893034311680 They don't just archive online content either. They spend many resources scanning magazines, ingesting (and cataloging) CDs and floppies before they all rot away.
  - astrodust 10y ago
    As much as it's probably a logistical nightmare, they basically need to back up everything now. This is triage time because everything is at risk of being shut down. You know when the Internet Archive itself is making a contingency plan to move out of the US that things are truly fucked. The most difficult thing here is any prirotization they end up doing must be kept absolutely secret or you're going to be providing a hit-list to the administration should they ever want one.
    - barney54 10y ago
      Where is the evidence that any of the data sites are at risk of being shutdown?
      - lazerwalker 10y ago
        I don't think there's direct concrete evidence, but that the Internet Archive themselves just ran a crowdfunding campaign to open a redundant data center in Canada suggests it is something they are worried about.
niftich 10y ago
Things to keep in mind: - In the US, the CFAA has language that may or may not be relevant [1]. This isn't intended to be FUD, although I can understand if you think so. As with most legal things, don't believe me -- a random guy -- on the internet; ask a lawyer, or at the very least be aware of your risks. - Check license and terms, if included. Reddit post claims these [2] are terms, seems okay, but I have not confirmed this for myself. - FTP is unencrypted and non-tamperproof, so data integrity and data authenticity, and connection privacy isn't guaranteed. And because the original source did not publish checksums over a secure channel, there is no way to know whether these files are the originals. It will be difficult to prove the provenance and accuracy of the data, despite having lots of third-party copies, without the original source having safeguarded the strong cryptographic checksums the entire time, or having published them over a tamperproof channel. [1] https://www.law.cornell.edu/uscode/text/18/1030 https://www.law.cornell.edu/uscode/text/18/1030 [2] https://www.reddit.com/r/DataHoarder/comments/5q4xxe/erik_fichtner_on_twitter_please_wget_m_np/dcwo823/ https://www.reddit.com/r/DataHoarder/comments/5q4xxe/erik_fi...
- jMyles 10y ago
  > strong cryptographic checksums They needn't even be strong, right? Even a weak checksum on a data set of this size is basically collision-proof, no?
  - niftich 10y ago
    It absolutely needs to be a strong cryptographic hash, because you want to protect against both deliberate tampering and accidental modification. Any other function, whether non-cryptographic, too short, broken or having known weaknesses, while perhaps suitable to detect accidental corruption, will not protect against the former. For example, SHA-384, or SHA-3 are great choices.
    - jMyles 10y ago
      Gotcha. TIL.
    - Dylan16807 10y ago
Proven 10y ago
Haha, screw that! Now that you have to rely on help from others suddenly there's "please". When the thugs from previous administration coerced and stole money from taxpayers to pay for this crap, "please" was nowhere to be seen. I may download this stuff several times to /dev/null from others who will be seeding it. Hope that helps!
losvedir 10y ago
Should I know who Erik Ficthner is? Is he an NOAA scientist? From what I heard on NPR this morning and a nytimes piece, the Trump administration doesn't seem to be doing anything unusual. I thought it was customary for the incoming administration to make sure the agency is all on the same page. I'm getting hysteria fatigue, I think.
- eropple 10y ago
  The anti-science posture of Trump specifically and Republicans generally is alarming by itself. The steady denials of observable fact by this administration and the immediate scrubbing of executive-controlled public relations pages of factual materials about topics that are very inconvenient for them makes worries of Orwell's memory hole very, very reasonable. This isn't hysteria. This is insurance. Don't be That Guy.
- JanneVee 10y ago
  I'll leave this here http://www.vanityfair.com/news/2017/01/why-does-peter-thiel-have-new-zealand-citizenship http://www.vanityfair.com/news/2017/01/why-does-peter-thiel-... It isn't alarmism to have a contingency plan if everything goes to hell. Much like Peter Thiel has one.
bane 10y ago
The reason I posted a reddit thread is that the discussion is full of many other related discussions.
smkellat 10y ago
Any particular reason we absolutely must DDoS this server at the moment? From a current session: ftp> open aftp.cmdl.noaa.gov Connected to aftp.cmdl.noaa.gov. 421 There are too many connected users, please try later. ftp>
- bane 10y ago
  One of the beautiful things about ftp servers is that you can set a connection limit and they'll just refuse connections above that limit until a slot frees up.
  - IgorPartola 10y ago
    You can do the same with almost all services I know. Well, at least connection-oriented ones.
richard_todd 10y ago
Making important data "impossible" to destroy/lose/vandalize is sensible on its own merit. The more signed and decentralized data is, the better. You don't have to be against the new US President to think it's a good idea, but if the political climate brings attention to the matter that would be one silver lining anyway.
deleted 10y ago
[deleted]
deleted 10y ago
[deleted]
Skunkleton 10y ago
It would be cool to get this all available via bittorrent. The data could be split up into reasonable sized chunks, and everyone could store a little, and everyone would still have immediate access to the whole thing.

85 points by bane 10y ago

Also note https://github.com/climate-mirror/datasets/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc https://github.com/climate-mirror/datasets/issues?q=is%3Aiss... Seems like archive.org should be archiving this stuff, so it's all in one place and future researchers don't have to go hunting all over the internet for it.

pimlottc 10y ago
Jason Scott (twitter handle @textfiles) is already on it: https://twitter.com/textfiles/status/824104225688981504 https://twitter.com/textfiles/status/824104225688981504
- bane 10y ago
  Jason Scott et al may find themselves in line for a medal of freedom at some point in the future.

barney54 10y ago

Why is there any threat of these databases going away?

astrodust 10y ago
If you're asking that question you haven't been paying attention. Science is the enemy. It will be destroyed if it continues to get in the way of the administration. It happened here (Canada) http://www.macleans.ca/news/canada/vanishing-canada-why-were-all-losers-in-ottawas-war-on-data/ http://www.macleans.ca/news/canada/vanishing-canada-why-were... and it could happen on an even bigger scale in America. The Conservative party went about cutting funding, slashing archives, burning everything to the ground, sometimes literally. They'd do it with little warning, zero fanfare, and an aggressive timeline. One day you had a climate archive, the next the shredding company had taken care of it. Those servers aren't free. They depend on budgets and grants to stay running. If that money is cut, those files are gone.
- deleted 10y ago
  [deleted]
rhizome 10y ago
Because science is a source of truth.
jakeogh 10y ago
Won't happen.
trhway 10y ago
because they most probably contradict alternative facts.

astrodust 10y ago

We should be mirroring fucking everything. Where's a good place to put data like this? Is there even a place where people can start to inventory what's at risk? The biggest problem is not knowing what's out there, what records need to be protected, as there's so many people doing important work.

kalleboo 10y ago
The internet archive has made this their mission. Archive Team keeps a constant eye out for major content sites that are at risk of being shut down and proactively mirroring their content. http://www.archiveteam.org/index.php?title=Main_Page http://www.archiveteam.org/index.php?title=Main_Page On EPA data https://twitter.com/textfiles/status/824110893034311680 https://twitter.com/textfiles/status/824110893034311680 They don't just archive online content either. They spend many resources scanning magazines, ingesting (and cataloging) CDs and floppies before they all rot away.
- astrodust 10y ago
  As much as it's probably a logistical nightmare, they basically need to back up everything now. This is triage time because everything is at risk of being shut down. You know when the Internet Archive itself is making a contingency plan to move out of the US that things are truly fucked. The most difficult thing here is any prirotization they end up doing must be kept absolutely secret or you're going to be providing a hit-list to the administration should they ever want one.
  - barney54 10y ago
    Where is the evidence that any of the data sites are at risk of being shutdown?
    - lazerwalker 10y ago
      I don't think there's direct concrete evidence, but that the Internet Archive themselves just ran a crowdfunding campaign to open a redundant data center in Canada suggests it is something they are worried about.

niftich 10y ago

Things to keep in mind: - In the US, the CFAA has language that may or may not be relevant [1]. This isn't intended to be FUD, although I can understand if you think so. As with most legal things, don't believe me -- a random guy -- on the internet; ask a lawyer, or at the very least be aware of your risks. - Check license and terms, if included. Reddit post claims these [2] are terms, seems okay, but I have not confirmed this for myself. - FTP is unencrypted and non-tamperproof, so data integrity and data authenticity, and connection privacy isn't guaranteed. And because the original source did not publish checksums over a secure channel, there is no way to know whether these files are the originals. It will be difficult to prove the provenance and accuracy of the data, despite having lots of third-party copies, without the original source having safeguarded the strong cryptographic checksums the entire time, or having published them over a tamperproof channel. [1] https://www.law.cornell.edu/uscode/text/18/1030 https://www.law.cornell.edu/uscode/text/18/1030 [2] https://www.reddit.com/r/DataHoarder/comments/5q4xxe/erik_fichtner_on_twitter_please_wget_m_np/dcwo823/ https://www.reddit.com/r/DataHoarder/comments/5q4xxe/erik_fi...

jMyles 10y ago
> strong cryptographic checksums They needn't even be strong, right? Even a weak checksum on a data set of this size is basically collision-proof, no?
- niftich 10y ago
  It absolutely needs to be a strong cryptographic hash, because you want to protect against both deliberate tampering and accidental modification. Any other function, whether non-cryptographic, too short, broken or having known weaknesses, while perhaps suitable to detect accidental corruption, will not protect against the former. For example, SHA-384, or SHA-3 are great choices.
  - jMyles 10y ago
    Gotcha. TIL.
  - Dylan16807 10y ago

Proven 10y ago

Haha, screw that! Now that you have to rely on help from others suddenly there's "please". When the thugs from previous administration coerced and stole money from taxpayers to pay for this crap, "please" was nowhere to be seen. I may download this stuff several times to /dev/null from others who will be seeding it. Hope that helps!

losvedir 10y ago

Should I know who Erik Ficthner is? Is he an NOAA scientist? From what I heard on NPR this morning and a nytimes piece, the Trump administration doesn't seem to be doing anything unusual. I thought it was customary for the incoming administration to make sure the agency is all on the same page. I'm getting hysteria fatigue, I think.

eropple 10y ago
The anti-science posture of Trump specifically and Republicans generally is alarming by itself. The steady denials of observable fact by this administration and the immediate scrubbing of executive-controlled public relations pages of factual materials about topics that are very inconvenient for them makes worries of Orwell's memory hole very, very reasonable. This isn't hysteria. This is insurance. Don't be That Guy.
JanneVee 10y ago
I'll leave this here http://www.vanityfair.com/news/2017/01/why-does-peter-thiel-have-new-zealand-citizenship http://www.vanityfair.com/news/2017/01/why-does-peter-thiel-... It isn't alarmism to have a contingency plan if everything goes to hell. Much like Peter Thiel has one.

bane 10y ago

The reason I posted a reddit thread is that the discussion is full of many other related discussions.

smkellat 10y ago

Any particular reason we absolutely must DDoS this server at the moment? From a current session: ftp> open aftp.cmdl.noaa.gov Connected to aftp.cmdl.noaa.gov. 421 There are too many connected users, please try later. ftp>

bane 10y ago
One of the beautiful things about ftp servers is that you can set a connection limit and they'll just refuse connections above that limit until a slot frees up.
- IgorPartola 10y ago
  You can do the same with almost all services I know. Well, at least connection-oriented ones.

richard_todd 10y ago

Making important data "impossible" to destroy/lose/vandalize is sensible on its own merit. The more signed and decentralized data is, the better. You don't have to be against the new US President to think it's a good idea, but if the political climate brings attention to the matter that would be one silver lining anyway.

deleted 10y ago

[deleted]

deleted 10y ago

[deleted]

Skunkleton 10y ago

It would be cool to get this all available via bittorrent. The data could be split up into reasonable sized chunks, and everyone could store a little, and everyone would still have immediate access to the whole thing.