31 ms·
Mirroring US climate data
- nandhp 10y agoIs this project collaborating with Archive Team? They have quite a lot of experience with this sort of scraping+archiving project (e.g. Geocities, Google Reader, Coursera, etc.). http://archiveteam.org/index.php?title=Main_Page http://archiveteam.org/index.php?title=Main_Page
- trothamel 10y agoCan I be the first to claim that the people running the mirror are changing the data for their own nefarious purposes? Not that I believe this or anything, but come on - you know that accusation is going to go around. It might make sense to use a remote server to timestamp files as they're uploaded, or something like that.
- caseysoftware 10y agoBefore publishing, they could do a checksum, insert it into the blockchain and then voila. Easy to validate if the data has changed.
- ajmurmann 10y agoI am not sure checksums and block chains are gonna convince those people.
- nirav72 10y agoWhen there is a broad consensus across the scientific community about climate change, does it even matter at this point if the data has been changed? The U.S isn't the only country that collects this data. So even if changed, it doesn't matter.
- ted_dunning 10y agoActually it really does matter. The US collects far more of this data than anybody else and archives far more of it. Losing that collection or archive would be a disaster for science and a loss for all of humanity. Destroying that data would be reminiscent of ISIS destroying relics in Palmyra except that destroying that data jeopardizes our future.
- gns24 10y agoPersonally I don't see that it's that important. We know already that CO2 and other gases cause climate change and we understand the mechanism. I think that most of us would probably also agree that the earth is a sufficiently complex system that predicting accurately what effects we will see when is impossible. We expect to see more droughts, bigger storms and rising sea levels - but we can't really predict very accurately how big those things will be when. By the time we have accurate models it will be too late. We know that we need to act now. Understanding climate change is not the most important research out there; what's important is developing technologies and industries to reduce our emissions. That research would be what I fear losing.
- tracker1 10y agoBut "we need to act now" can lead to a lot of ineffective policies and practices that won't lead to the most impacting outcomes long term. I think reciprocal clean working environments as part of trade negotiations could go farther than a lot of the attempts so far. I also feel that no participating country should be exempt, beyond a reasonable grace period (2-3 years). It's also worth considering the impact of some movements... as an example, there's significant impact in creating/shipping and replacing batteries on a large scale. It may not make as much sense compared to more efficient engines, and even additional looks at other engine techniques. I find pollution relatively abhorrent on its' own though... recently driving through west Louisianan/east Texas and my initial thoughts as to the painful to breath air there was how can this be allowed to persist. I have mixed feelings all around, I just don't want to spend a lot of taxpayer money on ill-advised and lesser impacting processes. We do too much of that already.
- lkbm 10y ago> When there is a broad consensus across the scientific community about climate change, does it even matter at this point if the data has been changed? There's more to climate science than the yes/no of whether climate change is a thing. We don't just need to know it's happening--we want to know details of how to model it, what different inputs have what affects, what the feedback loops are and how strong they are, how they and other effects interact... > The U.S isn't the only country that collects this data. So even if changed, it doesn't matter. Many countries collect data, but presumably the sets of data collected differ, and more data is better. Losing a large subset of our climate data would hurt advances in climate modeling.
- pdonis 10y ago> When there is a broad consensus across the scientific community about climate change, does it even matter at this point if the data has been changed? It does if the data that was changed is the data that was used to support the consensus. More generally, all data collected by research that is funded by taxpayers should be collected, cryptographically signed and verified, and archived where any taxpayer can see it. I have never understood why that is not already standard operating procedure.
- tracker1 10y agoUsually around IP incentives as part of the grant, in order to further entice applicants. This is taken further by pressure from educational institutions and their researchers to file for patents against said research. I'm not saying it's right, and agree with you. But often there's a lot of entropy with the status quo, and to the seated incumbents in a space.
- tukelully 10y agoDoes this carry over to federal agencies as well? I know the state of access to journals and papers published by universities is a shitshow but don't know how it is at the government level.
- tracker1 10y agoA lot of government research is completed by universities and private companies under grant programs.
- gus_massa 10y agoConsensus without data is indistinguishable from religion.
- battlebot 10y agoThere was once "broad consensus" that the world was flat. I am not at all comfortable when there is so much consensus. There needs to be debate, disagreement, and healthy skepticism at all times.
- pirocks 10y agoI would argue that there has never really been a "broad consensus" that the world was flat, since before the greeks. Most of the notable greek philosophers believed that the earth was a ball/sphere.
- dawnerd 10y agoOn the actual mirrors site you can see some have been signed and verified: https://climate.daknob.net/ https://climate.daknob.net/ Ultimately still comes down to trust. Still better to have a backup of data that MIGHT be poisoned vs not having it at all. If we have multiple copies from the original source we should be able to determine if somethings been messed with.
- esaym 10y agoI would say the same thing. Do you think you will really convince a climate denier with this data that you copied off of the Internet that, was copied from a gov server, on to another server and then torrented and copied to another server and then downloaded onto you PC... doubt it.
- tracker1 10y agoIt's worth noting that not everyone who rebuffs some of the proposed actions is a climate denier. There's still some question as to the extent of human involvement. That said, smog is enough for me to be in favor of working towards renewable, less polluting energy sources. In my lifetime I've gone from having allergy-like flare ups once every few years, to now 2-3 times a year, mostly seems to follow the pollution levels not just the pollen count. By the same token, not every "green" program is a great thing... and even in treaties, exempting the worst actors (China, India) because they are often considered third world doesn't do a lot to help the global issues. Improved working requirements as a requisite for international trade deals would go a long way towards "cleaning up." I bring this up, as I tend to see a lot of people who question the approach, intensity, and cost getting lumped in with climate deniers.
- tukelully 10y agoIn your first sentence which actions are you referring to?
- simonbarker87 10y agoNot OP but the general complaint is that we shouldn't switch from a cheap energy source (coal) to expensive renewables which are relatively untested at scale if the human impact of climate change is minimal or already past the point of no return. I don't personally believe this but this is a not unreasonable argument I have heard.
- czep 10y agoContribution to climate change shouldn't be the only criteria for evaluation of an energy source. Shouldn't we also keep in mind pollution? Coal burns, it releases fine particulates into the air that cause cancer in people's lungs. Pollution sucks. We should stop talking about climate change and go back to the 70s when people were talking about pollution.
- laurent123456 10y agoI think that's why they suggest to use Webrecorder, to "create verifiable web archive files".
- anc84 10y agoRun webrecorder yourself, MITM yourself, tamper however you like. Verifiability depends on a trusted authority first.
- ocschwar 10y agoNo. The right wing troll community beat you to it months ago.
- olivierlacan 10y agoFeels like the most important things people with some spare cash can do to help this project is help them raise money to host the mirrored climate science data on online.net: https://www.generosity.com/volunteer-fundraising/climate-mirror-back-up-climate-data https://www.generosity.com/volunteer-fundraising/climate-mir...
- hackuser 10y ago> We are working to set up a secure and completely anonymous file deposit system for the future. Provide a mailing address to which to send hard drives.
- sandebert 10y agoGood idea, I posted it to their web form.
- hackuser 10y agoIt's great to see their efforts. What are you doing? What am I doing? Reading about it isn't nearly enough.
- hunterwerlla 10y agoIn torrent form: http://climatetorrent.com/ http://climatetorrent.com/ Torrents are a great way of maintaining data integrity and allowing future sharing easily.
- dev1n 10y agoIs it possible to add ftp://aftp.cmdl.noaa.gov to that torrent?
- caterama 10y agoSimilar idea: http://academictorrents.com/ http://academictorrents.com/
- BHSPitMonkey 10y agoThanks, is this yours? It would be great if the homepage displayed a short description of the different torrents/datasets, their sizes/contents, and the number of seeders for each. That would greatly help me decide which one(s) to prioritize helping out with, as I don't have resources to keep a copy of it all.
- hunterwerlla 10y agoI don't know who is behind this, I just saw it on /r/DataHoarder
- danbruc 10y agoHaving backups is obviously always a good idea but that aside, how likely is it, that mirroring this data is actually necessary? There are obviously at least some people believing that it is necessary but then there are always people convinced that the most silly things are necessary. I am really unable to tell whether this is more of an overreaction or an urgent necessity, especially as someone not living in the US. Is the future just [perceived as] so unpredictable, that better safe than sorry is the right mindset?
- po 10y agoBesides redundancy, mirroring also helps spread the cost around. These groups tend to have pretty tight budgets.
- jefurii 10y agoLots of copies keeps stuff safe.
- dylz 10y agoThink of it this way: servers cost money, bandwidth power costs money. When your budget is cut, servers will shut down at some point in the future. It isn't a "the gov says rm -rf everything"
- danbruc 10y agoI totally see how data could become unavailable, I am more asking whether it just really became more likely to happen. It is certainly no mistake not to rely on single sources, but if that is the motivation, then there is nothing special about now and climate data and mirroring all kinds of data would have been as good an idea a decade ago as now.
- crispinb 10y agoIf you only create redundancy when you are sure it's necessary, you're too late.
- po 10y agoAwesome. Is there the same thing but for all public datasets? I saw recently someone showed a federal site had put up a (hastily hacked in) link to download all data on the day of the inauguration. I bet there are a lot of federal employees who would like a well-resourced site to upload to.
- beams_of_light 10y agoI'm sincerely dismayed that this is necessary.
- general_ai 10y agoIt's not.
- chordatum 10y agoNot yet.
- general_ai 10y agoNot ever. There's no plausible scenario that I can see under which this data might be irrecoverably lost.
- r00fus 10y agoThe most worrying part is that this is a symptom of a much bigger problem. This is likely going to be one of the less egregious issues raised in the next 4 years.
- koheripbal 10y agoThis is a great example of how fear of something possibly happening, gives a larger audience the illusion that it will happen. Literally no one in the administration has even suggested, ever, that they'd delete climate data.
- deleted 10y ago[deleted]
- jszymborski 10y agoIf you want to help towards archiving climate data and other gov't public datasets, join the ArchiveTeam effort. All you need is to run their ftp-gov-grab script and a bit of bandwidth. VPSs work very well for this sort of thing, so if you have a box lying around, just run the script. More info at the wiki: http://archiveteam.org/index.php?title=Ftp-gov http://archiveteam.org/index.php?title=Ftp-gov Ftp-gov-grab: https://github.com/ArchiveTeam/ftp-gov-grab https://github.com/ArchiveTeam/ftp-gov-grab Join us on IRC: #cheetoflee on freenode
- kpcyrd 10y agoCorrection: it seems to be #cheetoflee on efnet.org
- jszymborski 10y agooops, thanks for that.
- fsttx 10y agoAnother useful tactic is to archive news and commentary (left and right) about current political events, both for accountability and perhaps, to prevent the rewriting of history in the era of "alternative facts".
- jl6 10y agoAnd not just archive it, but sign it, time stamp it, verify it, etc.. I see little to prevent someone using ML to auto-generate archives worth of fake history. Imagine a purported DB dump of Reddit with billions of subtlely fraudulent comments scattered throughout.
- ordbajsare 10y agoMore communist propaganda
- mynewtb 10y agoThis would be perfect for IPFS I think.
- Tempest1981 10y agoI heard that Trump has given NOAA 100 days to justify their existence, or face heavy cuts. Does anyone know the details? Here is the closest article I could find: http://www.salon.com/2016/11/23/politicizing-climate-change-donald-trumps-budget-could-cut-climate-funding-for-nasa-other-federal-departments/ http://www.salon.com/2016/11/23/politicizing-climate-change-...
- smarx007 10y ago"Walker has hinted that what NASA loses in earth science research could be shifted over to NOAA."
- polyfractal 10y agoThis gets thrown around a lot, but has two major problems. First, it's unlikely that comparable funding will also be transferred from NASA to NOAA. E.g. NASA will be told to stop earth science research and have it's budget cut appropriately. NOAA will be told to start earth science research, but without the requisite budget increase. So effectively, it kills earth science under the guise of "moving" it. Second, NOAA is not operationally or logistically equipped to handle the level of earth science that NASA is performing. NOAA manages a few weather satellites, but has nowhere near the infrastructure that NASA has, and requires, to manage the existing earth science research. Even if you gave NOAA the appropriate money, they don't have the infra or engineers to keep everything flying. Planetary Radio did a great episode on this topic, but I'm struggling to find it right now.
- tracker1 10y agoIt seems like Trump has been doing similar to quite a few federal agencies. I'm concerned with some of the early moves, but it's going to be interesting, I do hope that some common sense prevails. For example, bowing out of TPP isn't so bad.
- stevehawk 10y agoBowing out of the TPP has been hilarious for me to watch. 6 months ago if Obama had done it he would have been praised on nerd sites. But he didn't and Trump did so now it's an awful idea. It's right up there with my fellow nerds demanding H1B visa reform, Net Neutrality, and subsidizing clean energy while also demanding free markets. I just hear Fezzik going "I don't think that means what you think it means..." * note - i support bailing on TPP, H1B visa reform, and subsidizing clean energy
- XorNot 10y agoIt would be a good use of IPFS to mirror this data into that network.
- philippnagel 10y agoMight be a use case for https://datproject.org https://datproject.org.
- neuromantik8086 10y agoOr git-annex / git LFS.
- dyukqu 10y agoSlightly off-topic, but my guess is that the next 4 years will be the golden age of privacy (in general), file-sharing and maybe even distributed web & federated services - for the US at least.
- gramstrong 10y agoDo you mean the golden age of private infrastructure, or the golden age of folks combatting a non-private infrastructure? Because from my point of view, privacy in the US is only going to continue to get worse (and that's not even necessarily a dig at the current administration).
- dyukqu 10y agoWhat I mean is (I guess) people will be seriously considering their privacy - especially online/digital one.
- stevehawk 10y agoYup. I already formatted/reinstalled everything.. beefed up the encryption usage.. pretty much changed to a whitelist of domains on my tomato router and got me some OTP/U2F devices.. need to switch to Tor for browsing though.. and what else would people recommend?
- allemagne 10y agoIt's plausible. My sincere, dark hope is that a Trump administration will shock everyone awake in a big way to why broad surveillance is a terrible idea. Unfortunately it will probably take a huge scandal that baldly and stupidly shows how unchecked those powers are and what can be possible with them. I think Trump is capable of that kind of misstep. Like say SNL writers were PMing each other a tasteless joke about assassinating Trump and the president tweets about it and orders their arrest. In the end, something dramatic like that could actually have a really positive effect on the consciousness of the public. It could also have a really negative effect where people accept the new normal. It's hard to know.
- cool_look 10y agoSo now we have complete polarisation. Left leaning people will not accept any science generated under a Right leaning government and as we already know Right leaning people find it hard to trust the science coming from a Left leaning government.
- fisherjeff 10y agoEh, no. This isn't about not accepting scientific output under a Republican president – science was well received under Bush, as I recall, and will continue to be under Trump. This is about the fear of losing scientific data that has already been produced.
- cool_look 10y agoOn the surface it is about preventing deletion. But the subtext is that any 'discoveries' or 'corrections' will be rejected out of hand. The prime ones being the recent corrections to the land temperature record that cooled the 1930s and 1940s and heated 2009 onwards yielding a steady linear increase graphic. When those corrections are placed under the microscope and likely reversed, what will the reaction be ? Scientific shills of the Republicans are making up Alternative Facts... here is our precious archived dataset that true believers can reference.
- nikdaheratik 10y agoNo, this is complete FUD, and disingenuous at best. What people who care are worried about is a repeat of what happened in Canada under the Harper government where the data was simply not available at all. [edit] the difference is that it's not the administration or even the EPA that is putting it's reputation on the line if the data was manipulated, but the individual scientists themselves. No one with any sense believes that a scientist is going to fall on their sword and ruin their career just to make up some BS talking point that Trump can run with.
- gglitch 10y agoIs there a publicly-available blockchain specifically for checksums of journalism and research?
- fenollp 10y agoAlternative facts will push us to find a way to discontinue them. What is left to do to have truly open computation? * somewhat-to completely free computational power?: CircleCI, Travis CI, whatever * open source repositories to run code from?: Github hooks and others * trusting trust? * open hardware? * decentralized websites?
- notsrg 10y agoSurprised there's no comment here regarding the claim that using Dropbox is anonymous...
- godshatter 10y agoAlthough NoScript and uBlock Origin show nothing being blocked, I still end up with a blank page when trying to view this site. Maybe it's a Pale Moon thing.
- intrasight 10y agoMe too
- cema 10y agoAs much as I maintain a skeptical attitude to all sides of this scientific (and political) issue, I absolutely support the effort to keep the data available to the public. Science cannot move forward without the data (and metadata) available for everyone. Even more generally, society would suffer if it is not available. (When I say "metadata", I refer to the data sources, methods of the data acquisition and so on. When a scientific field is politicised it is not only hard to trust the models but even the data, so I think the metadata are very important, if they are available.)
- angelohuang 10y agoWhile I appreciate the sentiment, I think this whole notion of an AntiScience threat due to the new administration is a bit sensationalized.