4 ms·

I'm a scientist working on satellite data products at one of the two four letter agencies you named. I have published papers on these data products and myself w

I'm a scientist working on satellite data products at one of the two four letter agencies you named. I have published papers on these data products and myself work on processing of level 2 and 3 data products. A key point that needs to follow your comments is that various data products that are often purpose made for the goals of the people that use them... and that interpreting said data requires understanding the choices made, although this can be found in the papers that document the products. What this means though, in the long term, is that there's often more than one data-set for each parameter and they're all horribly documented. [most data of this kind] .. is unfortunately mired in a sea of complexity, filled with shifting departments NOAA works directly with NASA for much of this kind of data, and stewards it online You seem to have a good grasp of the current state of "climate data" in the US (and Europe). It's hard for scientists to simply get enough money to launch a satellite and that money usually doesn't include a budget for "packaging and organizing data to be published". For the most part, NASA provides "official" data products for its satellites and then enters into partnerships with other agencies like NOAA (and other scientists) to use and host these data products. Often this means refining the NASA products into something more useful. Every grant I've seen that's creating/taking its own have had one line that says "will put data up on ftp" and nothing more. Also, as you're likely quickly finding out... the storage and processing requirements are often "enterprise class". For quick reference, the level 3 product I work on (EASE-gridded daily means) is roughly 30TB. This is one product from one satellite for one type of observation. The level 1 data that's at native time/spacial resolution is something like 3TB per day. There are several agencies who work on a similar product with different goals, from the same level 1 data. I can say in no uncertain terms that you're unlikely to see anyone organize these data sets any time soon. I admire you for attempting to tackle it. I'm not sure what your mission is but if I were to advise you myself (having seen 30 different agency sponsored hosting sites come and go) your time would be most wisely used in two ways: 1) If you're really determined to host the data yourself... find and choose what you think are the most important products from each satellite, archive those products and deeply document the sources (i.e. papers published and up-to-date links to related products that you aren't hosting). 2)Or what I think needs to be done: Create a web portal... like... a really good web portal. This doesn't exist, which is obviously the motivation for what you're doing. Something that cleverly organizes and documents the different types of climate data and the places that you can find them and the places/papers they're documented in. In my humble opinion, this is what needs done most. NASA/NOAA/everyone else will gladly host the data but they're never going to create a "climate data portal" that documents and catalogs data from the other agencies and sources. Well, not never... they've tried. It's just really difficult, these are the ones I know: https://climatedataguide.ucar.edu/ https://climatedataguide.ucar.edu/ (NCAR Boulder, CO, USA) https://rda.ucar.edu/ https://rda.ucar.edu/ (NCAR "research data archive") http://nsidc.org/data/search/ http://nsidc.org/data/search/ (NSIDC Boulder, CO, USA) https://data.nasa.gov/ https://data.nasa.gov/ (NASA, USA) http://cci.esa.int/data http://cci.esa.int/data (European Space Agency) http://www.geoportal.org/ http://www.geoportal.org/ (ESA portal) http://www.ipcc-data.org/ http://www.ipcc-data.org/ (IPCC focused data sources) http://www.ciesin.columbia.edu/ http://www.ciesin.columbia.edu/ (CIESIN Columbia University) https://iridl.ldeo.columbia.edu/index.html https://iridl.ldeo.columbia.edu/index.html (IRIDL Columbia) https://help.ceda.ac.uk/article/99-download-data-from-ceda-archives https://help.ceda.ac.uk/article/99-download-data-from-ceda-a... (CEDA, London UK) the right hand doesn't know what the left is doing here It's more so that the right hand has nothing to do with the left hand. Each agency, each satellite, each group of scientists, is totally independent of the other. While sometimes there's noble leaps taken to work together... there's not much incentive financially or otherwise. Not to mention that each team is so specialized that they often know nothing about the other 98% of the data. Take me for example, I've helped work on a very popular data set for a specific part of the earth system... I don't think I've ever once looked at ocean data... I don't even know what's out there or how good it is or where you find it.

throwawaymath 8y ago
This is all very useful information for me, thank you. Would you be willing to send me an email so I can ask you further questions? I'm in the process of reaching out to other climate scientists as well.
mturmon 8y ago
Thank you for writing the above well-informed comment. One important observation at the top of the comment is: "data products ... are often purpose made for the goals of the people that use them" -- and this explains a lot of the atomization and overlap among these data. You can see this effect across NASA science data (i.e., not just Earth). A group of investigators sees the potential to make an innovative new measurement, and develops the instrument or retrieval system to get it. This can be a decade-long process. The best people to talk about the features and limitations of that data are, clearly, that team. And they are generally quite approachable for other in-discipline specialists at the usual conferences and workshops. There will be published documents as well, but they are best if you already know what you want. But these methods do not help outsiders! It can be hard for out-of-discipline people to discover what data (among what's available) would be helpful or applicable for their purposes. It can be very tricky - "you can't use data X for purpose Y because we filter effect Z out of X, and Z is critical to doing Y." One other general point - related to the data maturity pipeline. NASA is chartered with developing, exploiting, and archiving novel measurements. NOAA is chartered with operational analysis and use. So there is a handoff needed there, and a difference in emphasis.

throwawaymath 8y ago

This is all very useful information for me, thank you. Would you be willing to send me an email so I can ask you further questions? I'm in the process of reaching out to other climate scientists as well.

mturmon 8y ago

Thank you for writing the above well-informed comment. One important observation at the top of the comment is: "data products ... are often purpose made for the goals of the people that use them" -- and this explains a lot of the atomization and overlap among these data. You can see this effect across NASA science data (i.e., not just Earth). A group of investigators sees the potential to make an innovative new measurement, and develops the instrument or retrieval system to get it. This can be a decade-long process. The best people to talk about the features and limitations of that data are, clearly, that team. And they are generally quite approachable for other in-discipline specialists at the usual conferences and workshops. There will be published documents as well, but they are best if you already know what you want. But these methods do not help outsiders! It can be hard for out-of-discipline people to discover what data (among what's available) would be helpful or applicable for their purposes. It can be very tricky - "you can't use data X for purpose Y because we filter effect Z out of X, and Z is critical to doing Y." One other general point - related to the data maturity pipeline. NASA is chartered with developing, exploiting, and archiving novel measurements. NOAA is chartered with operational analysis and use. So there is a handoff needed there, and a difference in emphasis.