4 ms·

Have you tried NASA's Earthdata Search? https://search.earthdata.nasa.gov/ https://search.earthdata.nasa.gov/

throwawaymath 8y ago
I have. I think it's definitely a step in the right direction. But I have a few observations specific to EarthData Search: 1. It places a huge emphasis on visualization. The interface is extraordinarily opinionated. The first thing you're presented with a map which is, in my opinion, completely superfluous. The intention seems to be an intuitive interface, but I would not call this intuitive whatsoever. 2. Search results are not organized in a transparent way, and metadata is not presented consistently. There is no single format for the principal name, stewarding agency, associated department(s), citation information, dataset size, category, temporal resolution(s), spatial resolution(s), date range, geographic range, update frequency, access information, etc for datasets in each result. Contrast this with Wolfram Alpha or Dark Sky, which both neatly inform you of the sources used in various queries and requests. 3. The search experience is very slow. This doesn't seem to be intrinsic to how much data is being searched - rather, it seems to be an artifact of how results are being presented. It's not actually reading over the datasets themselves, it's just returning dataset metadata. But search queries for generic climate terminology can take tens of seconds to load, with no progress indicator. 4. This site still suffers from what appears to be a systemic problem of maintainability: it's easy to trigger "Error retrieving site preferences There was a problem completing the request" errors (on latest Chrome, latest macOS). An alert in the upper right hand corner indicates that ASDC_DAAC LARC data is currently experiencing download errors and there's explicitly no estimate for when this will be resolved. EarthData Search is one of the websites I was thinking of when I said "portals" - this is clearly a well-intentioned effort, and it's a lot better than browsing undocumented FTP servers. But for climate scientists (or cross-pollinated researchers, such as statisticians and "data scientists"), a much more static interface with a common format for searching and documenting all of this data would be much more useful.
- deleted 8y ago
  [deleted]
- earthscienceman 8y ago
  I'm a scientist working on satellite data products at one of the two four letter agencies you named. I have published papers on these data products and myself work on processing of level 2 and 3 data products. A key point that needs to follow your comments is that various data products that are often purpose made for the goals of the people that use them... and that interpreting said data requires understanding the choices made, although this can be found in the papers that document the products. What this means though, in the long term, is that there's often more than one data-set for each parameter and they're all horribly documented. [most data of this kind] .. is unfortunately mired in a sea of complexity, filled with shifting departments NOAA works directly with NASA for much of this kind of data, and stewards it online You seem to have a good grasp of the current state of "climate data" in the US (and Europe). It's hard for scientists to simply get enough money to launch a satellite and that money usually doesn't include a budget for "packaging and organizing data to be published". For the most part, NASA provides "official" data products for its satellites and then enters into partnerships with other agencies like NOAA (and other scientists) to use and host these data products. Often this means refining the NASA products into something more useful. Every grant I've seen that's creating/taking its own have had one line that says "will put data up on ftp" and nothing more. Also, as you're likely quickly finding out... the storage and processing requirements are often "enterprise class". For quick reference, the level 3 product I work on (EASE-gridded daily means) is roughly 30TB. This is one product from one satellite for one type of observation. The level 1 data that's at native time/spacial resolution is something like 3TB per day. There are several agencies who work on a similar product with different goals, from the same level 1 data. I can say in no uncertain terms that you're unlikely to see anyone organize these data sets any time soon. I admire you for attempting to tackle it. I'm not sure what your mission is but if I were to advise you myself (having seen 30 different agency sponsored hosting sites come and go) your time would be most wisely used in two ways: 1) If you're really determined to host the data yourself... find and choose what you think are the most important products from each satellite, archive those products and deeply document the sources (i.e. papers published and up-to-date links to related products that you aren't hosting). 2)Or what I think needs to be done: Create a web portal... like... a really good web portal. This doesn't exist, which is obviously the motivation for what you're doing. Something that cleverly organizes and documents the different types of climate data and the places that you can find them and the places/papers they're documented in. In my humble opinion, this is what needs done most. NASA/NOAA/everyone else will gladly host the data but they're never going to create a "climate data portal" that documents and catalogs data from the other agencies and sources. Well, not never... they've tried. It's just really difficult, these are the ones I know: https://climatedataguide.ucar.edu/ https://climatedataguide.ucar.edu/ (NCAR Boulder, CO, USA) https://rda.ucar.edu/ https://rda.ucar.edu/ (NCAR "research data archive") http://nsidc.org/data/search/ http://nsidc.org/data/search/ (NSIDC Boulder, CO, USA) https://data.nasa.gov/ https://data.nasa.gov/ (NASA, USA) http://cci.esa.int/data http://cci.esa.int/data (European Space Agency) http://www.geoportal.org/ http://www.geoportal.org/ (ESA portal) http://www.ipcc-data.org/ http://www.ipcc-data.org/ (IPCC focused data sources) http://www.ciesin.columbia.edu/ http://www.ciesin.columbia.edu/ (CIESIN Columbia University) https://iridl.ldeo.columbia.edu/index.html https://iridl.ldeo.columbia.edu/index.html (IRIDL Columbia) https://help.ceda.ac.uk/article/99-download-data-from-ceda-archives https://help.ceda.ac.uk/article/99-download-data-from-ceda-a... (CEDA, London UK) the right hand doesn't know what the left is doing here It's more so that the right hand has nothing to do with the left hand. Each agency, each satellite, each group of scientists, is totally independent of the other. While sometimes there's noble leaps taken to work together... there's not much incentive financially or otherwise. Not to mention that each team is so specialized that they often know nothing about the other 98% of the data. Take me for example, I've helped work on a very popular data set for a specific part of the earth system... I don't think I've ever once looked at ocean data... I don't even know what's out there or how good it is or where you find it.

I have. I think it's definitely a step in the right direction. But I have a few observations specific to EarthData Search: 1. It places a huge emphasis on visualization. The interface is extraordinarily opinionated. The first thing you're presented with a map which is, in my opinion, completely superfluous. The intention seems to be an intuitive interface, but I would not call this intuitive whatsoever. 2. Search results are not organized in a transparent way, and metadata is not presented consistently. There is no single format for the principal name, stewarding agency, associated department(s), citation information, dataset size, category, temporal resolution(s), spatial resolution(s), date range, geographic range, update frequency, access information, etc for datasets in each result. Contrast this with Wolfram Alpha or Dark Sky, which both neatly inform you of the sources used in various queries and requests. 3. The search experience is very slow. This doesn't seem to be intrinsic to how much data is being searched - rather, it seems to be an artifact of how results are being presented. It's not actually reading over the datasets themselves, it's just returning dataset metadata. But search queries for generic climate terminology can take tens of seconds to load, with no progress indicator. 4. This site still suffers from what appears to be a systemic problem of maintainability: it's easy to trigger "Error retrieving site preferences There was a problem completing the request" errors (on latest Chrome, latest macOS). An alert in the upper right hand corner indicates that ASDC_DAAC LARC data is currently experiencing download errors and there's explicitly no estimate for when this will be resolved. EarthData Search is one of the websites I was thinking of when I said "portals" - this is clearly a well-intentioned effort, and it's a lot better than browsing undocumented FTP servers. But for climate scientists (or cross-pollinated researchers, such as statisticians and "data scientists"), a much more static interface with a common format for searching and documenting all of this data would be much more useful.

deleted 8y ago
[deleted]
earthscienceman 8y ago
I'm a scientist working on satellite data products at one of the two four letter agencies you named. I have published papers on these data products and myself work on processing of level 2 and 3 data products. A key point that needs to follow your comments is that various data products that are often purpose made for the goals of the people that use them... and that interpreting said data requires understanding the choices made, although this can be found in the papers that document the products. What this means though, in the long term, is that there's often more than one data-set for each parameter and they're all horribly documented. [most data of this kind] .. is unfortunately mired in a sea of complexity, filled with shifting departments NOAA works directly with NASA for much of this kind of data, and stewards it online You seem to have a good grasp of the current state of "climate data" in the US (and Europe). It's hard for scientists to simply get enough money to launch a satellite and that money usually doesn't include a budget for "packaging and organizing data to be published". For the most part, NASA provides "official" data products for its satellites and then enters into partnerships with other agencies like NOAA (and other scientists) to use and host these data products. Often this means refining the NASA products into something more useful. Every grant I've seen that's creating/taking its own have had one line that says "will put data up on ftp" and nothing more. Also, as you're likely quickly finding out... the storage and processing requirements are often "enterprise class". For quick reference, the level 3 product I work on (EASE-gridded daily means) is roughly 30TB. This is one product from one satellite for one type of observation. The level 1 data that's at native time/spacial resolution is something like 3TB per day. There are several agencies who work on a similar product with different goals, from the same level 1 data. I can say in no uncertain terms that you're unlikely to see anyone organize these data sets any time soon. I admire you for attempting to tackle it. I'm not sure what your mission is but if I were to advise you myself (having seen 30 different agency sponsored hosting sites come and go) your time would be most wisely used in two ways: 1) If you're really determined to host the data yourself... find and choose what you think are the most important products from each satellite, archive those products and deeply document the sources (i.e. papers published and up-to-date links to related products that you aren't hosting). 2)Or what I think needs to be done: Create a web portal... like... a really good web portal. This doesn't exist, which is obviously the motivation for what you're doing. Something that cleverly organizes and documents the different types of climate data and the places that you can find them and the places/papers they're documented in. In my humble opinion, this is what needs done most. NASA/NOAA/everyone else will gladly host the data but they're never going to create a "climate data portal" that documents and catalogs data from the other agencies and sources. Well, not never... they've tried. It's just really difficult, these are the ones I know: https://climatedataguide.ucar.edu/ https://climatedataguide.ucar.edu/ (NCAR Boulder, CO, USA) https://rda.ucar.edu/ https://rda.ucar.edu/ (NCAR "research data archive") http://nsidc.org/data/search/ http://nsidc.org/data/search/ (NSIDC Boulder, CO, USA) https://data.nasa.gov/ https://data.nasa.gov/ (NASA, USA) http://cci.esa.int/data http://cci.esa.int/data (European Space Agency) http://www.geoportal.org/ http://www.geoportal.org/ (ESA portal) http://www.ipcc-data.org/ http://www.ipcc-data.org/ (IPCC focused data sources) http://www.ciesin.columbia.edu/ http://www.ciesin.columbia.edu/ (CIESIN Columbia University) https://iridl.ldeo.columbia.edu/index.html https://iridl.ldeo.columbia.edu/index.html (IRIDL Columbia) https://help.ceda.ac.uk/article/99-download-data-from-ceda-archives https://help.ceda.ac.uk/article/99-download-data-from-ceda-a... (CEDA, London UK) the right hand doesn't know what the left is doing here It's more so that the right hand has nothing to do with the left hand. Each agency, each satellite, each group of scientists, is totally independent of the other. While sometimes there's noble leaps taken to work together... there's not much incentive financially or otherwise. Not to mention that each team is so specialized that they often know nothing about the other 98% of the data. Take me for example, I've helped work on a very popular data set for a specific part of the earth system... I don't think I've ever once looked at ocean data... I don't even know what's out there or how good it is or where you find it.