6 ms·

This is great news for a project I'm currently working on: https://news.ycombinator.com/item?id=17237445 https://news.ycombinator.com/item?id=17237445. I'll be

This is great news for a project I'm currently working on: https://news.ycombinator.com/item?id=17237445 https://news.ycombinator.com/item?id=17237445. I'll be sure to look into this and see what's new compared to what's already out there. There is a lot of public satellite imagery. That said, I wouldn't be surprised if this isn't actually the first surfacing of this particular dataset. My experience is that most data of this kind (satellite, land station, oceanic/marine, balloon, radar, etc...) is already available online. It's unfortunately mired in a sea of complexity, filled with shifting departments, confusing acronyms, funding depletion and musical chairs. Here is my favorite example. The NOAA works directly with NASA for much of this kind of data, and stewards it online here: https://www.ncdc.noaa.gov/data-access https://www.ncdc.noaa.gov/data-access. Observe that the URL indicates the department is the NCDC (National Climatic Data Center). As it turns out, the website informs us that the NCDC no longer exists, and is now known as the NCEI (National Centers for Environmental Information). But let's put that detail aside. Now we go to the page listed for satellite datasets: https://www.ncdc.noaa.gov/data-access/satellite-data/satellite-data-access-datasets https://www.ncdc.noaa.gov/data-access/satellite-data/satelli.... Lots of acronyms - acronyms for everything! Let's try the OISST (Optimum Interpolation Sea Surface Temperature) dataset: http://www.ncdc.noaa.gov/sst/ http://www.ncdc.noaa.gov/sst/. Whoops, that 404s. It also doesn't tell us what the acronym stands for. Fortunately we can find the canonical page by Googling for it. Okay, let's try another dataset back on that first page. How about the SSM/I-SSMIS (Special Sensor Microwave/Imager and Special Sensor Microwave Imager Sounder): https://www.ncdc.noaa.gov/ssmi https://www.ncdc.noaa.gov/ssmi? Well, we're told we have to go to http://www.class.noaa.gov/ http://www.class.noaa.gov/ to get that data. Now the first thing we see on that page is an "urgent notice" that URLs are breaking, and anyone using the current ones (for e.g. FTP access) has to migrate to a slightly different URL with a choice of subdomains appended. Moving past that initial notice, if we actually try to find the SSMI dataset, we find out that it's actually listed as the DMSP (Defense Meteorological Satellite Program): https://www.bou.class.noaa.gov/saa/products/search?datatype_family=DMSP https://www.bou.class.noaa.gov/saa/products/search?datatype_.... Now to download this data, we're presented with this wacky interface: https://www.bou.class.noaa.gov/saa/products/psearchDMSP https://www.bou.class.noaa.gov/saa/products/psearchDMSP. This is basically why I decided to work on my current project to consolidate, document and organize all of this data in a public search engine. It feels like so much climate data is only public in the absolute minimum sense of the word, because trying to access it (or even make sense of all of it) transports me back to 2006. There's a very real sense that the right hand doesn't know what the left is doing here. Clearly these organizations are aware that accessing the data is difficult, and they must have good intentions by going through the effort to document it and host it in FTP servers. But there are so many conflicting names and acronyms for the "stewards" for overseeing this or that data, which redirect you to various "portals" for searching it, which tell you to add data to your "cart" to "order" in some sort of contrived interface. It's maddening. Well, there ends my rant for the day. If anyone has any good ideas for working on this, please feel free to get in touch.

mkstowegnv 8y ago
Thank you for this work. Would that other datasets had such an overseer
- throwawaymath 8y ago
  Don't thank me yet, there's still quite a lot to be done. I wouldn't call myself an "overseer" :)
  - deleted 8y ago
    [deleted]
craftsman 8y ago
Have you tried NASA's Earthdata Search? https://search.earthdata.nasa.gov/ https://search.earthdata.nasa.gov/
- throwawaymath 8y ago
  I have. I think it's definitely a step in the right direction. But I have a few observations specific to EarthData Search: 1. It places a huge emphasis on visualization. The interface is extraordinarily opinionated. The first thing you're presented with a map which is, in my opinion, completely superfluous. The intention seems to be an intuitive interface, but I would not call this intuitive whatsoever. 2. Search results are not organized in a transparent way, and metadata is not presented consistently. There is no single format for the principal name, stewarding agency, associated department(s), citation information, dataset size, category, temporal resolution(s), spatial resolution(s), date range, geographic range, update frequency, access information, etc for datasets in each result. Contrast this with Wolfram Alpha or Dark Sky, which both neatly inform you of the sources used in various queries and requests. 3. The search experience is very slow. This doesn't seem to be intrinsic to how much data is being searched - rather, it seems to be an artifact of how results are being presented. It's not actually reading over the datasets themselves, it's just returning dataset metadata. But search queries for generic climate terminology can take tens of seconds to load, with no progress indicator. 4. This site still suffers from what appears to be a systemic problem of maintainability: it's easy to trigger "Error retrieving site preferences There was a problem completing the request" errors (on latest Chrome, latest macOS). An alert in the upper right hand corner indicates that ASDC_DAAC LARC data is currently experiencing download errors and there's explicitly no estimate for when this will be resolved. EarthData Search is one of the websites I was thinking of when I said "portals" - this is clearly a well-intentioned effort, and it's a lot better than browsing undocumented FTP servers. But for climate scientists (or cross-pollinated researchers, such as statisticians and "data scientists"), a much more static interface with a common format for searching and documenting all of this data would be much more useful.

mkstowegnv 8y ago

Thank you for this work. Would that other datasets had such an overseer

throwawaymath 8y ago
Don't thank me yet, there's still quite a lot to be done. I wouldn't call myself an "overseer" :)
- deleted 8y ago
  [deleted]

craftsman 8y ago

Have you tried NASA's Earthdata Search? https://search.earthdata.nasa.gov/ https://search.earthdata.nasa.gov/

throwawaymath 8y ago
I have. I think it's definitely a step in the right direction. But I have a few observations specific to EarthData Search: 1. It places a huge emphasis on visualization. The interface is extraordinarily opinionated. The first thing you're presented with a map which is, in my opinion, completely superfluous. The intention seems to be an intuitive interface, but I would not call this intuitive whatsoever. 2. Search results are not organized in a transparent way, and metadata is not presented consistently. There is no single format for the principal name, stewarding agency, associated department(s), citation information, dataset size, category, temporal resolution(s), spatial resolution(s), date range, geographic range, update frequency, access information, etc for datasets in each result. Contrast this with Wolfram Alpha or Dark Sky, which both neatly inform you of the sources used in various queries and requests. 3. The search experience is very slow. This doesn't seem to be intrinsic to how much data is being searched - rather, it seems to be an artifact of how results are being presented. It's not actually reading over the datasets themselves, it's just returning dataset metadata. But search queries for generic climate terminology can take tens of seconds to load, with no progress indicator. 4. This site still suffers from what appears to be a systemic problem of maintainability: it's easy to trigger "Error retrieving site preferences There was a problem completing the request" errors (on latest Chrome, latest macOS). An alert in the upper right hand corner indicates that ASDC_DAAC LARC data is currently experiencing download errors and there's explicitly no estimate for when this will be resolved. EarthData Search is one of the websites I was thinking of when I said "portals" - this is clearly a well-intentioned effort, and it's a lot better than browsing undocumented FTP servers. But for climate scientists (or cross-pollinated researchers, such as statisticians and "data scientists"), a much more static interface with a common format for searching and documenting all of this data would be much more useful.