3 ms·

This isn't new. Also, some data sets require sending a hard drive into the USGS I believe.

by atomical 12y ago

RIMR 12y ago
Let's say I wanted all of it. How much data is that, or would I have to ask USGS?
- jleader 12y ago
  Drawing a rough box on a map of the US I estimated about 40 degrees of longitude by 20 degrees of latitude. The most detailed maps are the 7.5-minute quadrangles, which I believe are available for nearly all of the continental US. Since 7.5 minutes is 1/8 of a degree, that gives 40208*8, or a bit more than 50,000 maps. A quick browse through the USGS site found this quote (http://store.usgs.gov/b2c_usgs/usgs/zproductinformation/%28xcm=r3standardpitrex_prd&layout=6_1_61_50_2&uiarea=2&ctype=areaDetails&carea=0000000009%29/.do#75_15Minute http://store.usgs.gov/b2c_usgs/usgs/zproductinformation/%28x...): "It takes about 57,000 maps to cover the conterminous 48 States, Hawaii, and territories." They go on to mention that Alaska's most detailed coverage is the 15-minute quadrangle series. A similar estimate for the 15-minute coverage of Alaska is around 3000 maps, plus 1/4 of 57K (14K) for the 15-minute maps of the area already covered by the 7.5-minute series, plus a few thousand for the smaller-scale series (30x60 minutes, 1x2 degrees, index maps, etc.) So I'd guess somewhere around 75-85 thousand maps. The majority of these maps were originally intended to be printed on 23x27 inch sheets of paper. Assuming they were printed at around 300 dpi, somewhere between 1-8 bits per pixel (after compression, this is the shakiest part of my estimate!), that's around 50 million pixels per sheet, somewhere around 6-50MB per sheet. So somewhere between 1/2TB to 4TB for the whole data set (and I could easily be too low or too high by a factor of 2).
  - jleader 12y ago
    Heh, looking at my neighborhood, the smallest available file is a scan of the 1933 7.5-minute Sierra Madre quad, 5.24MB, and the largest is the 1979 30x60 minute Los Angeles quad, 44.53MB. Most of the 7.5 minute quads seem to be in the 15-30MB range. So I don't feel so bad about my "shakey" estimate of individual map sizes!
    - _almosnow 12y ago
      Derailing the thread a little. "Always do the math" is a very nice piece of advice that I got from who knows where. More often than not you nail it.
- kornholi 12y ago
  A lot if you want the raw data. The 10-meter (also knows as 1/3 arcsecond) national elevation dataset is somewhere around 330GB in an unprojected form. There's also the 3-meter tiles for limited areas which take up 9 times as much storage. However, it doesn't stop there. As part of the 3DEP (3D Elevation Program)[1], Lidar/IfSAR data is being collected across CONUS. One of the products that will be available (hopefully next year) is a 1-meter DEM, which should roughly be around 35TB (100 times as much data compared to 10-meter). [1] http://pubs.usgs.gov/circ/1399/pdf/circ1399.pdf http://pubs.usgs.gov/circ/1399/pdf/circ1399.pdf
- jlarocco 12y ago
  Assuming you mean the raw data, I would guess it's a few petabytes. A while back I made a script to download NED elevation data from the USGS site. The files were elevation data for 1/3 arcsecond or 1 arcsecond squares and were available in a float format and/or ArcGIS. The files for a 1/3 arcsecond square in ArcGIS format were zipped up and came to ~320 Mb. The float format files are even larger. I don't remember exactly, but I think there are a few thousand of them for the entire United States. And that was just NED elevation data, not feature data like roads and trails or lakes, or aerial imagery or topo quadrangles. FWIW, the script is on GitHub, but doesn't work any more due to changes on the USGS website. It actually uses PyQt's webview to do the downloads because the download process used a ton of Javascript with callbacks and notifications because the data was in a tape library and had to be fetched and loaded before the download could begin. https://github.com/jl2/GIS-Stuff/blob/master/map_download/neddown.py https://github.com/jl2/GIS-Stuff/blob/master/map_download/ne...

This isn't new. Also, some data sets require sending a hard drive into the USGS I believe.

RIMR 12y ago

Let's say I wanted all of it. How much data is that, or would I have to ask USGS?

jleader 12y ago
Drawing a rough box on a map of the US I estimated about 40 degrees of longitude by 20 degrees of latitude. The most detailed maps are the 7.5-minute quadrangles, which I believe are available for nearly all of the continental US. Since 7.5 minutes is 1/8 of a degree, that gives 40208*8, or a bit more than 50,000 maps. A quick browse through the USGS site found this quote (http://store.usgs.gov/b2c_usgs/usgs/zproductinformation/%28xcm=r3standardpitrex_prd&layout=6_1_61_50_2&uiarea=2&ctype=areaDetails&carea=0000000009%29/.do#75_15Minute http://store.usgs.gov/b2c_usgs/usgs/zproductinformation/%28x...): "It takes about 57,000 maps to cover the conterminous 48 States, Hawaii, and territories." They go on to mention that Alaska's most detailed coverage is the 15-minute quadrangle series. A similar estimate for the 15-minute coverage of Alaska is around 3000 maps, plus 1/4 of 57K (14K) for the 15-minute maps of the area already covered by the 7.5-minute series, plus a few thousand for the smaller-scale series (30x60 minutes, 1x2 degrees, index maps, etc.) So I'd guess somewhere around 75-85 thousand maps. The majority of these maps were originally intended to be printed on 23x27 inch sheets of paper. Assuming they were printed at around 300 dpi, somewhere between 1-8 bits per pixel (after compression, this is the shakiest part of my estimate!), that's around 50 million pixels per sheet, somewhere around 6-50MB per sheet. So somewhere between 1/2TB to 4TB for the whole data set (and I could easily be too low or too high by a factor of 2).
- jleader 12y ago
  Heh, looking at my neighborhood, the smallest available file is a scan of the 1933 7.5-minute Sierra Madre quad, 5.24MB, and the largest is the 1979 30x60 minute Los Angeles quad, 44.53MB. Most of the 7.5 minute quads seem to be in the 15-30MB range. So I don't feel so bad about my "shakey" estimate of individual map sizes!
  - _almosnow 12y ago
    Derailing the thread a little. "Always do the math" is a very nice piece of advice that I got from who knows where. More often than not you nail it.
kornholi 12y ago
A lot if you want the raw data. The 10-meter (also knows as 1/3 arcsecond) national elevation dataset is somewhere around 330GB in an unprojected form. There's also the 3-meter tiles for limited areas which take up 9 times as much storage. However, it doesn't stop there. As part of the 3DEP (3D Elevation Program)[1], Lidar/IfSAR data is being collected across CONUS. One of the products that will be available (hopefully next year) is a 1-meter DEM, which should roughly be around 35TB (100 times as much data compared to 10-meter). [1] http://pubs.usgs.gov/circ/1399/pdf/circ1399.pdf http://pubs.usgs.gov/circ/1399/pdf/circ1399.pdf
jlarocco 12y ago
Assuming you mean the raw data, I would guess it's a few petabytes. A while back I made a script to download NED elevation data from the USGS site. The files were elevation data for 1/3 arcsecond or 1 arcsecond squares and were available in a float format and/or ArcGIS. The files for a 1/3 arcsecond square in ArcGIS format were zipped up and came to ~320 Mb. The float format files are even larger. I don't remember exactly, but I think there are a few thousand of them for the entire United States. And that was just NED elevation data, not feature data like roads and trails or lakes, or aerial imagery or topo quadrangles. FWIW, the script is on GitHub, but doesn't work any more due to changes on the USGS website. It actually uses PyQt's webview to do the downloads because the download process used a ton of Javascript with callbacks and notifications because the data was in a tape library and had to be fetched and loaded before the download could begin. https://github.com/jl2/GIS-Stuff/blob/master/map_download/neddown.py https://github.com/jl2/GIS-Stuff/blob/master/map_download/ne...