7 ms·

I find it fascinating how difficult it is to find geological data. The combined datasets of oil and mining companies plus government data has a huge amount of E

I find it fascinating how difficult it is to find geological data. The combined datasets of oil and mining companies plus government data has a huge amount of Earth mapped. And yet, this data is extremely hard to find in a computer-consmable way. Most of it is locked up in pdf or image scans of maps, or locked in proprietary MapInfo/Autodesk formats. It seems to me that a large dataset of all human knowledge of Earth would be massively valuable to humanity. Unfortunately, oil/mineral maps are a cornerstone of a lot of very powerful companies. So I don't think we'll see them any time soon. Organizing this data would also be a hell of an effort because the maps use different projections, are from a huge variety of times, and are often inconsistent (overlapping areas with different mineral deposit analyses). I suppose I can dream, though.

colek42 8y ago
Most US government maps are available in a single clearinghouse. https://nationalmap.gov/ https://nationalmap.gov/. State governments and counties also have websites with geological (need it for a septic permit) data and land plots. The data is just getting more open, which is awesome. It may be in different formats, but nothing a few lines of python and a PostGIS database can't handle.
- peterlk 8y ago
  This is definitely the right direction. But the thing I really want is a map of minerals, faults, and topography. I am finding topography, but not the other two. Perhaps I'm just not seeing them? Imagine how much better our earthquake predicting abilities might become if we could couple global seismic data with a full map of Earth's fault lines. I know thin is pretty unlikely, but humanity would be better off, and most importantly, it would be really cool (we could learn a lot of stuff)
  - close04 8y ago
    The reason some geological maps are almost impossible to find is because they are treated as a national safety item. In some countries the law forbids companies from making these available. So an oil company that created the map can only use it and in a pretty controlled fashion. More that that, even if the map can be used more freely in the country (by multiple parties) it may be illegal for the map to cross the border in any form. This also means electronic signals over the internet if they cross the border. I guess the reason some of them are getting out in the open is because they are no longer considered "critical". Disclaimer: My work intersected the oil and gas domain at some point.
    - samstave 8y ago
      >The reason some geological maps are almost impossible to find is because they are treated as a national safety item. In some countries the law forbids companies from making these available. So an oil company that created the map can only use it and in a pretty controlled fashion. /THE/ reason. Also, this is why some believe the war in Afghanistan was even waged - was to get the "boots on ground*" == on-top of the $1T in mineral deposits. While we kvetch constantly about stupid political distractions, there are, in fact, many government-corporate entities that are in the long-game. https://www.livescience.com/47682-rare-earth-minerals-found-under-afghanistan.html https://www.livescience.com/47682-rare-earth-minerals-found-... Look at even the completely obvious: https://www.amazon.com/Hundred-Year-Marathon-Strategy-Replace-Superpower/dp/1250081343#customerReviews https://www.amazon.com/Hundred-Year-Marathon-Strategy-Replac... and even as openly published ten years ago: http://www.dailymail.co.uk/news/article-1036105/How-Chinas-taking-Africa-West-VERY-worried.html http://www.dailymail.co.uk/news/article-1036105/How-Chinas-t... --- The point is that we are already in the long war for resource control. And sadly, I would say that the US is not in the upper-hand at this time, aside from building up the military...
      - sizzle 8y ago
        Who has the upper hand, China?
        deleted 8y ago
        [deleted]
        philliphaydon 8y ago
        By a long shot....
        baq 8y ago
        Yeah, the Chinese Africa is going to be an interesting war zone.
      - peterlk 8y ago
        I received the following (candid) comment from a geologist that I asked. He said that I could share his comment > I don't have time to write the essay on this, but we definitely DONT wage war to get "boots on the ground". That's absurd and if true why go to that effort when there is the rest of the world to but your boots on without war. That's one of the biggest problems. The Afghan Geo Survey was at PDAC pleading people to come work in their country but, I for one won't be going AND NO ONE is going to put money into that huge risk. Why would they with the rest of the world available to make a real map you have to have boots on the ground, (like Iv done for 40 years) not a computer with .tab and .shp files. That is a derivative product that comes after collection of real data that costs money and is proprietary intellectual capital. Nobody sees my map without a non disclosure agreement or confidentiality agreement. And that happens across the world for any private expenditure. Exxon, Total, Shell, etc possibly have the the most vast earth science databases created by mankind. They represent $trillions of investment and are not about to make that publicly available. Publicly available earth science information is created by, curated by, and made available through governments, =tax $. Thus, it is meager, everywhere.
  - cgdcraig 8y ago
    That data is the secret sauce for mining and oil and gas companies. Them giving it away would be like Facebook giving away their datasets.
neuromantik8086 8y ago
> I find it fascinating how difficult it is to find geological data. Discoverability for openly released scientific datasets is a huge problem in general. While some enterprising folks have worked on adding parsers for scientific data formats such NetCDF and HDF5 to Apache Tika (which can then be indexed by Solr/Elasticsearch/whatever) [0], the vast majority of scientific file formats don't have parsers available. Even worse, in the climate of publish-or-perish, most scientists are unaware of or less likely to prioritize the incorporation of metadata extraction / indexing tools, even though these would make their data more readily searchable based on relevant metadata (such as equipment settings, etc). I have some personal experience in this area- when I was working as a research assistant, I basically did helpdesk support for an open access dataset, answering questions from researchers at other institutions. I'd estimate that of the questions I received in my inbox, close to 70% could have been resolved with a good implementation of faceted search. A related issue I encountered is that rather than relevant metadata existing alongside a dataset, sometimes I'd have to dive into an article's methods section to find it, often in a weird place that wasn't obvious at first glance due to the obtuse writing style that is encouraged for scientific publications. The bigger problem, however, is that the culture of science in academia right now puts way too much emphasis on flashiness over sustainability and admittedly non-sexy tasks like properly versioning and packaging scientific software, documenting analyses, and producing well-characterized datasets. [0] https://www.slideshare.net/chrismattmann/scientific-data-curation-and-processing-with-apache-tika https://www.slideshare.net/chrismattmann/scientific-data-cur...
sien 8y ago
The Australian government has a huge amount of data available in computer consumable ways. To search the catalog from GA go to: https://ecat.ga.gov.au/geonetwork/srv/eng/catalog.search#/search?resultType=details&from=1&to=20&sortBy=changeDateDesc https://ecat.ga.gov.au/geonetwork/srv/eng/catalog.search#/se... There is an emphasis on open formats and using open source software. GA data is also extensively used by the Australian National Map at: https://nationalmap.gov.au/ https://nationalmap.gov.au/ Also GA is available from data.gov.au https://data.gov.au/dataset?tags=Earth+Sciences https://data.gov.au/dataset?tags=Earth+Sciences
- chii 8y ago
  Wow, didn't know this exists! Thanks for the links!
sbr464 8y ago
It's not up currently, but make a note about a domain for a project I'm working on and releasing soon. It's a general platform for accumulating any type of geo/metadata/media possible about a point in space. thislocation.com I mentioned a few details about it in this comment https://news.ycombinator.com/item?id=17532101 https://news.ycombinator.com/item?id=17532101
- foobarbecue 8y ago
  Sort of like Wikimapia?
  - sbr464 8y ago
    In concept, but not execution. More scientific and the platform doing a lot of work for you. You upload tons of specific raw data. Think lidar, drone footage, data from core samples, raw precision gps data, material analysis, etc. The platform understands a lot of things already and is able to organize/chart/map/visualize automatically, and extensible where not.
voltagex_ 8y ago
In Australia, I'm pretty sure mineral maps are what's funding Geoscience Australia, which is pretty much the reason we have digital maps at all. As for data being "locked away" in formats, well, no one's really come up with the One True Geospatial format, so you've got TAB and SHP. I prefer TAB, because I hate column name limitations. Use QGIS [1] or GDAL [2] to get the data into a format you want, or for really heavy lifting, apply for a home use licence for FME Desktop [3] (which uses GDAL among other things). TL;DR: Mineral maps are worth cash money, and that's why you can't get at them. 1: https://qgis.org/en/site/ https://qgis.org/en/site/ - can a benevolent billionaire please throw a couple of million dollars at this thing? 2: https://www.gdal.org/ https://www.gdal.org/ - GDAL makes the world go round (or ellipsoid, depending on projection) 3: https://www.safe.com/free-fme-licenses/home-use/ https://www.safe.com/free-fme-licenses/home-use/
peterlk 8y ago
A comment from a geologist who I asked about this (he said I could share on his behalf): > Faults of the world that effect mankind are mostly known and well mapped at the surface and below the surface through assorted geophysical means. We can't see there otherwise. Google Earth has made a tremendous contribution to mankind and there are many efforts to make Google geo-earth at least in small chunks. Universities are the best homes for this type of effort. Quality of data is a huge factor. The location of the San Andreas Fault is very well understood. Yet 20 years ago, there were NO geologic maps of even rudimentary quality for the San Francisco Bay area. Maps yes. Useful, printed or digital No. I doubt that has changed. Reason? Politics, funding, right people to do the job, etc BART and bridge routes are well studied for engineering purposes and should be public information, but try find a geology map derived from that suitable for public use. When I work in foreign countries and even in the US, there are regional maps, say 1:1,000,000. But that offers very little value. Now I am writing the book on the geology where I am working. I am the first one there to figure out what I am looking at and the potential value for any mineral commodity. You can't do what I do from a computer or satellite. You HAVE to be boots on the ground. The way forward for advancing geologic understanding is quality mappers on the ground knowing what they are looking at, translating that to useful information that conveys both facts and interpretations and then that to digital. We are a looooong loooong way from having geology Google Earth, but if we did, it would be great for mankind. Raise taxes?

colek42 8y ago

Most US government maps are available in a single clearinghouse. https://nationalmap.gov/ https://nationalmap.gov/. State governments and counties also have websites with geological (need it for a septic permit) data and land plots. The data is just getting more open, which is awesome. It may be in different formats, but nothing a few lines of python and a PostGIS database can't handle.

peterlk 8y ago
This is definitely the right direction. But the thing I really want is a map of minerals, faults, and topography. I am finding topography, but not the other two. Perhaps I'm just not seeing them? Imagine how much better our earthquake predicting abilities might become if we could couple global seismic data with a full map of Earth's fault lines. I know thin is pretty unlikely, but humanity would be better off, and most importantly, it would be really cool (we could learn a lot of stuff)
- close04 8y ago
  The reason some geological maps are almost impossible to find is because they are treated as a national safety item. In some countries the law forbids companies from making these available. So an oil company that created the map can only use it and in a pretty controlled fashion. More that that, even if the map can be used more freely in the country (by multiple parties) it may be illegal for the map to cross the border in any form. This also means electronic signals over the internet if they cross the border. I guess the reason some of them are getting out in the open is because they are no longer considered "critical". Disclaimer: My work intersected the oil and gas domain at some point.
  - samstave 8y ago
    >The reason some geological maps are almost impossible to find is because they are treated as a national safety item. In some countries the law forbids companies from making these available. So an oil company that created the map can only use it and in a pretty controlled fashion. /THE/ reason. Also, this is why some believe the war in Afghanistan was even waged - was to get the "boots on ground*" == on-top of the $1T in mineral deposits. While we kvetch constantly about stupid political distractions, there are, in fact, many government-corporate entities that are in the long-game. https://www.livescience.com/47682-rare-earth-minerals-found-under-afghanistan.html https://www.livescience.com/47682-rare-earth-minerals-found-... Look at even the completely obvious: https://www.amazon.com/Hundred-Year-Marathon-Strategy-Replace-Superpower/dp/1250081343#customerReviews https://www.amazon.com/Hundred-Year-Marathon-Strategy-Replac... and even as openly published ten years ago: http://www.dailymail.co.uk/news/article-1036105/How-Chinas-taking-Africa-West-VERY-worried.html http://www.dailymail.co.uk/news/article-1036105/How-Chinas-t... --- The point is that we are already in the long war for resource control. And sadly, I would say that the US is not in the upper-hand at this time, aside from building up the military...
    - sizzle 8y ago
      Who has the upper hand, China?
      - deleted 8y ago
        [deleted]
      - philliphaydon 8y ago
        By a long shot....
      - baq 8y ago
        Yeah, the Chinese Africa is going to be an interesting war zone.
    - peterlk 8y ago
      I received the following (candid) comment from a geologist that I asked. He said that I could share his comment > I don't have time to write the essay on this, but we definitely DONT wage war to get "boots on the ground". That's absurd and if true why go to that effort when there is the rest of the world to but your boots on without war. That's one of the biggest problems. The Afghan Geo Survey was at PDAC pleading people to come work in their country but, I for one won't be going AND NO ONE is going to put money into that huge risk. Why would they with the rest of the world available to make a real map you have to have boots on the ground, (like Iv done for 40 years) not a computer with .tab and .shp files. That is a derivative product that comes after collection of real data that costs money and is proprietary intellectual capital. Nobody sees my map without a non disclosure agreement or confidentiality agreement. And that happens across the world for any private expenditure. Exxon, Total, Shell, etc possibly have the the most vast earth science databases created by mankind. They represent $trillions of investment and are not about to make that publicly available. Publicly available earth science information is created by, curated by, and made available through governments, =tax $. Thus, it is meager, everywhere.
- cgdcraig 8y ago
  That data is the secret sauce for mining and oil and gas companies. Them giving it away would be like Facebook giving away their datasets.

neuromantik8086 8y ago

> I find it fascinating how difficult it is to find geological data. Discoverability for openly released scientific datasets is a huge problem in general. While some enterprising folks have worked on adding parsers for scientific data formats such NetCDF and HDF5 to Apache Tika (which can then be indexed by Solr/Elasticsearch/whatever) [0], the vast majority of scientific file formats don't have parsers available. Even worse, in the climate of publish-or-perish, most scientists are unaware of or less likely to prioritize the incorporation of metadata extraction / indexing tools, even though these would make their data more readily searchable based on relevant metadata (such as equipment settings, etc). I have some personal experience in this area- when I was working as a research assistant, I basically did helpdesk support for an open access dataset, answering questions from researchers at other institutions. I'd estimate that of the questions I received in my inbox, close to 70% could have been resolved with a good implementation of faceted search. A related issue I encountered is that rather than relevant metadata existing alongside a dataset, sometimes I'd have to dive into an article's methods section to find it, often in a weird place that wasn't obvious at first glance due to the obtuse writing style that is encouraged for scientific publications. The bigger problem, however, is that the culture of science in academia right now puts way too much emphasis on flashiness over sustainability and admittedly non-sexy tasks like properly versioning and packaging scientific software, documenting analyses, and producing well-characterized datasets. [0] https://www.slideshare.net/chrismattmann/scientific-data-curation-and-processing-with-apache-tika https://www.slideshare.net/chrismattmann/scientific-data-cur...

sien 8y ago

The Australian government has a huge amount of data available in computer consumable ways. To search the catalog from GA go to: https://ecat.ga.gov.au/geonetwork/srv/eng/catalog.search#/search?resultType=details&from=1&to=20&sortBy=changeDateDesc https://ecat.ga.gov.au/geonetwork/srv/eng/catalog.search#/se... There is an emphasis on open formats and using open source software. GA data is also extensively used by the Australian National Map at: https://nationalmap.gov.au/ https://nationalmap.gov.au/ Also GA is available from data.gov.au https://data.gov.au/dataset?tags=Earth+Sciences https://data.gov.au/dataset?tags=Earth+Sciences

chii 8y ago
Wow, didn't know this exists! Thanks for the links!

sbr464 8y ago

It's not up currently, but make a note about a domain for a project I'm working on and releasing soon. It's a general platform for accumulating any type of geo/metadata/media possible about a point in space. thislocation.com I mentioned a few details about it in this comment https://news.ycombinator.com/item?id=17532101 https://news.ycombinator.com/item?id=17532101

foobarbecue 8y ago
Sort of like Wikimapia?
- sbr464 8y ago
  In concept, but not execution. More scientific and the platform doing a lot of work for you. You upload tons of specific raw data. Think lidar, drone footage, data from core samples, raw precision gps data, material analysis, etc. The platform understands a lot of things already and is able to organize/chart/map/visualize automatically, and extensible where not.

voltagex_ 8y ago

In Australia, I'm pretty sure mineral maps are what's funding Geoscience Australia, which is pretty much the reason we have digital maps at all. As for data being "locked away" in formats, well, no one's really come up with the One True Geospatial format, so you've got TAB and SHP. I prefer TAB, because I hate column name limitations. Use QGIS [1] or GDAL [2] to get the data into a format you want, or for really heavy lifting, apply for a home use licence for FME Desktop [3] (which uses GDAL among other things). TL;DR: Mineral maps are worth cash money, and that's why you can't get at them. 1: https://qgis.org/en/site/ https://qgis.org/en/site/ - can a benevolent billionaire please throw a couple of million dollars at this thing? 2: https://www.gdal.org/ https://www.gdal.org/ - GDAL makes the world go round (or ellipsoid, depending on projection) 3: https://www.safe.com/free-fme-licenses/home-use/ https://www.safe.com/free-fme-licenses/home-use/

peterlk 8y ago

A comment from a geologist who I asked about this (he said I could share on his behalf): > Faults of the world that effect mankind are mostly known and well mapped at the surface and below the surface through assorted geophysical means. We can't see there otherwise. Google Earth has made a tremendous contribution to mankind and there are many efforts to make Google geo-earth at least in small chunks. Universities are the best homes for this type of effort. Quality of data is a huge factor. The location of the San Andreas Fault is very well understood. Yet 20 years ago, there were NO geologic maps of even rudimentary quality for the San Francisco Bay area. Maps yes. Useful, printed or digital No. I doubt that has changed. Reason? Politics, funding, right people to do the job, etc BART and bridge routes are well studied for engineering purposes and should be public information, but try find a geology map derived from that suitable for public use. When I work in foreign countries and even in the US, there are regional maps, say 1:1,000,000. But that offers very little value. Now I am writing the book on the geology where I am working. I am the first one there to figure out what I am looking at and the potential value for any mineral commodity. You can't do what I do from a computer or satellite. You HAVE to be boots on the ground. The way forward for advancing geologic understanding is quality mappers on the ground knowing what they are looking at, translating that to useful information that conveys both facts and interpretations and then that to digital. We are a looooong loooong way from having geology Google Earth, but if we did, it would be great for mankind. Raise taxes?