4 ms·

Learning About the GeoTIFF File Format with a Hex Editor (2018)

93 points by oldpatricka 5y ago

ivanhoe 5y ago
speaking of this, what hex editor would community recommend for examining (potentially quite large) binaries?
- redrobein 5y ago
  ImHex
- gspr 5y ago
  I haven't gotten my hands dirty with it myself yet, but Poke looks super interesting! https://www.jemarch.net/poke https://www.jemarch.net/poke https://media.ccc.de/v/ASG2019-127-gnu-poke-an-extensible-editor-for-structured-binary-data https://media.ccc.de/v/ASG2019-127-gnu-poke-an-extensible-ed... Actually, it sounds like a fun weekend task to redo this GeoTIFF writeup in Poke :-)
  - 1vuio0pswjnm7 5y ago
    I typed .doc without a TOPIC and poke segfaulted. Oh well. The hex editor I rely on for multi-GB files is ired.
- MontagFTB 5y ago
  On the Mac, I cannot recommend Hex Fiend enough: https://hexfiend.com/ https://hexfiend.com/
punnerud 5y ago
Could we build a static “Google Maps” using really big GeoTIFF files and downloading part of them with the section of a file functionality. Just like the SQLite example: https://news.ycombinator.com/item?id=27025829 https://news.ycombinator.com/item?id=27025829
- gregsadetsky 5y ago
  I think that you meant to link to https://news.ycombinator.com/item?id=27016630 https://news.ycombinator.com/item?id=27016630 instead which was the recently posted SQLite/SQL.js/HTTP range library And there was a specific sub-thread in the conversation that mentioned using this for maps too! https://news.ycombinator.com/item?id=27020372 https://news.ycombinator.com/item?id=27020372
  - punnerud 5y ago
    Thank you, a bit too fast there
- kylebarron 5y ago
  You could, you'd just have to be wary of how large the header metadata would be. If you had a single Cloud-Optimized GeoTIFF of the world with internal tiles up to zoom 14, the header metadata could be a few MB, which wouldn't be ideal to load in every client.
  - gregsadetsky 5y ago
    The posts on this topic on your site are really fascinating! https://kylebarron.dev/blog/cog-mosaic/overview https://kylebarron.dev/blog/cog-mosaic/overview https://kylebarron.dev/blog/cog-mosaic/naip https://kylebarron.dev/blog/cog-mosaic/naip
    - kylebarron 5y ago
      Thanks! The text is still relevant but those demos are a little old. If you're curious these demos are from last week (using Cloud-Optimized GeoTIFFs under the hood to serve image data quickly): - https://studio.unfolded.ai/public/9a2c5cd8-f0f0-43ce-b231-78bb6af905fb https://studio.unfolded.ai/public/9a2c5cd8-f0f0-43ce-b231-78... - https://studio.unfolded.ai/public/96c70224-dd89-431f-ae7a-0a56d9e2efb7 https://studio.unfolded.ai/public/96c70224-dd89-431f-ae7a-0a... - https://studio.unfolded.ai/public/af792fd8-1990-4cdc-bf2b-99e6a1ad3b45 https://studio.unfolded.ai/public/af792fd8-1990-4cdc-bf2b-99...
kylebarron 5y ago
This is a good hex editor overview, but also the part about Cloud-Optimized GeoTIFFs is quite good. In the past couple of years COGs have really revolutionized the satellite imagery industry. Even USGS has seen this is the future and publishes the official Landsat data in Cloud-Optimized GeoTIFF format [0]. I've been leveraging COGs recently to quickly bring satellite imagery into the browser for analysis [1]. [0]: https://www.usgs.gov/core-science-systems/nli/landsat/landsat-collection-2 https://www.usgs.gov/core-science-systems/nli/landsat/landsa... [1]: https://www.unfolded.ai/blog/2021-04-28-raster-layer/ https://www.unfolded.ai/blog/2021-04-28-raster-layer/
- alleycat5000 5y ago
  We use COGs at work with mapserver (and GDAL's /vsis3/ magic) to serve up imagery internally for qa/qc: https://github.com/pedros007/mapserver-docker https://github.com/pedros007/mapserver-docker Literally saves hundreds of thousands of dollars a year by avoiding having to stage the imagery locally before serving it out!
darekkay 5y ago
I recently did something similar with Foobar 2000's playlist format [1]. It was a fun learning experience :) [1] https://darekkay.com/blog/foobar2000-playlist-index-format/ https://darekkay.com/blog/foobar2000-playlist-index-format/
- oldpatricka 5y ago
  Ah wow this is great! Very clear explanation. I particularly liked that there are fields that you couldn’t identify but you were still able to get the information you needed!

93 points by oldpatricka 5y ago

ivanhoe 5y ago

speaking of this, what hex editor would community recommend for examining (potentially quite large) binaries?

redrobein 5y ago
ImHex
gspr 5y ago
I haven't gotten my hands dirty with it myself yet, but Poke looks super interesting! https://www.jemarch.net/poke https://www.jemarch.net/poke https://media.ccc.de/v/ASG2019-127-gnu-poke-an-extensible-editor-for-structured-binary-data https://media.ccc.de/v/ASG2019-127-gnu-poke-an-extensible-ed... Actually, it sounds like a fun weekend task to redo this GeoTIFF writeup in Poke :-)
- 1vuio0pswjnm7 5y ago
  I typed .doc without a TOPIC and poke segfaulted. Oh well. The hex editor I rely on for multi-GB files is ired.
MontagFTB 5y ago
On the Mac, I cannot recommend Hex Fiend enough: https://hexfiend.com/ https://hexfiend.com/

punnerud 5y ago

Could we build a static “Google Maps” using really big GeoTIFF files and downloading part of them with the section of a file functionality. Just like the SQLite example: https://news.ycombinator.com/item?id=27025829 https://news.ycombinator.com/item?id=27025829

gregsadetsky 5y ago
I think that you meant to link to https://news.ycombinator.com/item?id=27016630 https://news.ycombinator.com/item?id=27016630 instead which was the recently posted SQLite/SQL.js/HTTP range library And there was a specific sub-thread in the conversation that mentioned using this for maps too! https://news.ycombinator.com/item?id=27020372 https://news.ycombinator.com/item?id=27020372
- punnerud 5y ago
  Thank you, a bit too fast there
kylebarron 5y ago
You could, you'd just have to be wary of how large the header metadata would be. If you had a single Cloud-Optimized GeoTIFF of the world with internal tiles up to zoom 14, the header metadata could be a few MB, which wouldn't be ideal to load in every client.
- gregsadetsky 5y ago
  The posts on this topic on your site are really fascinating! https://kylebarron.dev/blog/cog-mosaic/overview https://kylebarron.dev/blog/cog-mosaic/overview https://kylebarron.dev/blog/cog-mosaic/naip https://kylebarron.dev/blog/cog-mosaic/naip
  - kylebarron 5y ago
    Thanks! The text is still relevant but those demos are a little old. If you're curious these demos are from last week (using Cloud-Optimized GeoTIFFs under the hood to serve image data quickly): - https://studio.unfolded.ai/public/9a2c5cd8-f0f0-43ce-b231-78bb6af905fb https://studio.unfolded.ai/public/9a2c5cd8-f0f0-43ce-b231-78... - https://studio.unfolded.ai/public/96c70224-dd89-431f-ae7a-0a56d9e2efb7 https://studio.unfolded.ai/public/96c70224-dd89-431f-ae7a-0a... - https://studio.unfolded.ai/public/af792fd8-1990-4cdc-bf2b-99e6a1ad3b45 https://studio.unfolded.ai/public/af792fd8-1990-4cdc-bf2b-99...

kylebarron 5y ago

This is a good hex editor overview, but also the part about Cloud-Optimized GeoTIFFs is quite good. In the past couple of years COGs have really revolutionized the satellite imagery industry. Even USGS has seen this is the future and publishes the official Landsat data in Cloud-Optimized GeoTIFF format [0]. I've been leveraging COGs recently to quickly bring satellite imagery into the browser for analysis [1]. [0]: https://www.usgs.gov/core-science-systems/nli/landsat/landsat-collection-2 https://www.usgs.gov/core-science-systems/nli/landsat/landsa... [1]: https://www.unfolded.ai/blog/2021-04-28-raster-layer/ https://www.unfolded.ai/blog/2021-04-28-raster-layer/

alleycat5000 5y ago
We use COGs at work with mapserver (and GDAL's /vsis3/ magic) to serve up imagery internally for qa/qc: https://github.com/pedros007/mapserver-docker https://github.com/pedros007/mapserver-docker Literally saves hundreds of thousands of dollars a year by avoiding having to stage the imagery locally before serving it out!

darekkay 5y ago

I recently did something similar with Foobar 2000's playlist format [1]. It was a fun learning experience :) [1] https://darekkay.com/blog/foobar2000-playlist-index-format/ https://darekkay.com/blog/foobar2000-playlist-index-format/

oldpatricka 5y ago
Ah wow this is great! Very clear explanation. I particularly liked that there are fields that you couldn’t identify but you were still able to get the information you needed!