3 ms·

I found this really interesting. I was curious about the Japanese realtime sea-floor sensors they mentioned so I went hunting. The data appears to be available

I found this really interesting. I was curious about the Japanese realtime sea-floor sensors they mentioned so I went hunting. The data appears to be available here: http://www.jamstec.go.jp/scdc/top_e.html http://www.jamstec.go.jp/scdc/top_e.html After registration [1] you can download data in some funky binary format. Looks like there's a 10min delay on data. It looks like an interesting dataset, and I hope to play with it a bit. [1] includes such joys as password being set to username, and the seeming inability to change your password citing "invalid chars: 1" error.

eriknstr 9y ago
Did you find any documentation about the data format?
- comstock 9y ago
  Yes, it's documented. Doesn't look to bad. They appear to do some custom delta compression but nothing too bad.
angrygoat 9y ago
I used to work at an exploration geophysics firm, these data formats less scary than you'd often encounter in that world. There are a number of interchange formats in geophysics, notably SEG-Y, which date back a long way – most vendors then have their custom formats that they ingest that interchange format into: https://en.wikipedia.org/wiki/SEG_Y https://en.wikipedia.org/wiki/SEG_Y Fundamentally you've usually got one to three dimensional data, metadata for the entire file, and then metadata for (potentially) each column along some axis of the file. It's interesting that CDF, etc have had zero traction: I guess it's just that each field has its little data format subculture and there's not much incentive for collaboration?
- shiftpgdn 9y ago
  I currently work at an exploration geophysics firm and was just talking to my boss about this yesterday. His comment was that data acquisition had an arms race to the bottom in the late 80s and has become complete commodity object. This has results in data from the 80s through today being no different as they're still using the (mostly) same tools and techniques. There is no incentive to better your seismic surveying since stuff from the 80s is "good enough" for all the processing shops.
  - angrygoat 9y ago
    Yep, that sounds about right, at least on the data formats. A survey now is gathering orders of magnitude more data than it would have in the 80s (especially 3d/4d surveys), but I guess the formats didn't really need to change. I don't miss writing code to guess whether a file has traces encoded in IBM floating point or IEEE; or handling SEG-Y headers which ought to be EBCDIC but are sometimes ASCII :-)

eriknstr 9y ago

Did you find any documentation about the data format?

comstock 9y ago
Yes, it's documented. Doesn't look to bad. They appear to do some custom delta compression but nothing too bad.

angrygoat 9y ago

I used to work at an exploration geophysics firm, these data formats less scary than you'd often encounter in that world. There are a number of interchange formats in geophysics, notably SEG-Y, which date back a long way – most vendors then have their custom formats that they ingest that interchange format into: https://en.wikipedia.org/wiki/SEG_Y https://en.wikipedia.org/wiki/SEG_Y Fundamentally you've usually got one to three dimensional data, metadata for the entire file, and then metadata for (potentially) each column along some axis of the file. It's interesting that CDF, etc have had zero traction: I guess it's just that each field has its little data format subculture and there's not much incentive for collaboration?

shiftpgdn 9y ago
I currently work at an exploration geophysics firm and was just talking to my boss about this yesterday. His comment was that data acquisition had an arms race to the bottom in the late 80s and has become complete commodity object. This has results in data from the 80s through today being no different as they're still using the (mostly) same tools and techniques. There is no incentive to better your seismic surveying since stuff from the 80s is "good enough" for all the processing shops.
- angrygoat 9y ago
  Yep, that sounds about right, at least on the data formats. A survey now is gathering orders of magnitude more data than it would have in the 80s (especially 3d/4d surveys), but I guess the formats didn't really need to change. I don't miss writing code to guess whether a file has traces encoded in IBM floating point or IEEE; or handling SEG-Y headers which ought to be EBCDIC but are sometimes ASCII :-)