4 ms·

One reason to use binary formats like HDF5 is to avoid precision loss when storing floating-point values. I started using HDF5 once exactly for this reason and

One reason to use binary formats like HDF5 is to avoid precision loss when storing floating-point values. I started using HDF5 once exactly for this reason and it was overkill. HDFView requires Java installed and HDF library with single implementation and complex API is a problem too. For simple uses I now use '%a' printf specifier. It is specifically designed to avoid losing a single bit of informaiton. And you can easily read floats stored this way in numpy by using genfromtxt with optional converters= argument and builtin float.fromhex function.

bhouston 11y ago
You can still use binary formats without HDF5, just write the memory buffer of floats to disk directly skipping text format. Any save/load system that uses printf/scanf will be brutually slow (at least 10x slower than just writing/reading the memory buffers), as well as space inefficient.
- ycmbntrthrwaway 11y ago
  The problem with binary storage is endianess. If you want portable format, you either have to use text, specify endianess in the format specification or store endianess in the file. For highest speed you need to convert data to your endianess before first use, update information about endianess inside the file and then work with memory-mapped file.
  - bhouston 11y ago
    Endianness generally is fast to convert, just pick a standard for the file format and detect on the platform. You can convert endianness with just shifts in C if you need to or with assembly instructions that I believe are one tick on most platforms. It is much much faster to convert endianness (again at least 10x faster) than to parse text. It looks like you can just use these functions to convert between endianness conditionally based on the platform if you are in C/C++: http://stackoverflow.com/a/8671129 http://stackoverflow.com/a/8671129 And boost has one here: http://www.boost.org/doc/libs/1_58_0/libs/endian/doc/index.html http://www.boost.org/doc/libs/1_58_0/libs/endian/doc/index.h...
    - ycmbntrthrwaway 11y ago
      It is right that converting endianness is faster than reading text, but it still requires more code than calling printf and scanf. I am not into big data really and my program spends most of the time computing instead of reading and writing so 10x speedup in I/O is just not worth it. As I said, HDF5 was an overkill in my case.
      - mturmon 11y ago
        The OP, and any other file format spec, needs to be able to read/write large numeric datasets. Printf is very poorly suited to this case. Your comment is true for your use case, but it's not really responsive to the technical issue here.
      - Dylan16807 11y ago
        for (i=start; i<end; i++) arr[i] = ntohl(arr[i]); That's not more code than calling printf/scanf.
  - gaze 11y ago
    What big endian platforms do you have to support? Got an SGI Indigo somewhere in your lab?
    - jbandela1 11y ago
      Exactly. All the major processors are little endian now. Just standardize on little endian. If you are working on some esoteric platform that is big endian you will know it
  - DiThi 11y ago
    IEEE 754 floats don't have different endianess. And as the other comment says, swapping endianess is fast (if ever needed, since most platforms are little endian or hybrid).
    - ycmbntrthrwaway 11y ago
      > IEEE 754 floats don't have different endianess. The standard does not specify endianness [1]. From the standard point of view float is a sequence of bits. But if you memory-map the file and store float on little-endian machine, you get bytes reversed compared to how it is written in the standard. On big-endian machine sign is stored in the first byte, and on little-endian machine sign is stored in the last byte. [1] https://en.wikipedia.org/wiki/Endianness#Floating-point https://en.wikipedia.org/wiki/Endianness#Floating-point
      - DiThi 11y ago
        Damn, you're right... therefore I have no idea how a simple float quantization algorithm I did could even work.
iheartmemcache 11y ago
It's also really useful if you have a lot of numerical data streaming in that you want to store and use at a later date. CERN results I'm sure use something similar to HDF5, nearly all of HFT algo trading uses HDF5 for securities they are going to explore down the road but don't want to waste KDB+ licenses on, Google File System's chunking scheme seems to be somewhat similar to it as well. _"Third, most files are mutated by appending new data rather than overwriting existing data. ... Once written, the files are only read, and often only sequentially."_ [1] _That_ is the use case for HDF5. The problem is this guy tried to slam a circular peg into a square hole. I'm in no way an apologist for HDF5 but his complaints are terribly vague. "Limited support for parallel access" Then you go read the source[2] and and see GIL complaints abound. And again, this was meant for an append-only situation where you shouldn't even have to acquire a lock in the first place since there's no contention possibility! "Impossibility to explore datasets with standard Unix/Windows tools" right, but there are plenty of Java tools that perform quite well, even with a cold JVM. "Opacity of the development and slow reactivity of the development team." AFAIK it's an open-source project, this complaint is valid if you're paying a vendor fees for a product and have a support plan with an SLA, and it's not valid in the least otherwise. "High risks of data corruption" I've never once seen this happen when HDF5 was properly used, though I'd love to see a pdb dump of the state his program when that occurred. Open offer - I'll fix that bug if it's a fault with the C lib you're FFI'ing with. edit: oh, the Java tooling was already mentioned. [1] http://static.googleusercontent.com/media/research.google.com/en//archive/gfs-sosp2003.pdf http://static.googleusercontent.com/media/research.google.co... [2] https://github.com/h5py/h5py/blob/master/h5py/_locks.pxi https://github.com/h5py/h5py/blob/master/h5py/_locks.pxi
- batbomb 11y ago
  CERN uses ROOT.
  - iheartmemcache 11y ago
    Huh! Good to know. ftp://root.cern.ch/root/doc/11InputOutput.pdf[1] The spec for anyone interested. For comparison:https://www.hdfgroup.org/projects/hdf5_aip/aip15.gif https://www.hdfgroup.org/projects/hdf5_aip/aip15.gif to page 6 on CERN's PDF.
phunge 11y ago
Man, if I had a dollar for every time data roundtripping to files has corrupted things I'd have many dollars. The schema-based serialization libraries (Thrift, Protobufs) or MsgPack are a good way to avoid that too. They come with a lot less baggage than say HDF5. Also, if efficiency isn't paramount -- just use SQLite! Amazing tool when it's in its sweetspot. Lots of tradeoffs when dealing w/ serialization and file formats, no easy answers.
ipunchghosts 11y ago
FYI, HDFView is not the only viewer in town. its also really easy to write a viewer in python using matplotlib.
microcolonel 11y ago
Can't think of a common programming language which doesn't support saving bit-accurate copies of floats in a contiguous buffer. Even in JavaScript you can put your doubles in a Float64Array and convert that to or from a buffer. If you want headers, you can define those as text and just use extents to embed the binary data. Tar is another alternative, pretty easy to implement.
simonbyrne 11y ago
Hex-formatted floats have the further advantage of being extremely fast to print and parse (compared with decimals): I've seen 3x speed ups for io operations. They really should be more widely used
- deleted 11y ago
  [deleted]
hzhou321 11y ago
I deal with binary data all the time and I just use fwrite in C and pack in Perl, and a text readme in vim explaining the (simple) format. I don't use Matlab but often need exchange the data with Matlab user; I simply search help page and send the one or two lines of matlab code to them along with the data. I'll do the same thing if my fellow colleague switch to numpy tomorrow (it is surprising that they would read HDF5 tutorial but won't read my readme). Research data are almost always specific. The idea of a general format serving all research just sounds silly to me.

bhouston 11y ago

You can still use binary formats without HDF5, just write the memory buffer of floats to disk directly skipping text format. Any save/load system that uses printf/scanf will be brutually slow (at least 10x slower than just writing/reading the memory buffers), as well as space inefficient.

ycmbntrthrwaway 11y ago
The problem with binary storage is endianess. If you want portable format, you either have to use text, specify endianess in the format specification or store endianess in the file. For highest speed you need to convert data to your endianess before first use, update information about endianess inside the file and then work with memory-mapped file.
- bhouston 11y ago
  Endianness generally is fast to convert, just pick a standard for the file format and detect on the platform. You can convert endianness with just shifts in C if you need to or with assembly instructions that I believe are one tick on most platforms. It is much much faster to convert endianness (again at least 10x faster) than to parse text. It looks like you can just use these functions to convert between endianness conditionally based on the platform if you are in C/C++: http://stackoverflow.com/a/8671129 http://stackoverflow.com/a/8671129 And boost has one here: http://www.boost.org/doc/libs/1_58_0/libs/endian/doc/index.html http://www.boost.org/doc/libs/1_58_0/libs/endian/doc/index.h...
  - ycmbntrthrwaway 11y ago
    It is right that converting endianness is faster than reading text, but it still requires more code than calling printf and scanf. I am not into big data really and my program spends most of the time computing instead of reading and writing so 10x speedup in I/O is just not worth it. As I said, HDF5 was an overkill in my case.
    - mturmon 11y ago
      The OP, and any other file format spec, needs to be able to read/write large numeric datasets. Printf is very poorly suited to this case. Your comment is true for your use case, but it's not really responsive to the technical issue here.
    - Dylan16807 11y ago
      for (i=start; i<end; i++) arr[i] = ntohl(arr[i]); That's not more code than calling printf/scanf.
- gaze 11y ago
  What big endian platforms do you have to support? Got an SGI Indigo somewhere in your lab?
  - jbandela1 11y ago
    Exactly. All the major processors are little endian now. Just standardize on little endian. If you are working on some esoteric platform that is big endian you will know it
- DiThi 11y ago
  IEEE 754 floats don't have different endianess. And as the other comment says, swapping endianess is fast (if ever needed, since most platforms are little endian or hybrid).
  - ycmbntrthrwaway 11y ago
    > IEEE 754 floats don't have different endianess. The standard does not specify endianness [1]. From the standard point of view float is a sequence of bits. But if you memory-map the file and store float on little-endian machine, you get bytes reversed compared to how it is written in the standard. On big-endian machine sign is stored in the first byte, and on little-endian machine sign is stored in the last byte. [1] https://en.wikipedia.org/wiki/Endianness#Floating-point https://en.wikipedia.org/wiki/Endianness#Floating-point
    - DiThi 11y ago
      Damn, you're right... therefore I have no idea how a simple float quantization algorithm I did could even work.

iheartmemcache 11y ago

It's also really useful if you have a lot of numerical data streaming in that you want to store and use at a later date. CERN results I'm sure use something similar to HDF5, nearly all of HFT algo trading uses HDF5 for securities they are going to explore down the road but don't want to waste KDB+ licenses on, Google File System's chunking scheme seems to be somewhat similar to it as well. _"Third, most files are mutated by appending new data rather than overwriting existing data. ... Once written, the files are only read, and often only sequentially."_ [1] _That_ is the use case for HDF5. The problem is this guy tried to slam a circular peg into a square hole. I'm in no way an apologist for HDF5 but his complaints are terribly vague. "Limited support for parallel access" Then you go read the source[2] and and see GIL complaints abound. And again, this was meant for an append-only situation where you shouldn't even have to acquire a lock in the first place since there's no contention possibility! "Impossibility to explore datasets with standard Unix/Windows tools" right, but there are plenty of Java tools that perform quite well, even with a cold JVM. "Opacity of the development and slow reactivity of the development team." AFAIK it's an open-source project, this complaint is valid if you're paying a vendor fees for a product and have a support plan with an SLA, and it's not valid in the least otherwise. "High risks of data corruption" I've never once seen this happen when HDF5 was properly used, though I'd love to see a pdb dump of the state his program when that occurred. Open offer - I'll fix that bug if it's a fault with the C lib you're FFI'ing with. edit: oh, the Java tooling was already mentioned. [1] http://static.googleusercontent.com/media/research.google.com/en//archive/gfs-sosp2003.pdf http://static.googleusercontent.com/media/research.google.co... [2] https://github.com/h5py/h5py/blob/master/h5py/_locks.pxi https://github.com/h5py/h5py/blob/master/h5py/_locks.pxi

batbomb 11y ago
CERN uses ROOT.
- iheartmemcache 11y ago
  Huh! Good to know. ftp://root.cern.ch/root/doc/11InputOutput.pdf[1] The spec for anyone interested. For comparison:https://www.hdfgroup.org/projects/hdf5_aip/aip15.gif https://www.hdfgroup.org/projects/hdf5_aip/aip15.gif to page 6 on CERN's PDF.

phunge 11y ago

Man, if I had a dollar for every time data roundtripping to files has corrupted things I'd have many dollars. The schema-based serialization libraries (Thrift, Protobufs) or MsgPack are a good way to avoid that too. They come with a lot less baggage than say HDF5. Also, if efficiency isn't paramount -- just use SQLite! Amazing tool when it's in its sweetspot. Lots of tradeoffs when dealing w/ serialization and file formats, no easy answers.

ipunchghosts 11y ago

FYI, HDFView is not the only viewer in town. its also really easy to write a viewer in python using matplotlib.

microcolonel 11y ago

Can't think of a common programming language which doesn't support saving bit-accurate copies of floats in a contiguous buffer. Even in JavaScript you can put your doubles in a Float64Array and convert that to or from a buffer. If you want headers, you can define those as text and just use extents to embed the binary data. Tar is another alternative, pretty easy to implement.

simonbyrne 11y ago

Hex-formatted floats have the further advantage of being extremely fast to print and parse (compared with decimals): I've seen 3x speed ups for io operations. They really should be more widely used

deleted 11y ago
[deleted]

hzhou321 11y ago

I deal with binary data all the time and I just use fwrite in C and pack in Perl, and a text readme in vim explaining the (simple) format. I don't use Matlab but often need exchange the data with Matlab user; I simply search help page and send the one or two lines of matlab code to them along with the data. I'll do the same thing if my fellow colleague switch to numpy tomorrow (it is surprising that they would read HDF5 tutorial but won't read my readme). Research data are almost always specific. The idea of a general format serving all research just sounds silly to me.