4 ms·

The problem with binary storage is endianess. If you want portable format, you either have to use text, specify endianess in the format specification or store e

The problem with binary storage is endianess. If you want portable format, you either have to use text, specify endianess in the format specification or store endianess in the file. For highest speed you need to convert data to your endianess before first use, update information about endianess inside the file and then work with memory-mapped file.

bhouston 11y ago
Endianness generally is fast to convert, just pick a standard for the file format and detect on the platform. You can convert endianness with just shifts in C if you need to or with assembly instructions that I believe are one tick on most platforms. It is much much faster to convert endianness (again at least 10x faster) than to parse text. It looks like you can just use these functions to convert between endianness conditionally based on the platform if you are in C/C++: http://stackoverflow.com/a/8671129 http://stackoverflow.com/a/8671129 And boost has one here: http://www.boost.org/doc/libs/1_58_0/libs/endian/doc/index.html http://www.boost.org/doc/libs/1_58_0/libs/endian/doc/index.h...
- ycmbntrthrwaway 11y ago
  It is right that converting endianness is faster than reading text, but it still requires more code than calling printf and scanf. I am not into big data really and my program spends most of the time computing instead of reading and writing so 10x speedup in I/O is just not worth it. As I said, HDF5 was an overkill in my case.
  - mturmon 11y ago
    The OP, and any other file format spec, needs to be able to read/write large numeric datasets. Printf is very poorly suited to this case. Your comment is true for your use case, but it's not really responsive to the technical issue here.
  - Dylan16807 11y ago
    for (i=start; i<end; i++) arr[i] = ntohl(arr[i]); That's not more code than calling printf/scanf.
gaze 11y ago
What big endian platforms do you have to support? Got an SGI Indigo somewhere in your lab?
- jbandela1 11y ago
  Exactly. All the major processors are little endian now. Just standardize on little endian. If you are working on some esoteric platform that is big endian you will know it
DiThi 11y ago
IEEE 754 floats don't have different endianess. And as the other comment says, swapping endianess is fast (if ever needed, since most platforms are little endian or hybrid).
- ycmbntrthrwaway 11y ago
  > IEEE 754 floats don't have different endianess. The standard does not specify endianness [1]. From the standard point of view float is a sequence of bits. But if you memory-map the file and store float on little-endian machine, you get bytes reversed compared to how it is written in the standard. On big-endian machine sign is stored in the first byte, and on little-endian machine sign is stored in the last byte. [1] https://en.wikipedia.org/wiki/Endianness#Floating-point https://en.wikipedia.org/wiki/Endianness#Floating-point
  - DiThi 11y ago
    Damn, you're right... therefore I have no idea how a simple float quantization algorithm I did could even work.

bhouston 11y ago

Endianness generally is fast to convert, just pick a standard for the file format and detect on the platform. You can convert endianness with just shifts in C if you need to or with assembly instructions that I believe are one tick on most platforms. It is much much faster to convert endianness (again at least 10x faster) than to parse text. It looks like you can just use these functions to convert between endianness conditionally based on the platform if you are in C/C++: http://stackoverflow.com/a/8671129 http://stackoverflow.com/a/8671129 And boost has one here: http://www.boost.org/doc/libs/1_58_0/libs/endian/doc/index.html http://www.boost.org/doc/libs/1_58_0/libs/endian/doc/index.h...

ycmbntrthrwaway 11y ago
It is right that converting endianness is faster than reading text, but it still requires more code than calling printf and scanf. I am not into big data really and my program spends most of the time computing instead of reading and writing so 10x speedup in I/O is just not worth it. As I said, HDF5 was an overkill in my case.
- mturmon 11y ago
  The OP, and any other file format spec, needs to be able to read/write large numeric datasets. Printf is very poorly suited to this case. Your comment is true for your use case, but it's not really responsive to the technical issue here.
- Dylan16807 11y ago
  for (i=start; i<end; i++) arr[i] = ntohl(arr[i]); That's not more code than calling printf/scanf.

gaze 11y ago

What big endian platforms do you have to support? Got an SGI Indigo somewhere in your lab?

jbandela1 11y ago
Exactly. All the major processors are little endian now. Just standardize on little endian. If you are working on some esoteric platform that is big endian you will know it

DiThi 11y ago

IEEE 754 floats don't have different endianess. And as the other comment says, swapping endianess is fast (if ever needed, since most platforms are little endian or hybrid).

ycmbntrthrwaway 11y ago
> IEEE 754 floats don't have different endianess. The standard does not specify endianness [1]. From the standard point of view float is a sequence of bits. But if you memory-map the file and store float on little-endian machine, you get bytes reversed compared to how it is written in the standard. On big-endian machine sign is stored in the first byte, and on little-endian machine sign is stored in the last byte. [1] https://en.wikipedia.org/wiki/Endianness#Floating-point https://en.wikipedia.org/wiki/Endianness#Floating-point
- DiThi 11y ago
  Damn, you're right... therefore I have no idea how a simple float quantization algorithm I did could even work.