6 ms·

Swapping bytes is a huge pain in the butt! When reading large binary files, it's so convenient and efficient to be able to mmap and make struct pointers right i

by tlb 9y ago

Swapping bytes is a huge pain in the butt! When reading large binary files, it's so convenient and efficient to be able to mmap and make struct pointers right into the file. You can even deliver those files to web browsers and use JS's TypedArray to get random access into them. (That requires a bit more than simply little-endian. It requires struct alignment and floating point formats to be the same. But with only a tiny bit of care it can be.) For new code running in-core in 2018, I think little-endian is quite safe.

api 9y ago
Yes, and I think it follows that new protocols and new file formats should be little-endian if possible.
bitwize 9y ago
> Swapping bytes is a huge pain in the butt! When reading large binary files, it's so convenient and efficient to be able to mmap and make struct pointers right into the file. No no no don't do this don't do this don't do this. This is how horrors and abominations like .doc, .xls, .psd happen. The correct way to handle binary data is to unpack it into the struct byte by byte. The reason for this is that when you define a struct in C(++), there are not just endianness issues, but implementation-dependent issues of padding and alignment you have to consider. Recently, most compilers on most architectures standardized on self-alignment rules for all primitive types except char, but this is not guaranteed by the standard, it will bite you in the ass when you least expect it, and it will be decades yet before all the C code in the world is displaced by a single-implementation language like Rust. The best way to write portable code that will work as intended is to treat all data coming from disk or over the wire as a bag of bytes, and not attempt to alias it to a struct.
- api 9y ago
  Most compilers have directives for packing structs. What the OP suggests is something that can be done for high performance, but don't expect it to be portable outside of common architectures and compilers. I'd consider it something that shouldn't be done unless you really need the simplicity or performance.
- IshKebab 9y ago
  No, dumping internal representations to disk is orthogonal to zero-static data formats. You can have a well defined format that doesn't require byte-by-byte parsing. Look up FlatBuffers and CapnProto.
  - bitwize 9y ago
    Both of those appear to work by deferring the parsing step to access time. You're still treating the thing as a bag of bytes and unpacking stuff out of it bytewise.
    - kentonv 9y ago
      That's incorrect. When you load an int64 field from a Cap'n Proto field, you are doing a 64-bit load instruction directly from the source bytes. You are not doing byte-by-byte access nor any sort of translation or "parsing". Cap'n Proto works by laying out data structures like a C compiler would, but following consistent, portable rules so that the layout is the same on all platforms. It then generates inline-able accessor functions to manipulate these structures which do pointer arithmetic similar to what a compiler would generate for accessing a struct. The end result is that accessing primitive fields from a Cap'n Proto struct is essentially identical in terms of machine instructions to accessing fields of a C struct. To call that "deferring the parsing to access time" does not make sense. (Note that for pointers, there are more differences: namely, because data will not always be loaded at the same address, pointers need to be relative rather than absolute. They also need to be bounds-checked for security. This adds a few instructions to pointer accesses, but those instructions look nothing like traditional "parsing" and certainly aren't byte-by-byte operations.) Your statement earlier: > The correct way to handle binary data is to unpack it into the struct byte by byte. This is inaccurate. Byte-by-byte parsing is a valid way to do parsing but not the only way. Byte-by-byte parsers tend to be slow and -- arguably, more importantly -- overly complex and rigid. It is, for example, usually very hard to do "random access" with a byte-by-byte parser, because allowing out-of-order parsing tends to blow the code complexity through the roof. On the other hand, with Cap'n Proto and similar approaches, you can trivially mmap() a very large data structure and traverse it randomly, and it "just works". (Disclosure: I'm the author of Cap'n Proto, as well as the author of the first open source release of Google's Protocol Buffers, which does byte-by-byte binary parsing.)
- planteen 9y ago
  I totally agree with you and so does Rob Pike: https://commandcenter.blogspot.com/2012/04/byte-order-fallacy.html https://commandcenter.blogspot.com/2012/04/byte-order-fallac...
- deleted 9y ago
  [deleted]
- pipio21 9y ago
  This makes no sense in lots of applications. Today computers have hardware acceleration for moving chunks of data. Big chucks of data, unpack things byte by byte makes not sense on things that have to be efficient. It is like 20.000 times more efficient to do things with blocks. If you move text around you probably don't care about efficiency and can do it. If you care about efficiency and price(using cheap components) it is a very bad solution. We use in all our serial-deserial a very fine grained controlled C library and it has been working flawlessly for years. This library is interfaced with C++, java, objC, swift,clojure, rust...
- Const-me 9y ago
  I have to disagree with your “No no no”. It all depends on requirements. When you’re on a PC, and working with files that you can reasonably expect will be exchanged (like doc, xls or pdf) — then your “No no no” heuristic is absolutely correct. As you pointed out, differences between compilers and between e.g. x86/amd64 are very likely to render these formats incompatible. But when you’re working with files that only your software will access, unpacking it byte by byte will slow down IO by a huge factor compared to both mmap, and read/write of large blocks (the latter will likely translate to DMA i.e. the CPU will be free to do something else). When that’s the case (like for most videogames, even for PC ones), you don’t want to pay that performance penalty, you want to get that data in the memory ASAP.
  - phkahler 9y ago
    >> It all depends on requirements. Requirements change.
    - Qwertious 9y ago
      Requirements can change, sometimes unexpectedly. But if you "know" for a fact that they won't, then doing the hypothetically more-portable thing at the expense of today's performance, is just plain overengineering.
      - jsjohnst 9y ago
        You never know something like that “as a fact”.
  - bitwize 9y ago
    > But when you’re working with files that only your software will access, unpacking it byte by byte will slow down IO by a huge factor compared to both mmap, and read/write of large blocks (the latter will likely translate to DMA i.e. the CPU will be free to do something else). If: * you're working with files that only your software will access AND * you control, and understand, the CPU architecture of any machine that will touch that data structure and you KNOW that it will never change for the entire lifetime of that piece of data AND * you always use the same version, or an ABI-compatible future version, of the same compiler that you KNOW will always lay out data the same way and you know what that way is AND * you either don't need to touch this data structure in other programming languages, or you KNOW that this won't be an issue (for example because your programming language implementation was written in C, compiled against the same version of the compiler, and has an FFI that understands C structs) THEN, you may proceed to mmap structs into memory. In practice these constraints are fairly commonplace; for example, NetBSD wscons device drivers report HID events which are specified in a struct, and because the HID events are likely never to leave the originating machine, only be passed from kernel space to user space, it makes sense to simply read(2) them straight into the struct. And there's certainly nothing wrong with snarfing a large file into a char[], but when it comes time to extract meaning from it, unless you are absolutely sure that the underlying assumptions regarding data layout will never ever change, it will only cause marginal harm to do everything in byte offsets and shift and OR bytes to yield final values -- and this is the best portable way to access the data therein, agnostic of details about the compiler and CPU architecture. In general it's a good idea to err on the side of caution by default, profile, and then optimize the hot paths as necessary bearing the constraints you're assuming in mind (and perhaps documenting them for good measure). Maybe my initial statement was too strong, but I shudder whenever I see comments of the form "It's so convenient to just mmap() that sucker into a struct and access the fields!" Because they've never had to deal with the consequences of trying to access a struct from a Microsoft compiler serialized to disk, and finding out that GNU compilers have a quite different notion of how structure members are to be laid out in memory...

by tlb 9y ago

api 9y ago

Yes, and I think it follows that new protocols and new file formats should be little-endian if possible.

bitwize 9y ago

> Swapping bytes is a huge pain in the butt! When reading large binary files, it's so convenient and efficient to be able to mmap and make struct pointers right into the file. No no no don't do this don't do this don't do this. This is how horrors and abominations like .doc, .xls, .psd happen. The correct way to handle binary data is to unpack it into the struct byte by byte. The reason for this is that when you define a struct in C(++), there are not just endianness issues, but implementation-dependent issues of padding and alignment you have to consider. Recently, most compilers on most architectures standardized on self-alignment rules for all primitive types except char, but this is not guaranteed by the standard, it will bite you in the ass when you least expect it, and it will be decades yet before all the C code in the world is displaced by a single-implementation language like Rust. The best way to write portable code that will work as intended is to treat all data coming from disk or over the wire as a bag of bytes, and not attempt to alias it to a struct.

api 9y ago
Most compilers have directives for packing structs. What the OP suggests is something that can be done for high performance, but don't expect it to be portable outside of common architectures and compilers. I'd consider it something that shouldn't be done unless you really need the simplicity or performance.
IshKebab 9y ago
No, dumping internal representations to disk is orthogonal to zero-static data formats. You can have a well defined format that doesn't require byte-by-byte parsing. Look up FlatBuffers and CapnProto.
- bitwize 9y ago
  Both of those appear to work by deferring the parsing step to access time. You're still treating the thing as a bag of bytes and unpacking stuff out of it bytewise.
  - kentonv 9y ago
    That's incorrect. When you load an int64 field from a Cap'n Proto field, you are doing a 64-bit load instruction directly from the source bytes. You are not doing byte-by-byte access nor any sort of translation or "parsing". Cap'n Proto works by laying out data structures like a C compiler would, but following consistent, portable rules so that the layout is the same on all platforms. It then generates inline-able accessor functions to manipulate these structures which do pointer arithmetic similar to what a compiler would generate for accessing a struct. The end result is that accessing primitive fields from a Cap'n Proto struct is essentially identical in terms of machine instructions to accessing fields of a C struct. To call that "deferring the parsing to access time" does not make sense. (Note that for pointers, there are more differences: namely, because data will not always be loaded at the same address, pointers need to be relative rather than absolute. They also need to be bounds-checked for security. This adds a few instructions to pointer accesses, but those instructions look nothing like traditional "parsing" and certainly aren't byte-by-byte operations.) Your statement earlier: > The correct way to handle binary data is to unpack it into the struct byte by byte. This is inaccurate. Byte-by-byte parsing is a valid way to do parsing but not the only way. Byte-by-byte parsers tend to be slow and -- arguably, more importantly -- overly complex and rigid. It is, for example, usually very hard to do "random access" with a byte-by-byte parser, because allowing out-of-order parsing tends to blow the code complexity through the roof. On the other hand, with Cap'n Proto and similar approaches, you can trivially mmap() a very large data structure and traverse it randomly, and it "just works". (Disclosure: I'm the author of Cap'n Proto, as well as the author of the first open source release of Google's Protocol Buffers, which does byte-by-byte binary parsing.)
planteen 9y ago
I totally agree with you and so does Rob Pike: https://commandcenter.blogspot.com/2012/04/byte-order-fallacy.html https://commandcenter.blogspot.com/2012/04/byte-order-fallac...
deleted 9y ago
[deleted]
pipio21 9y ago
This makes no sense in lots of applications. Today computers have hardware acceleration for moving chunks of data. Big chucks of data, unpack things byte by byte makes not sense on things that have to be efficient. It is like 20.000 times more efficient to do things with blocks. If you move text around you probably don't care about efficiency and can do it. If you care about efficiency and price(using cheap components) it is a very bad solution. We use in all our serial-deserial a very fine grained controlled C library and it has been working flawlessly for years. This library is interfaced with C++, java, objC, swift,clojure, rust...
Const-me 9y ago
I have to disagree with your “No no no”. It all depends on requirements. When you’re on a PC, and working with files that you can reasonably expect will be exchanged (like doc, xls or pdf) — then your “No no no” heuristic is absolutely correct. As you pointed out, differences between compilers and between e.g. x86/amd64 are very likely to render these formats incompatible. But when you’re working with files that only your software will access, unpacking it byte by byte will slow down IO by a huge factor compared to both mmap, and read/write of large blocks (the latter will likely translate to DMA i.e. the CPU will be free to do something else). When that’s the case (like for most videogames, even for PC ones), you don’t want to pay that performance penalty, you want to get that data in the memory ASAP.
- phkahler 9y ago
  >> It all depends on requirements. Requirements change.
  - Qwertious 9y ago
    Requirements can change, sometimes unexpectedly. But if you "know" for a fact that they won't, then doing the hypothetically more-portable thing at the expense of today's performance, is just plain overengineering.
    - jsjohnst 9y ago
      You never know something like that “as a fact”.
- bitwize 9y ago
  > But when you’re working with files that only your software will access, unpacking it byte by byte will slow down IO by a huge factor compared to both mmap, and read/write of large blocks (the latter will likely translate to DMA i.e. the CPU will be free to do something else). If: * you're working with files that only your software will access AND * you control, and understand, the CPU architecture of any machine that will touch that data structure and you KNOW that it will never change for the entire lifetime of that piece of data AND * you always use the same version, or an ABI-compatible future version, of the same compiler that you KNOW will always lay out data the same way and you know what that way is AND * you either don't need to touch this data structure in other programming languages, or you KNOW that this won't be an issue (for example because your programming language implementation was written in C, compiled against the same version of the compiler, and has an FFI that understands C structs) THEN, you may proceed to mmap structs into memory. In practice these constraints are fairly commonplace; for example, NetBSD wscons device drivers report HID events which are specified in a struct, and because the HID events are likely never to leave the originating machine, only be passed from kernel space to user space, it makes sense to simply read(2) them straight into the struct. And there's certainly nothing wrong with snarfing a large file into a char[], but when it comes time to extract meaning from it, unless you are absolutely sure that the underlying assumptions regarding data layout will never ever change, it will only cause marginal harm to do everything in byte offsets and shift and OR bytes to yield final values -- and this is the best portable way to access the data therein, agnostic of details about the compiler and CPU architecture. In general it's a good idea to err on the side of caution by default, profile, and then optimize the hot paths as necessary bearing the constraints you're assuming in mind (and perhaps documenting them for good measure). Maybe my initial statement was too strong, but I shudder whenever I see comments of the form "It's so convenient to just mmap() that sucker into a struct and access the fields!" Because they've never had to deal with the consequences of trying to access a struct from a Microsoft compiler serialized to disk, and finding out that GNU compilers have a quite different notion of how structure members are to be laid out in memory...