4 ms·

Reading serialized data and then casting it into a structure type is ill-defined, very bad practice, and there's nothing in C that tells you it's a good or safe

by unwind 13y ago

Reading serialized data and then casting it into a structure type is ill-defined, very bad practice, and there's nothing in C that tells you it's a good or safe thing to be doing. So, please don't do that. The exact layout of a structure in memory is up to the compiler, and can easily change with compiler options, even if the compiler itself and the target architecture and platform remain the same. An externally-visible serialization format should of course be stable, and not depend on the compiler used to build the code, or the hardware platform which might have ideas of how various fields should be aligned (or even how to order the bytes in integers larger than 8 bits). Serializing data is something you do, just blindly copying the run-time representation that you have to an external medium is not serialization. You should do it field by field, with a well-defined format chosen for each field.

vidarh 13y ago
All of that is true unless it is, say, cache data that you can afford to ditch if the format has changed, and you validate it after reading (say with a carefully structured "magic" value).
viraptor 13y ago
That's the theory, sure. But in practice, if you know all your targets support the right "#pragma pack" and "__attribute__((packed))" and other markers that you use - what's wrong with making use of them?
- joosters 13y ago
  Also, if you're passing data between two processes (e.g. your code fork()d a child), you can be sure that the data format is going to match, no packing required. (still need to ensure alignment in memory though)
  - mikeash 13y ago
    This is not necessarily true. Lots of systems support running 32-bit and 64-bit binaries simultaneously, and structs won't necessarily be portable between the two. Some versions of Mac OS X even supported running 32-bit PowerPC binaries (in an emulator, but supported directly by the OS) next to both 32-bit and 64-bit x86 binaries, so you could not only get size mismatches, but also endian mismatches. This does not apply to simply forking, of course, but it does apply to the "two processes" case in general.
    - raverbashing 13y ago
      On the same machine, it's going to be rare But the most common occurrence of this I can think are online games.
      - mikeash 13y ago
        Depends on what system and context you're talking about. I'd assume it's rare on Linux where most things are built from source, but it's pretty common on e.g. Apple platforms where binary compatibility means 32-bit processes keep being supported for years, and they still need to IPC to various system services. I've seen one highly amusing bug in Apple's iOS frameworks where they didn't quite handle the 32->64 IPC case properly because of an implicit assumption that data types remained identical.
    - joosters 13y ago
      Oh yes, I was a bit unclear but I meant two processes (from the same binary) communicating. Without that condition, you've got all the original problems, as listed. You can also have incompatibilities between compilers, or even compiler versions - e.g. gcc has ABI changes between some versions.
- mikeash 13y ago
  The fact that you can't know all of your future targets will support that is a problem. Endian issues are also a big problem here. The need to byte-swap everything eliminates a lot of the convenience. It's also way too easy to make breaking changes to the struct, since there's no standard way to mark a struct as being something that you serialize/deserialize, and normally you can change struct fields at will in C code. Just take the extra five minutes to write code to translate between your struct and a stream of bytes. It'll be easier to understand, less risky, and more compatible.
  - raverbashing 13y ago
    "The fact that you can't know all of your future targets will support that is a problem." I don't see this as much of a problem as endianness, but yeah, maybe, x86 made us accustomed. And you can use portable types, defined on headers (linux does that, like u8, u16, etc) But sometimes you know your target won't change (for a long time) "It's also way too easy to make breaking changes to the struct, since there's no standard way to mark a struct as being something that you serialize/deserialize, and normally you can change struct fields at will in C code." In the same way you can break your software doing any change. You can mark it with a naming convention but the easiest way is seeing the "pack" directives on it. And unit tests "Just take the extra five minutes to write code to translate between your struct and a stream of bytes. It'll be easier to understand, less risky, and more compatible. " Well, it's not five minutes. And it makes your program slower (for the most convoluted data types). A simple example: http://en.wikipedia.org/wiki/BMP_file_format http://en.wikipedia.org/wiki/BMP_file_format And yes, GIMP reads field by field https://git.gnome.org/browse/gimp/tree/plug-ins/file-bmp/bmp-read.c?h=gimp-2-6 https://git.gnome.org/browse/gimp/tree/plug-ins/file-bmp/bmp...
    - mikeash 13y ago
      Does it really make your program slower? That padding you have to eliminate for this technique is added for speed, after all. The initial read may be faster, but every access to the data is going to be slower. I'm not quite sure what the BMP file format is supposed to be an example of, especially since file formats are separate from techniques used to read or write them.
- weland 13y ago
  > But in practice, if you know all your targets support the right "#pragma pack" and "__attribute__((packed))" and other markers that you use - what's wrong with making use of them? You know that at this moment they all support the right __attribute__((packed)) et co.. Bonus points for endianness crap.

by unwind 13y ago

vidarh 13y ago

All of that is true unless it is, say, cache data that you can afford to ditch if the format has changed, and you validate it after reading (say with a carefully structured "magic" value).

viraptor 13y ago

That's the theory, sure. But in practice, if you know all your targets support the right "#pragma pack" and "__attribute__((packed))" and other markers that you use - what's wrong with making use of them?

joosters 13y ago
Also, if you're passing data between two processes (e.g. your code fork()d a child), you can be sure that the data format is going to match, no packing required. (still need to ensure alignment in memory though)
- mikeash 13y ago
  This is not necessarily true. Lots of systems support running 32-bit and 64-bit binaries simultaneously, and structs won't necessarily be portable between the two. Some versions of Mac OS X even supported running 32-bit PowerPC binaries (in an emulator, but supported directly by the OS) next to both 32-bit and 64-bit x86 binaries, so you could not only get size mismatches, but also endian mismatches. This does not apply to simply forking, of course, but it does apply to the "two processes" case in general.
  - raverbashing 13y ago
    On the same machine, it's going to be rare But the most common occurrence of this I can think are online games.
    - mikeash 13y ago
      Depends on what system and context you're talking about. I'd assume it's rare on Linux where most things are built from source, but it's pretty common on e.g. Apple platforms where binary compatibility means 32-bit processes keep being supported for years, and they still need to IPC to various system services. I've seen one highly amusing bug in Apple's iOS frameworks where they didn't quite handle the 32->64 IPC case properly because of an implicit assumption that data types remained identical.
  - joosters 13y ago
    Oh yes, I was a bit unclear but I meant two processes (from the same binary) communicating. Without that condition, you've got all the original problems, as listed. You can also have incompatibilities between compilers, or even compiler versions - e.g. gcc has ABI changes between some versions.
mikeash 13y ago
The fact that you can't know all of your future targets will support that is a problem. Endian issues are also a big problem here. The need to byte-swap everything eliminates a lot of the convenience. It's also way too easy to make breaking changes to the struct, since there's no standard way to mark a struct as being something that you serialize/deserialize, and normally you can change struct fields at will in C code. Just take the extra five minutes to write code to translate between your struct and a stream of bytes. It'll be easier to understand, less risky, and more compatible.
- raverbashing 13y ago
  "The fact that you can't know all of your future targets will support that is a problem." I don't see this as much of a problem as endianness, but yeah, maybe, x86 made us accustomed. And you can use portable types, defined on headers (linux does that, like u8, u16, etc) But sometimes you know your target won't change (for a long time) "It's also way too easy to make breaking changes to the struct, since there's no standard way to mark a struct as being something that you serialize/deserialize, and normally you can change struct fields at will in C code." In the same way you can break your software doing any change. You can mark it with a naming convention but the easiest way is seeing the "pack" directives on it. And unit tests "Just take the extra five minutes to write code to translate between your struct and a stream of bytes. It'll be easier to understand, less risky, and more compatible. " Well, it's not five minutes. And it makes your program slower (for the most convoluted data types). A simple example: http://en.wikipedia.org/wiki/BMP_file_format http://en.wikipedia.org/wiki/BMP_file_format And yes, GIMP reads field by field https://git.gnome.org/browse/gimp/tree/plug-ins/file-bmp/bmp-read.c?h=gimp-2-6 https://git.gnome.org/browse/gimp/tree/plug-ins/file-bmp/bmp...
  - mikeash 13y ago
    Does it really make your program slower? That padding you have to eliminate for this technique is added for speed, after all. The initial read may be faster, but every access to the data is going to be slower. I'm not quite sure what the BMP file format is supposed to be an example of, especially since file formats are separate from techniques used to read or write them.
weland 13y ago
> But in practice, if you know all your targets support the right "#pragma pack" and "__attribute__((packed))" and other markers that you use - what's wrong with making use of them? You know that at this moment they all support the right __attribute__((packed)) et co.. Bonus points for endianness crap.