5 ms·

> The only good reason to use packed structures is when you need to map some memory (e.g. hardware registers exposed to memory) bit by bit to a structure. Anot

> The only good reason to use packed structures is when you need to map some memory (e.g. hardware registers exposed to memory) bit by bit to a structure. Another common reason is when two CPUs of different architecture need to access the same structure in memory. E.g. you have a RiscV and an Arm64 processor in the same system, sharing memory. Or you read structured binary data from disk and need to specify an exact layout.

tjalfi 3y ago
Some targets also don't use natural alignment. AIX, for example, uses 4-byte alignment for doubles that aren't the first field in the struct. GCC has the -malign-natural and -malign-power options for dealing with this.
mananaysiempre 3y ago
All of these sound weird to me—most non-stupid (hello 802.2) protocols and hardware are going to have natural-aligned structure fields, so basically any mainstream (8-bit-byte, two’s complement, etc.) ABI is going to lay them out the same way, packed or not. As for RV64 and Arm64, the layout rules for same-size scalar types in their common ABIs are outright identical aren’t they? We’re (most of us) a long way away from the time where each DOS compiler had its own opinions on whether long double should be 8-, 16-, 32-, or 64-bit aligned and 80 or 128 bits long.
- com2kid 3y ago
  > All of these sound weird to me—most non-stupid (hello 802.2) protocols and hardware are going to have natural-aligned structure fields, so basically any mainstream (8-bit-byte, two’s complement, etc.) ABI is going to lay them out the same way, packed or not. In the long ago year of 2015 I worked on a project where the same binary packet was: 1. Generated by an 8 bit micro controller 2. Consumed by a 32bit Cortex M3 3. Passed onto iPhones, Androids, Windows Phones, and Windows PCs running ObjC, Java, C#, and C++ respectively 4. Uploaded to a cloud provider The phrase "natural aligned" has no meaning in that context.
  - mananaysiempre 3y ago
    > The phrase "natural aligned" has no meaning in that context. The phrase “naturally aligned” as I’m accustomed to seeing it used refers to the alignment of a power-of-two-sized type (usually a scalar one) being equal to its size. Unless you’re working with, say, 18-bit or 24-bit integers (that do exist in obscure places), it does have a meaning, and unless you’re using non-eight-bit bytes that meaning is fairly universal (and if you’re not, your I/O is probably screwed up in hard-to-predict ways[1]). At least for your items 2, 3, and 4—excluding Java and C# which are not relevant to TFA about C and are likely to use manual packing code—you have, let’s see, - The bytes are eight bits wide, and ASCII byte strings have their usual meaning; - The integer types are wraparound unsigned and two’s complement signed least-endian with no padding bits or trap representations and come in 8-bit, 16-bit, 32-bit, and 64-bit sizes and identical alignments; - The floating-point types are IEEE 754 single and double precision floats, little endian, respectively 32 bits and 64 bits in size and of identical alignment, though you should probably avoid relying on subnormals or the exact choice of NaNs; - Structures and unions have the alignment requirement of their most strictly aligned member; - The members of a structure are laid out at increasing offsets, with each member starting at the earliest offset permitted by its alignment (while the members of a union all start at offset zero as the standard requires); - The structure or union is then padded at the end so that its alignment divides its size. If you avoid extended precision and SIMD types, the default ABI settings should get you completely compatible layouts here. (On an earlier ARM you might’ve run into mixed-endian floats, but not on any Cortex flavour.) Even bitfields would be entirely fine, except Microsoft bloody Windows had to be stupid there. Honestly the only potential problem is 1, an unspecified 8-bit controller, and that only because the implicit integer promotions of standard C make getting decent performance out of those a bit of a crapshoot, leading to noncompliant hacks like 8-bit ints or 48-bit long longs. Still, if the usual complement of 8/16/32/64-bit integers is available, the worst you’re likely to have to do is spell out any structure padding explicitly. [1] https://thephd.dev/conformance-should-mean-something-fputc-and-freestanding https://thephd.dev/conformance-should-mean-something-fputc-a...
    - colonwqbang 3y ago
      I think we agree that this makes sense on some metaphysical level. The problem is that there are definitely platforms where the normal alignment isn't what you describe above. And there isn't to my knowledge a switch in GCC to force it to follow these rules on any given platform. There isn't __attribute__((natural_alignment)). But there is __attribute__((packed)).
- packetlost 3y ago
  You assume it was RV64. It could have easily been RV32
  - mananaysiempre 3y ago
    I don’t think RV32 actually differs re alignment or struct layout, it’s just that with RV64 and Arm64 even the non-fixed-width names for the integer types are the same (LP64) except for Windows-on-ARM.
- colonwqbang 3y ago
  > As for RV64 and Arm64 This was just a placeholder, perhaps a bad example. I program a proprietary CPU architecture which does not require alignment. And for which the compiler naturally prefers to pack structs. Getting it to mimick Arm style struct padding is much harder and error prone than just having the Arm pack everything. Maybe you are right and we are heading for a One True Struct Layout in the future. Today I think it is still too scary to pass the same unpacked struct declaration to various compiler archs and hope they come up with the same interpretation.
gerdesj 3y ago
"The only good reason to " The word opinionated was coined and adopted in English to describe a certain attitude. It has functioned fine for (probably) centuries (who knows, and I can't be bothered to research too far). Then came the age of IT and blow me, are we not opinionated to the point of ridiculousness. A sentence construction along the lines of "The only good reason to" [do x] "is" [y], seems to invite a negation, quite aggressively. You might as well stand in the rain, wearing steel armour, and holding a long copper rod ... and shout "All Gods are bastards" (as a Knight of the Realm, sadly deceased, from hereabouts suggested might be an unwise life shortening decision). I'm pretty sure packed structures have other uses.
- juunpp 3y ago
  Good call. Such statement implies that there is no other reason, as obvious as that might sound. Just say "One reason to" to avoid implying that's the only one. > I'm pretty sure packed structures have other uses. Like reducing memory footprint.
- RobotToaster 3y ago
  >I'm pretty sure packed structures have other uses. And if you want to learn of them, there is no faster way.
Dylan16807 3y ago
> Another common reason is when two CPUs of different architecture need to access the same structure in memory. That sounds like mapping memory and something you could avoid with the right compiler flags. > Or you read structured binary data from disk and need to specify an exact layout. If you need it stay in the same form it has on the disk, I think it's fair to call that a concern in the category of "map some memory". Edit: And other spots in the text explicitly include file formats. Basically I think you're giving examples rather than exceptions.
kazinator 3y ago
Unless you're using weird C compilers, you shouldn't need to pack structures in order to have RiscV and Arm64 access the same structure. Packing structures will be detrimental in a situation where any of the processors have alignment requirements for accessing words. Extra instructions may be required to access that four byte member at offset 13, by using two accesses and some shifting and masking. Atomicity has gone out the window. If we look at, say, GCC, the structure layout rules are very consistent across the targets. They are not absolutely the same, but consistent enough to work with. Packing won't take care of byte order, though it's not super common to have a system where hosts of opposite byte order are accessing the same buses, and sharing memory. (I've worked on systems where a DPS had "weird order", like opposite endian from the host, but in 16 bit units. Not DCBA but CDBA or something like that.) You may be tempted to use a packed structure for conforming to some layout in a network packet or on disk, but it leaves the code nonportable and slow. In between the time the packed structure is read from the file and written again, it might be accessed many times. All those times may be slowed down due to the packing. You may be better off writing proper serializing and deserializing. That can also help if there are multiple versions of the format, or some optional fields and other crap. You may be able to map all the variants onto a single in-memory structure, dealing with the differences only in the serializing and deserializing routines. I'm skeptical of how common it is to need packing in order to map hardware registers bit-by-bit. Hardware registers tend to be very regularly sized and spaced. If you're literally mapping bit-by-bit, the language extension that is most beneficial is the ability to declare a bitfield to be of any integer type. E.g. if registers are consecutive 32 bit words: // Register AREG (32 bits) uint32_t AREG_foo_field : 13; uint32_t AREG_bar_bit : 1; uint32_t : 0; // done with this cell: go to next // Register BREG uint32_t BREG_bozo_bit : 1; Bitfield allocation is endian-dependent. In the end, for utmost portability, you want to access the 32 bit word as a word, and do the shifting and masking, where you can add a byte swap, if necessary.
throwaway894345 3y ago
Whoa, I didn’t know there were systems where two different CPUs shared memory… How does that work? Any examples I might recognize of such systems?
- colonwqbang 3y ago
  It is common for small CPUs to be embedded inside of various hardware devices. For instance, your GPU might have one or more control CPUs embedded inside. These CPUs would have either direct or at least DMA access to main memory. If you have heard about "firmware" being necessary for a hardware device to function, that "firmware" is really just software that runs on one of these auxiliary CPUs. Suppose you want to send a command from the main CPU to this subprocessor. For efficiency and simplicity, that command might be defined as a C struct, in a common header. In that case it can be good to use packed alignment so you don't have to worry about possible layout differences between CPUs.
scoutt 3y ago
Or you have to transmit a struct as a network packet "as is", between different architectures.

tjalfi 3y ago

Some targets also don't use natural alignment. AIX, for example, uses 4-byte alignment for doubles that aren't the first field in the struct. GCC has the -malign-natural and -malign-power options for dealing with this.

mananaysiempre 3y ago

All of these sound weird to me—most non-stupid (hello 802.2) protocols and hardware are going to have natural-aligned structure fields, so basically any mainstream (8-bit-byte, two’s complement, etc.) ABI is going to lay them out the same way, packed or not. As for RV64 and Arm64, the layout rules for same-size scalar types in their common ABIs are outright identical aren’t they? We’re (most of us) a long way away from the time where each DOS compiler had its own opinions on whether long double should be 8-, 16-, 32-, or 64-bit aligned and 80 or 128 bits long.

com2kid 3y ago
> All of these sound weird to me—most non-stupid (hello 802.2) protocols and hardware are going to have natural-aligned structure fields, so basically any mainstream (8-bit-byte, two’s complement, etc.) ABI is going to lay them out the same way, packed or not. In the long ago year of 2015 I worked on a project where the same binary packet was: 1. Generated by an 8 bit micro controller 2. Consumed by a 32bit Cortex M3 3. Passed onto iPhones, Androids, Windows Phones, and Windows PCs running ObjC, Java, C#, and C++ respectively 4. Uploaded to a cloud provider The phrase "natural aligned" has no meaning in that context.
- mananaysiempre 3y ago
  > The phrase "natural aligned" has no meaning in that context. The phrase “naturally aligned” as I’m accustomed to seeing it used refers to the alignment of a power-of-two-sized type (usually a scalar one) being equal to its size. Unless you’re working with, say, 18-bit or 24-bit integers (that do exist in obscure places), it does have a meaning, and unless you’re using non-eight-bit bytes that meaning is fairly universal (and if you’re not, your I/O is probably screwed up in hard-to-predict ways[1]). At least for your items 2, 3, and 4—excluding Java and C# which are not relevant to TFA about C and are likely to use manual packing code—you have, let’s see, - The bytes are eight bits wide, and ASCII byte strings have their usual meaning; - The integer types are wraparound unsigned and two’s complement signed least-endian with no padding bits or trap representations and come in 8-bit, 16-bit, 32-bit, and 64-bit sizes and identical alignments; - The floating-point types are IEEE 754 single and double precision floats, little endian, respectively 32 bits and 64 bits in size and of identical alignment, though you should probably avoid relying on subnormals or the exact choice of NaNs; - Structures and unions have the alignment requirement of their most strictly aligned member; - The members of a structure are laid out at increasing offsets, with each member starting at the earliest offset permitted by its alignment (while the members of a union all start at offset zero as the standard requires); - The structure or union is then padded at the end so that its alignment divides its size. If you avoid extended precision and SIMD types, the default ABI settings should get you completely compatible layouts here. (On an earlier ARM you might’ve run into mixed-endian floats, but not on any Cortex flavour.) Even bitfields would be entirely fine, except Microsoft bloody Windows had to be stupid there. Honestly the only potential problem is 1, an unspecified 8-bit controller, and that only because the implicit integer promotions of standard C make getting decent performance out of those a bit of a crapshoot, leading to noncompliant hacks like 8-bit ints or 48-bit long longs. Still, if the usual complement of 8/16/32/64-bit integers is available, the worst you’re likely to have to do is spell out any structure padding explicitly. [1] https://thephd.dev/conformance-should-mean-something-fputc-and-freestanding https://thephd.dev/conformance-should-mean-something-fputc-a...
  - colonwqbang 3y ago
    I think we agree that this makes sense on some metaphysical level. The problem is that there are definitely platforms where the normal alignment isn't what you describe above. And there isn't to my knowledge a switch in GCC to force it to follow these rules on any given platform. There isn't __attribute__((natural_alignment)). But there is __attribute__((packed)).
packetlost 3y ago
You assume it was RV64. It could have easily been RV32
- mananaysiempre 3y ago
  I don’t think RV32 actually differs re alignment or struct layout, it’s just that with RV64 and Arm64 even the non-fixed-width names for the integer types are the same (LP64) except for Windows-on-ARM.
colonwqbang 3y ago
> As for RV64 and Arm64 This was just a placeholder, perhaps a bad example. I program a proprietary CPU architecture which does not require alignment. And for which the compiler naturally prefers to pack structs. Getting it to mimick Arm style struct padding is much harder and error prone than just having the Arm pack everything. Maybe you are right and we are heading for a One True Struct Layout in the future. Today I think it is still too scary to pass the same unpacked struct declaration to various compiler archs and hope they come up with the same interpretation.

gerdesj 3y ago

"The only good reason to " The word opinionated was coined and adopted in English to describe a certain attitude. It has functioned fine for (probably) centuries (who knows, and I can't be bothered to research too far). Then came the age of IT and blow me, are we not opinionated to the point of ridiculousness. A sentence construction along the lines of "The only good reason to" [do x] "is" [y], seems to invite a negation, quite aggressively. You might as well stand in the rain, wearing steel armour, and holding a long copper rod ... and shout "All Gods are bastards" (as a Knight of the Realm, sadly deceased, from hereabouts suggested might be an unwise life shortening decision). I'm pretty sure packed structures have other uses.

juunpp 3y ago
Good call. Such statement implies that there is no other reason, as obvious as that might sound. Just say "One reason to" to avoid implying that's the only one. > I'm pretty sure packed structures have other uses. Like reducing memory footprint.
RobotToaster 3y ago
>I'm pretty sure packed structures have other uses. And if you want to learn of them, there is no faster way.

Dylan16807 3y ago

> Another common reason is when two CPUs of different architecture need to access the same structure in memory. That sounds like mapping memory and something you could avoid with the right compiler flags. > Or you read structured binary data from disk and need to specify an exact layout. If you need it stay in the same form it has on the disk, I think it's fair to call that a concern in the category of "map some memory". Edit: And other spots in the text explicitly include file formats. Basically I think you're giving examples rather than exceptions.

kazinator 3y ago

Unless you're using weird C compilers, you shouldn't need to pack structures in order to have RiscV and Arm64 access the same structure. Packing structures will be detrimental in a situation where any of the processors have alignment requirements for accessing words. Extra instructions may be required to access that four byte member at offset 13, by using two accesses and some shifting and masking. Atomicity has gone out the window. If we look at, say, GCC, the structure layout rules are very consistent across the targets. They are not absolutely the same, but consistent enough to work with. Packing won't take care of byte order, though it's not super common to have a system where hosts of opposite byte order are accessing the same buses, and sharing memory. (I've worked on systems where a DPS had "weird order", like opposite endian from the host, but in 16 bit units. Not DCBA but CDBA or something like that.) You may be tempted to use a packed structure for conforming to some layout in a network packet or on disk, but it leaves the code nonportable and slow. In between the time the packed structure is read from the file and written again, it might be accessed many times. All those times may be slowed down due to the packing. You may be better off writing proper serializing and deserializing. That can also help if there are multiple versions of the format, or some optional fields and other crap. You may be able to map all the variants onto a single in-memory structure, dealing with the differences only in the serializing and deserializing routines. I'm skeptical of how common it is to need packing in order to map hardware registers bit-by-bit. Hardware registers tend to be very regularly sized and spaced. If you're literally mapping bit-by-bit, the language extension that is most beneficial is the ability to declare a bitfield to be of any integer type. E.g. if registers are consecutive 32 bit words: // Register AREG (32 bits) uint32_t AREG_foo_field : 13; uint32_t AREG_bar_bit : 1; uint32_t : 0; // done with this cell: go to next // Register BREG uint32_t BREG_bozo_bit : 1; Bitfield allocation is endian-dependent. In the end, for utmost portability, you want to access the 32 bit word as a word, and do the shifting and masking, where you can add a byte swap, if necessary.

throwaway894345 3y ago

Whoa, I didn’t know there were systems where two different CPUs shared memory… How does that work? Any examples I might recognize of such systems?

colonwqbang 3y ago
It is common for small CPUs to be embedded inside of various hardware devices. For instance, your GPU might have one or more control CPUs embedded inside. These CPUs would have either direct or at least DMA access to main memory. If you have heard about "firmware" being necessary for a hardware device to function, that "firmware" is really just software that runs on one of these auxiliary CPUs. Suppose you want to send a command from the main CPU to this subprocessor. For efficiency and simplicity, that command might be defined as a C struct, in a common header. In that case it can be good to use packed alignment so you don't have to worry about possible layout differences between CPUs.

scoutt 3y ago

Or you have to transmit a struct as a network packet "as is", between different architectures.