17 ms·

This is a nice syntax, but the functionality seems essentially the same as bit fields in C. Does this provide anything additional?

WalterBright 4y ago
D just went ahead and implemented bit fields. They work, and it's hard to find any way to improve on them.
- addaon 4y ago
  It also specified the layout (little endian) and disallowed reordering / padding / etc.
  - leni536 4y ago
    How bitfields are allocated within a byte is related, but independent from endianness. endianness: how an array of bytes is interpreted as an integer/how an integer is layed out in memory as an array of bytes. bitfield allocations: how subsequent bitfields are allocated within an integer, typically starting from least significant bit to most significant, or the other way around.
    - addaon 4y ago
      Fair, and a good point. In this particular case, the two are related, because the LSBit-first bitfield allocations can spill over between bytes, giving LSByte-first endianness as well.
- mhh__ 4y ago
  It's hard but they aren't perfect. You shouldn't rely on them for packing memory at all if you want to talk to other programs. Similarly bitfields mean you end up with "ints" that are actually 3 bits wide and so on.
  - WalterBright 4y ago
    > You shouldn't rely on them for packing memory at all if you want to talk to other programs. While the layout is indeed implementation dependent, pragmatically if you stick to using ints the layouts are portable as far as I can tell. Just like the size of ints is implementation dependent, but is reliably 32 bits on 32 and 64 bit machines.
deleted 4y ago
[deleted]
slimsag 4y ago
Admittedly I didn't know C had bit fields like that, so I didn't cover them. I've never seen them used in practice and I've used quite a lot of C libraries. I wonder why that is? One major difference appears to be that C bitfields memory layout is compiler-dependant. The other major difference is Zig's arbitrary-bit-width integer types just leading to less footguns I would speculate
- bananaboy 4y ago
  I don't know why you haven't come across them but personally I've used bitfields a lot over my career as a video game programmer at different companies. You can have arbitrary bit width integers as well. It's not really very different to zig.
  - flohofwoe 4y ago
    > You can have arbitrary bit width integers as well. This is news to me, how? AFAIK they'll only eventually make it into C23 with _BitInt(N). ...and apart from building a C++ wrapper class of course, but how would this pack with data outside the class - like the 4 + 28 bits example in the blog post. Also IIRC when I tinkered with C/C++ bitfields, some compilers (at least MSVC I think?) didn't properly pack the bits (e.g. a single bit would be padded to a full byte, or they couldn't agree on a common size of the containing integer - e.g. one compiler packing <8 number of bits into an uint8_t, and another into a uint32_t). In the end C/C++ bitfields weren't all that useful for the use case described in the blog post, at least if portability across the three big compilers is needed (gcc, clang, msvc).
    - shultays 4y ago
      Maybe other members in same struct was enforcing such packing? Such as https://godbolt.org/z/8G4v8McsP https://godbolt.org/z/8G4v8McsP You probably want static asserts for sizes in your code if you are trying to optimize your struct paddings
      - flohofwoe 4y ago
audunw 4y ago
C bitfields are problematic and often aren't used in places where you'd think they'd be useful. Zig seems to be doing its best to get bitfields right. Just one thing that Zig improves: in C if you need the bit offsets of the fields you'll still need lots of defines with the offsets/masks and such. In Zig it can be extracted from the packed struct at comptime
- spaintech 4y ago
  Don’t get me wrong, it’s a cool feature, the only problem I see here is zero readability, what if I’m looking at this code or trying to figure out the value of a register mangled by such a call, “I” find it very hard to read and understand what is happening there. I can read hex offsets and masks to know what’s what is happening to bitfield at any given time… just MHO.
sqrt_1 4y ago
Here is a sample of how you do this in C++ https://godbolt.org/z/W3bMcqM5P https://godbolt.org/z/W3bMcqM5P
- flumpcakes 4y ago
  Wow... I thought I had seen a lot of strange C++ code but I have _never_ seen packed structs using this syntax: struct X { type alias : numeric_value = false; ... } I love the power of C++ but there is _so much_ to the language. I'm sure there would also be some template meta programming solution even if this syntax was available.
  - Joker_vD 4y ago
    Uh, that's standard C89. You know, the bit fields? Except for the "false" keyword, that's from stdbool.h and you need C99 for that to reliably exist.
    - rrampage 4y ago
      The rough C equivalent looks like https://godbolt.org/z/YqGPa7ecd https://godbolt.org/z/YqGPa7ecd , using _Static_assert ( available C11 on wards and as a GCC built-in from 4.6 ) with no default values for the struct members
    - stephencanon 4y ago
      The only real problem with C (and C++) bitfields is that the standard doesn’t say anything about how they’re laid out in memory, which unfortunately makes them mostly useless unless you either don’t care about layout or you’re targeting one specific compiler and that compiler documents its choices and doesn’t break them in a few years. Sign-extension of 1-bit fields also messes people up all the time, but that’s “just” an easy-to-fix bug.
      - pclmulqdq 4y ago
        That is a really big problem in some applications. I wish the C committee would come out and say "use the layout that GCC 11 .x.x does" to make these a lot more usable.
tempodox 4y ago
Bit fields in C are notoriously non-portable (exact physical layout depends on the compiler). Is Zig any better (apart from the fact that there is only one Zig compiler for now)?
- yellowapple 4y ago
  That's the point of a packed struct: to explicitly declare the physical layout (instead of letting the compiler optimize it). If a packed struct ends up being non-portable between Zig implementations, then my impression is that one and/or the other must have a bug that must be fixed in order to actually comply with the language spec.
  - fanf2 4y ago
    It is not possible to specify bitfields in a way that makes sense, is consistent with a bytewise view of memory, and is portable between big endian and little endian processors. Say you have, for instance (using C notation) struct { unsigned one : 8; unsigned two : 8; }; The fields are supposed to be represented in memory in the same order they are declared, so one is the first byte and two is the second byte. This should have the same representation as if I declared it as two uint8_t fields. If I type pun it and load it into a register as a uint16_t then it depends on the hardware whether the low and high bytes are one and two or two and one. It gets more tricky when you consider arbitrary bit widths. struct { unsigned u4 : 4; unsigned uC : 12; }; If the fields are allocated in order, is u4 the low bits of the first byte or the high bits? If you require it to be the low bits, then it works ok on a little endian machine, but on a big endian machine the uC field ends up split, so the 16 bit view looks like: CCCC4444CCCCCCCC
    - eqvinox 4y ago
      > It is not possible to specify bitfields in a way that makes sense, is consistent with a bytewise view of memory, and is portable between big endian and little endian processors. I'm not sure I accept "consistency with a bytewise view of memory" as a well-defined, reasonable concept. I do expect to give a list of bit widths, and get a field that has these in consecutive order. Why would it randomly do weird things on an 8-bit boundary? > If the fields are allocated in order, is u4 the low bits of the first byte or the high bits? It's the low bits on LE, and the high bits on BE. > If you require it to be the low bits, then it works ok on a little endian machine, but on a big endian machine the uC field ends up split That's why the direction is defined to match the endianness; you get a consecutive chain of bits in either case. > so the 16 bit view looks like: CCCC4444CCCCCCCC It's 4444CCCCCCCCCCCC on BE, and CCCCCCCCCCCC4444 on LE. If you need something else, it's no longer a question of defining an ABI-consistent structure, but rather expressing a representation of an externally given constraint.
scaredginger 4y ago
The in-memory representation of bit fields is implementation-defined. Therefore, if you're calling into an external API that takes a uint32_t like in the example without an explicit remapping, you may or may not like the results. In practice, everything you're likely to come across will be little endian nowadays, and the ABI you're using will most likely order your struct from top to bottom in memory, so they will look the same most of the time. However, it's still technically not portable.
- galangalalgol 4y ago
  I've dealt with oddities talking between big endian powerpc using these. Its been a few years but the difference wasn't just the endianess I think? Still, dealing with the mapping was way easier than masking for large structs. Is big endian really dead now?
  - scaredginger 4y ago
    As I said, it's also ABI. Though admittedly, endianness would be encompassed by ABI, so it's all really just ABI
- throwawaymaths 4y ago
  > In practice, everything you're likely to come across will be little endian nowadays The internet?
  - scaredginger 4y ago
    Not really relevant to a discussion about CPUs and compiler implementations...
    - hansvm 4y ago
      It's a language design feature that makes some sorts of networking code much easier to write. Why wouldn't that be relevant?

WalterBright 4y ago

D just went ahead and implemented bit fields. They work, and it's hard to find any way to improve on them.

addaon 4y ago
It also specified the layout (little endian) and disallowed reordering / padding / etc.
- leni536 4y ago
  How bitfields are allocated within a byte is related, but independent from endianness. endianness: how an array of bytes is interpreted as an integer/how an integer is layed out in memory as an array of bytes. bitfield allocations: how subsequent bitfields are allocated within an integer, typically starting from least significant bit to most significant, or the other way around.
  - addaon 4y ago
    Fair, and a good point. In this particular case, the two are related, because the LSBit-first bitfield allocations can spill over between bytes, giving LSByte-first endianness as well.
mhh__ 4y ago
It's hard but they aren't perfect. You shouldn't rely on them for packing memory at all if you want to talk to other programs. Similarly bitfields mean you end up with "ints" that are actually 3 bits wide and so on.
- WalterBright 4y ago
  > You shouldn't rely on them for packing memory at all if you want to talk to other programs. While the layout is indeed implementation dependent, pragmatically if you stick to using ints the layouts are portable as far as I can tell. Just like the size of ints is implementation dependent, but is reliably 32 bits on 32 and 64 bit machines.

deleted 4y ago

[deleted]

slimsag 4y ago

Admittedly I didn't know C had bit fields like that, so I didn't cover them. I've never seen them used in practice and I've used quite a lot of C libraries. I wonder why that is? One major difference appears to be that C bitfields memory layout is compiler-dependant. The other major difference is Zig's arbitrary-bit-width integer types just leading to less footguns I would speculate

bananaboy 4y ago
I don't know why you haven't come across them but personally I've used bitfields a lot over my career as a video game programmer at different companies. You can have arbitrary bit width integers as well. It's not really very different to zig.
- flohofwoe 4y ago
  > You can have arbitrary bit width integers as well. This is news to me, how? AFAIK they'll only eventually make it into C23 with _BitInt(N). ...and apart from building a C++ wrapper class of course, but how would this pack with data outside the class - like the 4 + 28 bits example in the blog post. Also IIRC when I tinkered with C/C++ bitfields, some compilers (at least MSVC I think?) didn't properly pack the bits (e.g. a single bit would be padded to a full byte, or they couldn't agree on a common size of the containing integer - e.g. one compiler packing <8 number of bits into an uint8_t, and another into a uint32_t). In the end C/C++ bitfields weren't all that useful for the use case described in the blog post, at least if portability across the three big compilers is needed (gcc, clang, msvc).
  - shultays 4y ago
    Maybe other members in same struct was enforcing such packing? Such as https://godbolt.org/z/8G4v8McsP https://godbolt.org/z/8G4v8McsP You probably want static asserts for sizes in your code if you are trying to optimize your struct paddings
    - flohofwoe 4y ago

audunw 4y ago

C bitfields are problematic and often aren't used in places where you'd think they'd be useful. Zig seems to be doing its best to get bitfields right. Just one thing that Zig improves: in C if you need the bit offsets of the fields you'll still need lots of defines with the offsets/masks and such. In Zig it can be extracted from the packed struct at comptime

spaintech 4y ago
Don’t get me wrong, it’s a cool feature, the only problem I see here is zero readability, what if I’m looking at this code or trying to figure out the value of a register mangled by such a call, “I” find it very hard to read and understand what is happening there. I can read hex offsets and masks to know what’s what is happening to bitfield at any given time… just MHO.

sqrt_1 4y ago

Here is a sample of how you do this in C++ https://godbolt.org/z/W3bMcqM5P https://godbolt.org/z/W3bMcqM5P

flumpcakes 4y ago
Wow... I thought I had seen a lot of strange C++ code but I have _never_ seen packed structs using this syntax: struct X { type alias : numeric_value = false; ... } I love the power of C++ but there is _so much_ to the language. I'm sure there would also be some template meta programming solution even if this syntax was available.
- Joker_vD 4y ago
  Uh, that's standard C89. You know, the bit fields? Except for the "false" keyword, that's from stdbool.h and you need C99 for that to reliably exist.
  - rrampage 4y ago
    The rough C equivalent looks like https://godbolt.org/z/YqGPa7ecd https://godbolt.org/z/YqGPa7ecd , using _Static_assert ( available C11 on wards and as a GCC built-in from 4.6 ) with no default values for the struct members
  - stephencanon 4y ago
    The only real problem with C (and C++) bitfields is that the standard doesn’t say anything about how they’re laid out in memory, which unfortunately makes them mostly useless unless you either don’t care about layout or you’re targeting one specific compiler and that compiler documents its choices and doesn’t break them in a few years. Sign-extension of 1-bit fields also messes people up all the time, but that’s “just” an easy-to-fix bug.
    - pclmulqdq 4y ago
      That is a really big problem in some applications. I wish the C committee would come out and say "use the layout that GCC 11 .x.x does" to make these a lot more usable.

tempodox 4y ago

Bit fields in C are notoriously non-portable (exact physical layout depends on the compiler). Is Zig any better (apart from the fact that there is only one Zig compiler for now)?

yellowapple 4y ago
That's the point of a packed struct: to explicitly declare the physical layout (instead of letting the compiler optimize it). If a packed struct ends up being non-portable between Zig implementations, then my impression is that one and/or the other must have a bug that must be fixed in order to actually comply with the language spec.
- fanf2 4y ago
  It is not possible to specify bitfields in a way that makes sense, is consistent with a bytewise view of memory, and is portable between big endian and little endian processors. Say you have, for instance (using C notation) struct { unsigned one : 8; unsigned two : 8; }; The fields are supposed to be represented in memory in the same order they are declared, so one is the first byte and two is the second byte. This should have the same representation as if I declared it as two uint8_t fields. If I type pun it and load it into a register as a uint16_t then it depends on the hardware whether the low and high bytes are one and two or two and one. It gets more tricky when you consider arbitrary bit widths. struct { unsigned u4 : 4; unsigned uC : 12; }; If the fields are allocated in order, is u4 the low bits of the first byte or the high bits? If you require it to be the low bits, then it works ok on a little endian machine, but on a big endian machine the uC field ends up split, so the 16 bit view looks like: CCCC4444CCCCCCCC
  - eqvinox 4y ago
    > It is not possible to specify bitfields in a way that makes sense, is consistent with a bytewise view of memory, and is portable between big endian and little endian processors. I'm not sure I accept "consistency with a bytewise view of memory" as a well-defined, reasonable concept. I do expect to give a list of bit widths, and get a field that has these in consecutive order. Why would it randomly do weird things on an 8-bit boundary? > If the fields are allocated in order, is u4 the low bits of the first byte or the high bits? It's the low bits on LE, and the high bits on BE. > If you require it to be the low bits, then it works ok on a little endian machine, but on a big endian machine the uC field ends up split That's why the direction is defined to match the endianness; you get a consecutive chain of bits in either case. > so the 16 bit view looks like: CCCC4444CCCCCCCC It's 4444CCCCCCCCCCCC on BE, and CCCCCCCCCCCC4444 on LE. If you need something else, it's no longer a question of defining an ABI-consistent structure, but rather expressing a representation of an externally given constraint.

scaredginger 4y ago

The in-memory representation of bit fields is implementation-defined. Therefore, if you're calling into an external API that takes a uint32_t like in the example without an explicit remapping, you may or may not like the results. In practice, everything you're likely to come across will be little endian nowadays, and the ABI you're using will most likely order your struct from top to bottom in memory, so they will look the same most of the time. However, it's still technically not portable.

galangalalgol 4y ago
I've dealt with oddities talking between big endian powerpc using these. Its been a few years but the difference wasn't just the endianess I think? Still, dealing with the mapping was way easier than masking for large structs. Is big endian really dead now?
- scaredginger 4y ago
  As I said, it's also ABI. Though admittedly, endianness would be encompassed by ABI, so it's all really just ABI
throwawaymaths 4y ago
> In practice, everything you're likely to come across will be little endian nowadays The internet?
- scaredginger 4y ago
  Not really relevant to a discussion about CPUs and compiler implementations...
  - hansvm 4y ago
    It's a language design feature that makes some sorts of networking code much easier to write. Why wouldn't that be relevant?