14 ms·
Packed structs in Zig make bit/flag sets trivial
- MobiusHorizons 4y agoThis is a nice syntax, but the functionality seems essentially the same as bit fields in C. Does this provide anything additional?
- WalterBright 4y agoD just went ahead and implemented bit fields. They work, and it's hard to find any way to improve on them.
- addaon 4y agoIt also specified the layout (little endian) and disallowed reordering / padding / etc.
- leni536 4y agoHow bitfields are allocated within a byte is related, but independent from endianness. endianness: how an array of bytes is interpreted as an integer/how an integer is layed out in memory as an array of bytes. bitfield allocations: how subsequent bitfields are allocated within an integer, typically starting from least significant bit to most significant, or the other way around.
- addaon 4y agoFair, and a good point. In this particular case, the two are related, because the LSBit-first bitfield allocations can spill over between bytes, giving LSByte-first endianness as well.
- mhh__ 4y agoIt's hard but they aren't perfect. You shouldn't rely on them for packing memory at all if you want to talk to other programs. Similarly bitfields mean you end up with "ints" that are actually 3 bits wide and so on.
- WalterBright 4y ago> You shouldn't rely on them for packing memory at all if you want to talk to other programs. While the layout is indeed implementation dependent, pragmatically if you stick to using ints the layouts are portable as far as I can tell. Just like the size of ints is implementation dependent, but is reliably 32 bits on 32 and 64 bit machines.
- deleted 4y ago[deleted]
- slimsag 4y agoAdmittedly I didn't know C had bit fields like that, so I didn't cover them. I've never seen them used in practice and I've used quite a lot of C libraries. I wonder why that is? One major difference appears to be that C bitfields memory layout is compiler-dependant. The other major difference is Zig's arbitrary-bit-width integer types just leading to less footguns I would speculate
- bananaboy 4y agoI don't know why you haven't come across them but personally I've used bitfields a lot over my career as a video game programmer at different companies. You can have arbitrary bit width integers as well. It's not really very different to zig.
- flohofwoe 4y ago> You can have arbitrary bit width integers as well. This is news to me, how? AFAIK they'll only eventually make it into C23 with _BitInt(N). ...and apart from building a C++ wrapper class of course, but how would this pack with data outside the class - like the 4 + 28 bits example in the blog post. Also IIRC when I tinkered with C/C++ bitfields, some compilers (at least MSVC I think?) didn't properly pack the bits (e.g. a single bit would be padded to a full byte, or they couldn't agree on a common size of the containing integer - e.g. one compiler packing <8 number of bits into an uint8_t, and another into a uint32_t). In the end C/C++ bitfields weren't all that useful for the use case described in the blog post, at least if portability across the three big compilers is needed (gcc, clang, msvc).
- audunw 4y agoC bitfields are problematic and often aren't used in places where you'd think they'd be useful. Zig seems to be doing its best to get bitfields right. Just one thing that Zig improves: in C if you need the bit offsets of the fields you'll still need lots of defines with the offsets/masks and such. In Zig it can be extracted from the packed struct at comptime
- spaintech 4y agoDon’t get me wrong, it’s a cool feature, the only problem I see here is zero readability, what if I’m looking at this code or trying to figure out the value of a register mangled by such a call, “I” find it very hard to read and understand what is happening there. I can read hex offsets and masks to know what’s what is happening to bitfield at any given time… just MHO.
- sqrt_1 4y agoHere is a sample of how you do this in C++ https://godbolt.org/z/W3bMcqM5P https://godbolt.org/z/W3bMcqM5P
- flumpcakes 4y agoWow... I thought I had seen a lot of strange C++ code but I have _never_ seen packed structs using this syntax: struct X { type alias : numeric_value = false; ... } I love the power of C++ but there is _so much_ to the language. I'm sure there would also be some template meta programming solution even if this syntax was available.
- Joker_vD 4y agoUh, that's standard C89. You know, the bit fields? Except for the "false" keyword, that's from stdbool.h and you need C99 for that to reliably exist.
- rrampage 4y agoThe rough C equivalent looks like https://godbolt.org/z/YqGPa7ecd https://godbolt.org/z/YqGPa7ecd , using _Static_assert ( available C11 on wards and as a GCC built-in from 4.6 ) with no default values for the struct members
- stephencanon 4y agoThe only real problem with C (and C++) bitfields is that the standard doesn’t say anything about how they’re laid out in memory, which unfortunately makes them mostly useless unless you either don’t care about layout or you’re targeting one specific compiler and that compiler documents its choices and doesn’t break them in a few years. Sign-extension of 1-bit fields also messes people up all the time, but that’s “just” an easy-to-fix bug.
- pclmulqdq 4y agoThat is a really big problem in some applications. I wish the C committee would come out and say "use the layout that GCC 11 .x.x does" to make these a lot more usable.
- tempodox 4y agoBit fields in C are notoriously non-portable (exact physical layout depends on the compiler). Is Zig any better (apart from the fact that there is only one Zig compiler for now)?
- yellowapple 4y agoThat's the point of a packed struct: to explicitly declare the physical layout (instead of letting the compiler optimize it). If a packed struct ends up being non-portable between Zig implementations, then my impression is that one and/or the other must have a bug that must be fixed in order to actually comply with the language spec.
- fanf2 4y agoIt is not possible to specify bitfields in a way that makes sense, is consistent with a bytewise view of memory, and is portable between big endian and little endian processors. Say you have, for instance (using C notation) struct { unsigned one : 8; unsigned two : 8; }; The fields are supposed to be represented in memory in the same order they are declared, so one is the first byte and two is the second byte. This should have the same representation as if I declared it as two uint8_t fields. If I type pun it and load it into a register as a uint16_t then it depends on the hardware whether the low and high bytes are one and two or two and one. It gets more tricky when you consider arbitrary bit widths. struct { unsigned u4 : 4; unsigned uC : 12; }; If the fields are allocated in order, is u4 the low bits of the first byte or the high bits? If you require it to be the low bits, then it works ok on a little endian machine, but on a big endian machine the uC field ends up split, so the 16 bit view looks like: CCCC4444CCCCCCCC
- eqvinox 4y ago> It is not possible to specify bitfields in a way that makes sense, is consistent with a bytewise view of memory, and is portable between big endian and little endian processors. I'm not sure I accept "consistency with a bytewise view of memory" as a well-defined, reasonable concept. I do expect to give a list of bit widths, and get a field that has these in consecutive order. Why would it randomly do weird things on an 8-bit boundary? > If the fields are allocated in order, is u4 the low bits of the first byte or the high bits? It's the low bits on LE, and the high bits on BE. > If you require it to be the low bits, then it works ok on a little endian machine, but on a big endian machine the uC field ends up split That's why the direction is defined to match the endianness; you get a consecutive chain of bits in either case. > so the 16 bit view looks like: CCCC4444CCCCCCCC It's 4444CCCCCCCCCCCC on BE, and CCCCCCCCCCCC4444 on LE. If you need something else, it's no longer a question of defining an ABI-consistent structure, but rather expressing a representation of an externally given constraint.
- scaredginger 4y agoThe in-memory representation of bit fields is implementation-defined. Therefore, if you're calling into an external API that takes a uint32_t like in the example without an explicit remapping, you may or may not like the results. In practice, everything you're likely to come across will be little endian nowadays, and the ABI you're using will most likely order your struct from top to bottom in memory, so they will look the same most of the time. However, it's still technically not portable.
- galangalalgol 4y agoI've dealt with oddities talking between big endian powerpc using these. Its been a few years but the difference wasn't just the endianess I think? Still, dealing with the mapping was way easier than masking for large structs. Is big endian really dead now?
- scaredginger 4y agoAs I said, it's also ABI. Though admittedly, endianness would be encompassed by ABI, so it's all really just ABI
- throwawaymaths 4y ago> In practice, everything you're likely to come across will be little endian nowadays The internet?
- scaredginger 4y agoNot really relevant to a discussion about CPUs and compiler implementations...
- hansvm 4y agoIt's a language design feature that makes some sorts of networking code much easier to write. Why wouldn't that be relevant?
- arjvik 4y agoif (mask & (WGPUColorWriteMask_Alpha|WGPUColorWriteMask_Blue)) { // alpha and blue are set.. } Doesn't this give you if alpha OR blue is set? Errors like this are another reason syntactical sugar for readability is important.
- flohofwoe 4y agoYes it would need to be: if ((mask & (WGPUColorWriteMask_Alpha|WGPUColorWriteMask_Blue)) == (WGPUColorWriteMask_Alpha|WGPUColorWriteMask_Blue)) { ... } ...which is quite a mouthful.
- deleted 4y ago[deleted]
- tempodox 4y agoIn C I'm using a macro for stuff like that (improving readability, avoiding errors). In Zig I could use a `comptime` function, I presume?
- messe 4y agoIt depends on what else you’re doing. In this situation normal function would suffice in either C or Zig.
- chrismorgan 4y agoI’ve definitely desired an operator that combines these before. I’ve settled on &== as the best syntax in C-style languages, with a &== b being equivalent to a & b == b (but with a single evaluation of b). if (mask &== WGPUColorWriteMask_Alpha|WGPUColorWriteMask_Blue) { … } I find only two very minor problems with it: firstly, that it’s not commutative (that is, a &== b is not equivalent to b &== a). Secondly, that it can be confused with the bitwise-and assignment operator &= (a &= b being equivalent to a = a & b for singly-evaluated lvalue a). I’d then add |== for consistency and because it is conceivably useful. ^== is tempting, but since a ^== b would be just another spelling of a == 0, I’d skip it.
- shultays 4y agoSo `unsigned flag:1`?
- infradig 4y agoI just use 'bool flag:1' in C.
- ok123456 4y agoAfter you set all the pragmas or attributes to not add any additional padding/alignment, and that you really, really mean it. Maybe add a few static asserts so you know at compile time if you got the size right. These are not standard, so you need some preprocessor magic to choose the right thing. And so on...
- flohofwoe 4y agoOne area where this is still less flexible than regular bit masking is that quite often I want to access groups of bits as a value. For instance if the bits represent in/out pins on a chip emulator, sometimes I want to access unique data bus pins, and sometimes I want to set or get the data bus value. Maybe Zig can handle this case with unions though, I haven't tried this yet (this would require that unions can work on the 'bit level' in packed structs).
- elcritch 4y agoI've come to really like Nim's built-in `set` type for that sorta scenario. It's just a bit-vector but it took me months before I realized that. You can cast an `int32` to a `set[MyEnum]` and be able to do set unions, differences, etc. Makes it easy to define a mask just by `const myMask = {A, B, E}`. Making a Zig BitSet would probably be doable, for those cases where bitfields are overkill. 1: https://nim-lang.org/docs/manual.html#types-set-type https://nim-lang.org/docs/manual.html#types-set-type
- alphazino 4y agoI don't have experience with this area, but... > this would require that unions can work on the 'bit level' in packed structs Just as Zig has packed structs, it also has packed unions. So that part shouldn't be an issue.
- Someone 4y agoSwift has OptionSet. See https://developer.apple.com/documentation/swift/optionset https://developer.apple.com/documentation/swift/optionset. It allows setting, testing, or clearing multiple fields with a nice syntax. One thing that is a bit noisy is that you have to specify the bit index when you name the individual bits, as in (from that article): static let secondDay = ShippingOptions(rawValue: 1 << 1) Upside from that is that it makes it clear what bit each value specifies, and won’t easily accidentally change them when you reorder definitions, or insert or remove them. I can see why they made that choice. I also guess OptionSet could have been implemented in Swift in a third party library, whereas this Zig feature cannot. I also think/guess neither language guarantees multiple bits would get read or written in one instruction. If your hardware needs that, you probably have to go down a level, or look at the disassembly of your code to check what your compiler did.
- andolanra 4y ago> This all works, people have been doing it for years in C, C++, Java, Rust, and more. In Zig, we can do better. We can also do better in those other languages, too. For example, in Rust, I can use a crate like `bitfield` which gives me a macro with which I can write bitfield! { pub struct Color(u32); red, set_red: 0; green, set_green: 1; blue, set_blue: 2; alpha, set_alpha: 3; } Don't get me wrong: it's cool that functionality like this is built-in in Zig, since having to rely on third-party functionality for something like this is not always what you want. But Zig is not, as this article implies, uniquely capable of expressing this kind of thing.
- deleted 4y ago[deleted]
- zppln 4y agoIs it something I'd ever want to rely on third-party functionality for?
- masklinn 4y agoYes? If you have lots of bitfields and the convenience / readability is worth it, why wouldn’t you?
- zppln 4y agoI meant as opposed to having it built in.
- masklinn 4y agoThere are lots of reasons to not want it built-in e.g. it makes the language more complex, and if bad semantics are standardised you’re stuck with them. If the language supports implementing a feature externally then it’s a good thing, as it allows getting wide experience with the feature without saddling the language with it, and if the semantics are fine and it’s in wide-spread use, then nothing precludes adding it to the language later on. It’s much easier to add a feature to a language than to remove it.
- lsof 4y agoThe language I'm working on has the following syntax bitfield ColorWriteMaskFlags : u32 { red 1 bool, green 1 bool, blue 1 bool, alpha 1 bool, }; The number after the field name is the bit size. An (offset, size) pair can be used instead (offsets start at 0 when not explicit). After that can be nothing (the bit field is an unsigned number), or the word 'bool' (the bit field is a boolean) or the word 'signed' (the bit field is a signed number in two's complement). The raw value of the bitfield can always be accessed with 'foo.#raw'. EDIT: There's also no restriction to how multiple fields can overlap, as long as they all fit within the backing type.
- masklinn 4y agoThe one thing that would worry me here is the endianness e.g. is red the LSB or the MSB? Also what happens to the padding (bits not covered by subfields)? How’s that going to look when shoved into a file or over a socket? how does the langage handle overflow (more bits in the bit fields than there are in the parent field)?
- lsof 4y agoRed is the least significant bit, because offsets start at bit #0. Endianness is not a concern because you have to specify the underlying integer type, and bit endianness is not a thing, so it's just the native endianness (if you, e.g., need a network format with specific endianness and a 32-bit bitfield, you can instead use 4 consecutive 8-bit bitfields laid out in a consistent way). Unspecified bits are ignored; they'll be zero if the bitfield is initialized through conventional means but if they can be accessed directly through the raw value (or e.g. through memcpy, pointer casting, and any other direct memory access means), but when reading a field only its specified bits are read so unspecified bits doesn't change that result. When writing to a field, the source value is truncated to the field size, so you never end up writing to other bits.
- masklinn 4y ago> Red is the least significant bit, because offsets start at bit #0. So red is the LSB because you decided it was the LSB. That is not a by-definition thing. > Endianness is not a concern because you have to specify the underlying integer type, and bit endianness is not a thing That’s not actually true. There are formats which process bytes LSB to MSB, and formats which process them MSB to LSB. E.g. git’s offset-encoding, the leading byte is a bitmap of continuation bytes, bit 0 indicates whether byte 7 is present. Both are perfectly justifiable, one is offset-based, while the other is visualisation-based as bytes are usually represented MSB-first (as binary numbers, essentially). > but when reading a field only its specified bits are read so unspecified bits doesn't change that result. When writing to a field, the source value is truncated to the field size, so you never end up writing to other bits. I’m quite confused by “bitfield” do you mean the container field (the one that’s actually defined by the `bitfield` keyword) or the contained sub-fields?
- stephc_int13 4y agoIn my opinion, syntactic sugar is, in fine, more useful and positive than "powerful" or "expressive" features that can lead to shorter but harder to decipher code. Syntactic sugar neve hurts.
- laserbeam 4y agoI wouldn't say this is syntactic sugar here. packed structs are the most valuable tool in the language to handle interop with other languages and building protocols. The fact that you can use that tool to build bitfields feels like an interesting, cool and useful side effect. However, it's not their purpose. Note that the main difference between packed structs and regular structs is not the dense bit packing, rather that regular structs are allowed to reorder the fields however they wish in memory. The compiler is free to optimize your struct. It's not free to do so in other languages (like C for example). Thus you get a way to define structs exactly, with bit level precision, to build complex protocols where you can decide what every bit means.