9 ms·

Type-safe Bitmasks in C++

97 points by nice_byte 8y ago

jerrre 8y ago
edit: did not read correctly: https://godbolt.org/g/XjdL7i https://godbolt.org/g/XjdL7i - uint8_t controller_state = BTN_A | BTN_B; if I read correctly this example would not be possible anymore right? bitmask<Button> controller_state = Button::A | Button::B; or bitmask<Button> controller_state { Button::A | Button::B }; won't work.
- nice_byte 8y ago
  that's what the operator| at the end is for
omgtehlion 8y ago
This is all good and nice. But in my opinion it violates principle of least surprise: when I encounter any enum, am I able to OR two values? These do not seem like flags, is this correct usage? When a function expects bitmask<T>, I should know that T is just flag positions. But when I do not have some API point, only enum, how am I supposed to know that? Plus, in case of interop with other languages, even with C, I will have Button::B == 2 in CPP, and BUTTON_B == 4 in C (or Button.B == 4 in Java). This of course adds to confusion...
8xde0wcNwpslOw 8y ago
Bitfields would be, in theory, safer still: no manual bit-twiddling to begin with and also quite straightforward usage. Unfortunately, they are (and I guess will remain) underspecified as far as portability is concerned. You can, of course, do something similar with member functions, e.g. class controller { unsigned value; static constexpr unsigned BTN_A = 0x01; ... public: bool a() const { return value & BTN_A; } void a(bool set) { if (set) value |= BTN_A; else value &= ~BTN_A; } };
- inetknght 8y ago
  This is pretty similar to what I've done in the past
TheCycoONE 8y ago
The article claims no runtime overhead, but my understanding is that despite the consexpr that function needs to be evaluated at runtime because the inputs are unknown?
- vostok 8y ago
  In many cases the input is known at compile time. For example instead of #define BTN_A 0x01 you have bitmask<Button>(A) Both of these are known at compile time and they are equivalent.
- trebeling 8y ago
  Constexpr aside any modern optomizer will handle this easily. Constexpr is more for allowing results to be passed to templates, static_assert, etc.. and less about optimization imo
pjmlp 8y ago
With C++17 I would vote for using static_assert and constexpr if for type validation instead of SFINAE with enable_if. Other than that, nice article.
pwagland 8y ago
Nice technique, however, they don't allow you to do everything that a bitmask does. For example, there is no way to clear a flag, or to select a subset of flags. That is, given: enum class Button { A, B, Select, Start, Up, Down, Right, Left }; There is no way to define: bitmask<Button> controller_state { Button::A | Button::B }; And then to find out if either of these buttons are pressed. The only way to do it is: bool fn2(bitmask<Button> controller_state) { return controller_state & Button::A || controller_state & Button::B; } which _does_ generate the optimal code for checking if either is set. However you can't do something like void handleDirectionButtons(bitmask<Button> controller_state); and then call that with something like: bitmask<Button> direction_buttons { Button::Up | Button::Down | Button::Right | Button::Left }; handleDirectionButtons(state & directionButtons) Of course, this could be added, see https://godbolt.org/g/48asye https://godbolt.org/g/48asye You could also do something similar for the ~ case as well, however that would end up looking a bit uglier, as you would need to create more bit masks, something like: ~bitmask(Button::Up) & ~bitmask(Button::Down) For an example, see https://godbolt.org/g/wN2xUV https://godbolt.org/g/wN2xUV
pubby 8y ago
Something simpler is to put all bitmask enums in a namespace and use ADL to find the operators. Like this: https://pastebin.com/raw/ANh93JnN https://pastebin.com/raw/ANh93JnN Personally though, I wouldn't use this safety in most code. You'd have to be in some pretty serious shit for these templates to reduce mental overload rather than add it.
- zelos 8y ago
  I think it reduces some mental load because it documents the interface: you don't have to look up which flags are valid for which functions, the types tell you. Plus, with a small amount extra code you can do things like iterate over all the flags that are set etc.
  - jstimpfle 8y ago
    The best way to iterate and do stuff with enums that I know is this way: enum { FOO_A, FOO_B, FOO_C, NUM_FOOS, }; struct FooInfo { int kind; // FOO_?? const char *str; const char *long; }; static const struct FooInfo fooInfo[NUM_FOOS] = { { FOO_A, "FOO_A", "The A of all the foos" }, { FOO_B, "FOO_B", "The B of all the foos" }, { FOO_C, "FOO_C", "The C of all the foos" }, }; for (int i = 0; i < NUM_FOOS; i++) { printf("Got #%d, %s. Description: %s\n", fooInfo[i].kind, fooInfo[i].str, fooInfo[i].long); } More goodies: // if you want to use them as bitmasks // (but only few enums are used this way) enum { FOO_A_BIT = 1 << FOO_A, FOO_B_BIT = 1 << FOO_B, FOO_C_BIT = 1 << FOO_C, }; // if you want to make strings out of C symbols, you // can also use a macro like this: #define FOOINFO(name, long) { name, #name, long }, static const struct FooInfo fooInfo[NUM_FOOS] = { FOOINFO( FOO_A, "The A of all the foos" ), FOOINFO( FOO_B, "The B of all the foos" ), FOOINFO( FOO_C, "The C of all the foos" ), };
    - lomnakkus 8y ago
      Both of these suffer from the potential maintenance problem of duplicating the list of enumerated values. You can avoid that by using the X-Macro pattern, but that's quite heavy-handed and tedious to do if you have a lot of different enums. It also doesn't use "enum class" which means that the namespace will be polluted unnecessarily.
chabsf 8y ago
The fact that this relies on overloading an operator for every enum makes this a complete non-starter as is in any professional codebase unfortunately. That can be fixed with a trait type though.
pdkl95 8y ago
> Note that we don't explicitly set [enum values] to be powers of two > return 1 << static_cast<underlying_type>(o); Relying on default enum values to and automagically converting these into power of two bit masks is very brittle, and can create unexpected bugs and interoperability problems. The mask values change if the enum changes, unused bits in the middle of the bitmask are hard to handle, and the bit order is assumed with the first item being the LSB. The last point is particularly relevant given the NES buttons example, because it uses a bit order opposite to how the controller data is usually read[1]. This article defines #define BTN_A 0x01 /* 00000001 in binary */ #define BTN_B 0x02 /* 00000010 in binary */ /* ... */ #define BTN_RIGHT 0x80 /* 10000000 */ The standard controller[2] state is sent from a shift register in the controller one bit at a time. This is usually read by the program as 8 reads from the I/O port, and shifted into a button state bitmask. E.g. something like: n = 8; buttons = 0; while(buttons--) { buttons = buttons << 1; buttons |= (CONTROLLER_1_IO_PORT & 0x01); } With the buttons being read in the order [A, B, ..., RIGHT], the recommended bitmask is: BUTTON_A = 1 << 7 BUTTON_B = 1 << 6 /* ... */ BUTTON_RIGHT = 1 << 0 Obviously this is a minor mistake in a (probably untested) example. I just want to highlight how easily this type of "clever" - but brittle - coding style can introduce errors. Defining bit values (and other externally defined magic numbers) explicitly in e.g. a defined-to-be-canonical header file (which might be just those 8 #defines) is much safer and easier to debug/maintain. Then wrap those explicitly defined values into the C++ template for type safety or other fancy features. [1] https://wiki.nesdev.com/w/index.php/Controller_Reading https://wiki.nesdev.com/w/index.php/Controller_Reading [2] https://wiki.nesdev.com/w/index.php/Standard_controller https://wiki.nesdev.com/w/index.php/Standard_controller
- unwind 8y ago
  You meant: while (n--)
- pubby 8y ago
  Oh, the NES controller bit order is 100% arbitrary. There's no standard - it all depends on if the programmer decided to write a ROL or ROR instruction that day. But I agree with your point.
- 8y ago
elteto 8y ago
From the first comment on the page: "The non-member function operator overloads that return bitmask<option_type> will not be considered for overload resolution if the enums are in a different namespace than the bitmask definition. The template deduction of operator| would not be sufficient since it does not trigger ADL from outside the namespace -- the overload must be available from within the scope where it used. This effectively requires a using directive or declaration to make them visible to participate in resolution (and at which, this will attempt to participate in resolution of any enum type where it may be considered a better match)." Is this correct? Because if it is, C++ is bonkers...
- MaxBarraclough 8y ago
  Deep dark arcane corner-cases in C++? Say it ain't so!
- simias 8y ago
  C++ ADL is pretty bonkers indeed, or at least somewhat surprising if you don't know the rules very well. Thing is without ADL some patterns would be a lot harder to write (or use). I personally dislike overloading very much so I'm probably not the best person to defend it though.
Animats 8y ago
That's doing it the hard way. This is the language-standard way: struct gamecontroller { bool up: 1; bool down: 1; bool left: 1; bool right: 1; bool a: 1; bool b: 1; bool x: 1; bool y: 1; }; That's a structure with a size of one byte. Bit fields work just fine in C and C++, yet many programmers don't know this.
- kolpa 8y ago
  Does the OP offer benefits over the :1 way?
- dokem 8y ago
  Unfortunately this cannot be used to pack an integer that is sent over the wire or to produce a register for a hardware peripheral. Basically this is not a valid substitution for most use cases of bit packing and can only be used to save a bytes of memory without reverting to undefined behavior. ie, casting this memory to a byte/int is undefined. The language does not define that the first field will be the lsb, msb or some arbitrary bit. The reason "many programmers don't know this" is because its a mostly useless language feature.
  - Animats 8y ago
    Portability is a problem. There is this: Microsoft Specific The ordering of data declared as bit fields is from low to high bit. END Microsoft Specific[1] In practice, everybody does it that way. For fields only one byte wide, this isn't an issue for major compilers. NTP has been using this for decades without problems. But for structs more than one byte wide, there are little-endian/big endian problems. Just like with the numeric types. This is only an issue if you intend to write the bitfield to a network, file, or device. Locally, you usually don't care about the representation. [1] https://docs.microsoft.com/en-gb/cpp/cpp/cpp-bit-fields https://docs.microsoft.com/en-gb/cpp/cpp/cpp-bit-fields
  - kolpa 8y ago
    The OP also is not a valid substitution for "most" (by what metric, anyway?) uses of bit packing. The network wire doesn't need a C++ ABI-compatible object, and clearly isn't portable when communicating between systems running different software. For transmitting data over network, convert from local format to network format in transit. > can only be used to save a bytes of memory Yes, that's what "packing" is for.

97 points by nice_byte 8y ago

edit: did not read correctly: https://godbolt.org/g/XjdL7i https://godbolt.org/g/XjdL7i - uint8_t controller_state = BTN_A | BTN_B; if I read correctly this example would not be possible anymore right? bitmask<Button> controller_state = Button::A | Button::B; or bitmask<Button> controller_state { Button::A | Button::B }; won't work.

nice_byte 8y ago
that's what the operator| at the end is for

omgtehlion 8y ago

This is all good and nice. But in my opinion it violates principle of least surprise: when I encounter any enum, am I able to OR two values? These do not seem like flags, is this correct usage? When a function expects bitmask<T>, I should know that T is just flag positions. But when I do not have some API point, only enum, how am I supposed to know that? Plus, in case of interop with other languages, even with C, I will have Button::B == 2 in CPP, and BUTTON_B == 4 in C (or Button.B == 4 in Java). This of course adds to confusion...

8xde0wcNwpslOw 8y ago

Bitfields would be, in theory, safer still: no manual bit-twiddling to begin with and also quite straightforward usage. Unfortunately, they are (and I guess will remain) underspecified as far as portability is concerned. You can, of course, do something similar with member functions, e.g. class controller { unsigned value; static constexpr unsigned BTN_A = 0x01; ... public: bool a() const { return value & BTN_A; } void a(bool set) { if (set) value |= BTN_A; else value &= ~BTN_A; } };

inetknght 8y ago
This is pretty similar to what I've done in the past

TheCycoONE 8y ago

The article claims no runtime overhead, but my understanding is that despite the consexpr that function needs to be evaluated at runtime because the inputs are unknown?

vostok 8y ago
In many cases the input is known at compile time. For example instead of #define BTN_A 0x01 you have bitmask<Button>(A) Both of these are known at compile time and they are equivalent.
trebeling 8y ago
Constexpr aside any modern optomizer will handle this easily. Constexpr is more for allowing results to be passed to templates, static_assert, etc.. and less about optimization imo

pjmlp 8y ago

With C++17 I would vote for using static_assert and constexpr if for type validation instead of SFINAE with enable_if. Other than that, nice article.

pwagland 8y ago

Nice technique, however, they don't allow you to do everything that a bitmask does. For example, there is no way to clear a flag, or to select a subset of flags. That is, given: enum class Button { A, B, Select, Start, Up, Down, Right, Left }; There is no way to define: bitmask<Button> controller_state { Button::A | Button::B }; And then to find out if either of these buttons are pressed. The only way to do it is: bool fn2(bitmask<Button> controller_state) { return controller_state & Button::A || controller_state & Button::B; } which _does_ generate the optimal code for checking if either is set. However you can't do something like void handleDirectionButtons(bitmask<Button> controller_state); and then call that with something like: bitmask<Button> direction_buttons { Button::Up | Button::Down | Button::Right | Button::Left }; handleDirectionButtons(state & directionButtons) Of course, this could be added, see https://godbolt.org/g/48asye https://godbolt.org/g/48asye You could also do something similar for the ~ case as well, however that would end up looking a bit uglier, as you would need to create more bit masks, something like: ~bitmask(Button::Up) & ~bitmask(Button::Down) For an example, see https://godbolt.org/g/wN2xUV https://godbolt.org/g/wN2xUV

pubby 8y ago

Something simpler is to put all bitmask enums in a namespace and use ADL to find the operators. Like this: https://pastebin.com/raw/ANh93JnN https://pastebin.com/raw/ANh93JnN Personally though, I wouldn't use this safety in most code. You'd have to be in some pretty serious shit for these templates to reduce mental overload rather than add it.

zelos 8y ago
I think it reduces some mental load because it documents the interface: you don't have to look up which flags are valid for which functions, the types tell you. Plus, with a small amount extra code you can do things like iterate over all the flags that are set etc.
- jstimpfle 8y ago
  The best way to iterate and do stuff with enums that I know is this way: enum { FOO_A, FOO_B, FOO_C, NUM_FOOS, }; struct FooInfo { int kind; // FOO_?? const char *str; const char *long; }; static const struct FooInfo fooInfo[NUM_FOOS] = { { FOO_A, "FOO_A", "The A of all the foos" }, { FOO_B, "FOO_B", "The B of all the foos" }, { FOO_C, "FOO_C", "The C of all the foos" }, }; for (int i = 0; i < NUM_FOOS; i++) { printf("Got #%d, %s. Description: %s\n", fooInfo[i].kind, fooInfo[i].str, fooInfo[i].long); } More goodies: // if you want to use them as bitmasks // (but only few enums are used this way) enum { FOO_A_BIT = 1 << FOO_A, FOO_B_BIT = 1 << FOO_B, FOO_C_BIT = 1 << FOO_C, }; // if you want to make strings out of C symbols, you // can also use a macro like this: #define FOOINFO(name, long) { name, #name, long }, static const struct FooInfo fooInfo[NUM_FOOS] = { FOOINFO( FOO_A, "The A of all the foos" ), FOOINFO( FOO_B, "The B of all the foos" ), FOOINFO( FOO_C, "The C of all the foos" ), };
  - lomnakkus 8y ago
    Both of these suffer from the potential maintenance problem of duplicating the list of enumerated values. You can avoid that by using the X-Macro pattern, but that's quite heavy-handed and tedious to do if you have a lot of different enums. It also doesn't use "enum class" which means that the namespace will be polluted unnecessarily.

chabsf 8y ago

The fact that this relies on overloading an operator for every enum makes this a complete non-starter as is in any professional codebase unfortunately. That can be fixed with a trait type though.

pdkl95 8y ago

> Note that we don't explicitly set [enum values] to be powers of two > return 1 << static_cast<underlying_type>(o); Relying on default enum values to and automagically converting these into power of two bit masks is very brittle, and can create unexpected bugs and interoperability problems. The mask values change if the enum changes, unused bits in the middle of the bitmask are hard to handle, and the bit order is assumed with the first item being the LSB. The last point is particularly relevant given the NES buttons example, because it uses a bit order opposite to how the controller data is usually read[1]. This article defines #define BTN_A 0x01 /* 00000001 in binary */ #define BTN_B 0x02 /* 00000010 in binary */ /* ... */ #define BTN_RIGHT 0x80 /* 10000000 */ The standard controller[2] state is sent from a shift register in the controller one bit at a time. This is usually read by the program as 8 reads from the I/O port, and shifted into a button state bitmask. E.g. something like: n = 8; buttons = 0; while(buttons--) { buttons = buttons << 1; buttons |= (CONTROLLER_1_IO_PORT & 0x01); } With the buttons being read in the order [A, B, ..., RIGHT], the recommended bitmask is: BUTTON_A = 1 << 7 BUTTON_B = 1 << 6 /* ... */ BUTTON_RIGHT = 1 << 0 Obviously this is a minor mistake in a (probably untested) example. I just want to highlight how easily this type of "clever" - but brittle - coding style can introduce errors. Defining bit values (and other externally defined magic numbers) explicitly in e.g. a defined-to-be-canonical header file (which might be just those 8 #defines) is much safer and easier to debug/maintain. Then wrap those explicitly defined values into the C++ template for type safety or other fancy features. [1] https://wiki.nesdev.com/w/index.php/Controller_Reading https://wiki.nesdev.com/w/index.php/Controller_Reading [2] https://wiki.nesdev.com/w/index.php/Standard_controller https://wiki.nesdev.com/w/index.php/Standard_controller

unwind 8y ago
You meant: while (n--)
pubby 8y ago
Oh, the NES controller bit order is 100% arbitrary. There's no standard - it all depends on if the programmer decided to write a ROL or ROR instruction that day. But I agree with your point.
8y ago

elteto 8y ago

From the first comment on the page: "The non-member function operator overloads that return bitmask<option_type> will not be considered for overload resolution if the enums are in a different namespace than the bitmask definition. The template deduction of operator| would not be sufficient since it does not trigger ADL from outside the namespace -- the overload must be available from within the scope where it used. This effectively requires a using directive or declaration to make them visible to participate in resolution (and at which, this will attempt to participate in resolution of any enum type where it may be considered a better match)." Is this correct? Because if it is, C++ is bonkers...

MaxBarraclough 8y ago
Deep dark arcane corner-cases in C++? Say it ain't so!
simias 8y ago
C++ ADL is pretty bonkers indeed, or at least somewhat surprising if you don't know the rules very well. Thing is without ADL some patterns would be a lot harder to write (or use). I personally dislike overloading very much so I'm probably not the best person to defend it though.

Animats 8y ago

That's doing it the hard way. This is the language-standard way: struct gamecontroller { bool up: 1; bool down: 1; bool left: 1; bool right: 1; bool a: 1; bool b: 1; bool x: 1; bool y: 1; }; That's a structure with a size of one byte. Bit fields work just fine in C and C++, yet many programmers don't know this.

kolpa 8y ago
Does the OP offer benefits over the :1 way?
dokem 8y ago
Unfortunately this cannot be used to pack an integer that is sent over the wire or to produce a register for a hardware peripheral. Basically this is not a valid substitution for most use cases of bit packing and can only be used to save a bytes of memory without reverting to undefined behavior. ie, casting this memory to a byte/int is undefined. The language does not define that the first field will be the lsb, msb or some arbitrary bit. The reason "many programmers don't know this" is because its a mostly useless language feature.
- Animats 8y ago
  Portability is a problem. There is this: Microsoft Specific The ordering of data declared as bit fields is from low to high bit. END Microsoft Specific[1] In practice, everybody does it that way. For fields only one byte wide, this isn't an issue for major compilers. NTP has been using this for decades without problems. But for structs more than one byte wide, there are little-endian/big endian problems. Just like with the numeric types. This is only an issue if you intend to write the bitfield to a network, file, or device. Locally, you usually don't care about the representation. [1] https://docs.microsoft.com/en-gb/cpp/cpp/cpp-bit-fields https://docs.microsoft.com/en-gb/cpp/cpp/cpp-bit-fields
- kolpa 8y ago
  The OP also is not a valid substitution for "most" (by what metric, anyway?) uses of bit packing. The network wire doesn't need a C++ ABI-compatible object, and clearly isn't portable when communicating between systems running different software. For transmitting data over network, convert from local format to network format in transit. > can only be used to save a bytes of memory Yes, that's what "packing" is for.