5 ms·

This is a good list of positives. I'm surprised there's no mention of bitfields though. I like the u32::from_le_bytes and into() and so on, but I feel like Rus

by boulos 6y ago

This is a good list of positives. I'm surprised there's no mention of bitfields though. I like the u32::from_le_bytes and into() and so on, but I feel like Rust would be a perfect match for explicit "u3" types when dealing with low-level protocols. https://immunant.com/blog/2020/01/bitfields/ https://immunant.com/blog/2020/01/bitfields/ and similar articles show some options, but none of them feel as natural (ergonomics wise) or even as "safe" as a C bitfield checked by clang. bcantrill (or others): am I just missing something here?

bcantrill 6y ago
No, you're not missing something -- this is an area that I would say is (to use a winter weather term) "unsettled." The Immunant blog entry you pointed to is quite good, and lays out all of the existing solutions and the issues with them. Better solutions are definitely possible here, and I'm confident that we'll see them over time. Especially given the incredible obstacles that Rust has overcome with respect to low-level software (e.g., no_std, asm!), making bitfields more ergonomic seems like a very reasonable aspiration, especially considering how many others see the same problem!
- boulos 6y ago
  Re-reading the end of that post, I’m reminded that Josh had a talk about potential improvements. I only return to Rust about once or twice a year (currently) and keep cutting myself on an ergonomics thing like this. But I’ve also seen it improve year over year at a remarkable pace. So I’m hopeful, but also always wondering “huh, am I holding the phone wrong?”.
steveklabnik 6y ago
(I work with bcantrill) We use https://crates.io/crates/bitfield https://crates.io/crates/bitfield, and some of our dependencies use https://crates.io/crates/bitflags https://crates.io/crates/bitflags. They're not perfect, but they get the job done.
- boulos 6y ago
  Disclaimer: I haven’t used bitflags (but I don’t think it applies for my need). The nginx example in the above blog post seems like a bitfield! equivalent would be... unpleasant. For very small structs that are just only wire formats, I might use bitfield!. But encoding/decoding protocol headers (e.g., a VP8 header [1], which has a 19-bit “size” field) or interop with other systems where you’d like some checking a la serde, bitfield! seems more verbose than “fine, I’ll do it myself right here against the u8”. That is, even if that nginx example had a two-bit field in there, I’d take everything at the end and round up to a u8. Then read/write some of these “by hand”, rather than pollute the struct with lots and lots of macro “noise”. [1] https://tools.ietf.org/html/rfc6386#section-9.1 https://tools.ietf.org/html/rfc6386#section-9.1
  - steveklabnik 6y ago
    Yes, this is what I mean by "it's not perfect", there's totally things that could be better. All I'm saying is that this works out well enough for our particular needs generally, and isn't a giant pain point, so that's probably why it wasn't mentioned.
cryptonector 6y ago
Be careful with C and bitfields, as they can be a minefield too. First, always use an unsigned integral type for them, and second, be careful with bitfield packing rules -- maybe just use 1-bit fields only even.
necubi 6y ago
I have an (in-progress, but usable) project for generating rust code for parsing and generating these kinds of protocols: https://github.com/mwylde/protogen/tree/to_vec https://github.com/mwylde/protogen/tree/to_vec. It supports integers of arbitrary bit sizes up to 64.
- boulos 6y ago
  Cool! I was curious to see what you chose to do for representing arbitrary bit width. I didn't see an example / test for something like u19 though. Your backend compilation seems to generate calls to an arbitrary width [1], but I only see explicit 8/16/32/64s generated or tested [2]. The code definitely seems ready for size != width, and answering some of my question "sizes less than 8 round up to u8, otherwise round up to nearest power of 2" [3]. Either way, it looks like a great start! Your little language seems like it probably has enough in it already for my VP8 example. [1] https://github.com/mwylde/protogen/blob/f812dd74e5cebe680422ae2382e97eb487e78c68/protogen-compiler/src/backend.rs#L325 https://github.com/mwylde/protogen/blob/f812dd74e5cebe680422... [2] https://github.com/mwylde/protogen/blob/f812dd74e5cebe680422ae2382e97eb487e78c68/protogen/src/lib.rs#L340 https://github.com/mwylde/protogen/blob/f812dd74e5cebe680422... [3] https://github.com/mwylde/protogen/blob/f812dd74e5cebe680422ae2382e97eb487e78c68/protogen-compiler/src/backend.rs#L241 https://github.com/mwylde/protogen/blob/f812dd74e5cebe680422...

by boulos 6y ago

bcantrill 6y ago

No, you're not missing something -- this is an area that I would say is (to use a winter weather term) "unsettled." The Immunant blog entry you pointed to is quite good, and lays out all of the existing solutions and the issues with them. Better solutions are definitely possible here, and I'm confident that we'll see them over time. Especially given the incredible obstacles that Rust has overcome with respect to low-level software (e.g., no_std, asm!), making bitfields more ergonomic seems like a very reasonable aspiration, especially considering how many others see the same problem!

boulos 6y ago
Re-reading the end of that post, I’m reminded that Josh had a talk about potential improvements. I only return to Rust about once or twice a year (currently) and keep cutting myself on an ergonomics thing like this. But I’ve also seen it improve year over year at a remarkable pace. So I’m hopeful, but also always wondering “huh, am I holding the phone wrong?”.

steveklabnik 6y ago

(I work with bcantrill) We use https://crates.io/crates/bitfield https://crates.io/crates/bitfield, and some of our dependencies use https://crates.io/crates/bitflags https://crates.io/crates/bitflags. They're not perfect, but they get the job done.

boulos 6y ago
Disclaimer: I haven’t used bitflags (but I don’t think it applies for my need). The nginx example in the above blog post seems like a bitfield! equivalent would be... unpleasant. For very small structs that are just only wire formats, I might use bitfield!. But encoding/decoding protocol headers (e.g., a VP8 header [1], which has a 19-bit “size” field) or interop with other systems where you’d like some checking a la serde, bitfield! seems more verbose than “fine, I’ll do it myself right here against the u8”. That is, even if that nginx example had a two-bit field in there, I’d take everything at the end and round up to a u8. Then read/write some of these “by hand”, rather than pollute the struct with lots and lots of macro “noise”. [1] https://tools.ietf.org/html/rfc6386#section-9.1 https://tools.ietf.org/html/rfc6386#section-9.1
- steveklabnik 6y ago
  Yes, this is what I mean by "it's not perfect", there's totally things that could be better. All I'm saying is that this works out well enough for our particular needs generally, and isn't a giant pain point, so that's probably why it wasn't mentioned.

cryptonector 6y ago

Be careful with C and bitfields, as they can be a minefield too. First, always use an unsigned integral type for them, and second, be careful with bitfield packing rules -- maybe just use 1-bit fields only even.

necubi 6y ago

I have an (in-progress, but usable) project for generating rust code for parsing and generating these kinds of protocols: https://github.com/mwylde/protogen/tree/to_vec https://github.com/mwylde/protogen/tree/to_vec. It supports integers of arbitrary bit sizes up to 64.

boulos 6y ago
Cool! I was curious to see what you chose to do for representing arbitrary bit width. I didn't see an example / test for something like u19 though. Your backend compilation seems to generate calls to an arbitrary width [1], but I only see explicit 8/16/32/64s generated or tested [2]. The code definitely seems ready for size != width, and answering some of my question "sizes less than 8 round up to u8, otherwise round up to nearest power of 2" [3]. Either way, it looks like a great start! Your little language seems like it probably has enough in it already for my VP8 example. [1] https://github.com/mwylde/protogen/blob/f812dd74e5cebe680422ae2382e97eb487e78c68/protogen-compiler/src/backend.rs#L325 https://github.com/mwylde/protogen/blob/f812dd74e5cebe680422... [2] https://github.com/mwylde/protogen/blob/f812dd74e5cebe680422ae2382e97eb487e78c68/protogen/src/lib.rs#L340 https://github.com/mwylde/protogen/blob/f812dd74e5cebe680422... [3] https://github.com/mwylde/protogen/blob/f812dd74e5cebe680422ae2382e97eb487e78c68/protogen-compiler/src/backend.rs#L241 https://github.com/mwylde/protogen/blob/f812dd74e5cebe680422...