5 ms·
Deserializing Binary Data Files in Rust
- mirekrusin 5y agoI think c struct is missing alignment attribute? This idea has been explored in flatbuffers, capnproto and probably others.
- masklinn 5y agoC structs are generally assumed to be laid out in the order defined with proper padding automatically inserted (as most platforms at best dislike unaligned accesses). Alignment information should only be necessary if you want to pack the structure or need to apply wider than standard alignment (e.g. need to align a 32 or 64b type to 128 bytes). The one possible issue is that reading padding is UB (i think). But here there should be no padding: the struct is a pair number of bytes (alignment 1) and a word (alignment 2), which us aligned due to the previous, and the struct is if alignment 2.
- gameswithgo 5y agoyeah C will be order defined in code, rust will reorder to save space unless you attribute it not to. interesting trivia a blind guy wrote the reordering code. if anyone knows his name please post.
- masklinn 5y agoThe rust structure is defined as repr(C) so there is no reordering.
- sumtechguy 5y agoDoes the C spec actually specify it needs to be in that order? Been awhile since I messed that that sort of thing. I know in practice they typically just put them in the same order and align on some sort of default packing depending on CPU. Usually to minimize on extra instructions. I have eeked out some extra perf on some CPUs and structs just by moving stuff around and making sure it fits into a cacheline. Sometimes the extra instructions are worth it to unpack some vars if you can get the data into a cacheline. But that depends on your data and usage.
- kzrdude 5y agoYes, it does
- kzrdude 5y agoImplemented in https://github.com/rust-lang/rust/pull/37429 https://github.com/rust-lang/rust/pull/37429 ; his website seems to be https://ahicks.io/ https://ahicks.io/
- diath 5y agoThat's his point, you want to 1-align the struct to disable the padding to ensure the size of the structure is portable.
- masklinn 5y agoThat makes no sense? There isn’t any padding, and if the bytes are not 8 bits it’ll never work anyway.
- alfalfasprout 5y agoWider than standard alignment can be pretty important though. Especially for usage with SIMD instructions.
- masklinn 5y agoSure but that has no relevance whatsoever to the article do the objection is irrelevant.
- Labo333 5y agoFor such a problem, I really recommend https://kaitai.io/ https://kaitai.io/ From their website: > Kaitai Struct is a declarative language used to describe various binary data structures, laid out in files or in memory: i.e. binary file formats, network stream packet formats, etc. They even have a Rust interface: https://github.com/kaitai-io/kaitai_struct_rust_runtime https://github.com/kaitai-io/kaitai_struct_rust_runtime
- rudibe 5y agoA native Rust solution, with meta-programming: https://crates.io/crates/packed_struct https://crates.io/crates/packed_struct (written by me)
- coldnose 5y agoI really wish there was a safe way to initialize structs with arbitrary binary data in rust. Perhaps I'm missing something, but it seems intuitively safe, especially if the struct only contains integer primitives.
- sophacles 5y agoHow would it be possible to make it safe? The compiler can't verify that N random bytes input from an arbitrary source are going to be valid for the types in the struct - you have to tell tell the compiler "trust this even if you can't verify it from the source code" - hence the unsafe keyword.
- Alexendoo 5y agoEnter safe transmute - https://github.com/rust-lang/project-safe-transmute/blob/master/rfcs/0000-safe-transmute.md https://github.com/rust-lang/project-safe-transmute/blob/mas...
- sophacles 5y agoOh cool! Thanks for the pointer.
- coldnose 5y agoFor integer primitives, all possible values are valid. Just allowing binary initialization for structs that consist of integer types would allow compatibility with C structure. As it is, I have to either resort to unsafe code or manually deserialize every struct
- sophacles 5y agoOh I see, that makes sense. Also there's a sibling reply to yours for a cool looking proposal to do just that if you haven't seen it yet.
- crateless 5y ago> We get lucky here because the flags field is at offset 202 How does one calculate this offset based on the fields in the struct?
- masklinn 5y agoJust add all the sizes, and account for padding (there isn’t any here because everything other than the flags field is bytes arrays)
- monad_june 5y agoBy the way there is an easier way to convert C style null terminated strings into an &str than the bespoke c_string function used in this article. You can use the CStr type that is already in the rust standard library: https://doc.rust-lang.org/std/ffi/struct.CStr.html https://doc.rust-lang.org/std/ffi/struct.CStr.html