5 ms·
Sending structures that include padding or that you don’t know the exact layout of over the network is another thing you shouldn’t do, period. Code doing that
by Someone 8y ago
Sending structures that include padding or that you don’t know the exact layout of over the network is another thing you shouldn’t do, period.
Code doing that may break with a compiler change or a compilation flag change on one end of the connection, even if you can guarantee both ends use the same CPU.
- mehrdadn 8y ago> Sending structures that include padding or that you don’t know the exact layout of over the network is another thing you shouldn’t do, period. What if you do know the exact layout and it happens to have a gap between fields?
- noobermin 8y agoTbh I'm not a network person, but my naive brain says instead of casting to char* and writing to the buffer, couldn't you just write out individual fields to a buffer so you control every byte? That seems smarter and safer.
- mehrdadn 8y agoDo you write a separate serialization function for every single struct and sub-struct you send over a network? And then continue maintaining every single one to keep it consistent whenever a field is added/removed/modified in any struct? (Though I should mention that macros actually do make this doable, but they're hardly pretty, and I have yet to see many people embrace this approach, if they realize it's possible at all.)
- kllrnohj 8y ago> Do you write a separate serialization function for every single struct and sub-struct you send over a network? And then continue maintaining every single one to keep it consistent whenever a field is added/removed/modified in any struct? Yes? Of course you do that? Or pick one of a hundred libraries that do that for you - protobuf, cap'n proto, flatbuffers, etc...
- repsilat 8y agoThat's the protobuf approach, but things like Capnp are different, right? Isn't its wire format its in-memory format? I thought it was closer to the "straight memcpy" approach. Protobuf serialization functions can turn into a lot of generated code too (read: big binaries) and stomp all over your icache, because they're different for each type. More "data-driven" approaches using type-generic functions with some runtime type-specific data (like proto descriptors&reflection) can be a good tradeoff for code that isn't as "hot". Typically more branchy than a memcpy or a type-specific function though.
- shaklee3 8y agoThere are dozens, if not hundreds of serialization libraries. None of them are going to compare to something like sending a raw struct on the wire, if you need absolute performance. Probably the biggest advantage of things like grpc, flatbuffers, capnproto, etc, are they provide serialization libraries for many different languages. Sending a raw struct will be up to the implementer to get the other side correct. And by sticking with the serialization libraries with higher performance, like flatbuffers and capnproto, you also lose a little bit of future flexibility, since its typically harder to change the format of these structures compared to higher level libraries.
- kentonv 8y agoAuthor of Cap'n Proto here... > None of them are going to compare to something like sending a raw struct on the wire, if you need absolute performance. I don't know about that. For simple flat structs containing integers only (no pointers), Cap'n Proto is nearly identical to raw structs. For complex structures with pointers, sending raw structs doesn't work. Admittedly you may suffer unwanted code bloat linking in the Cap'n Proto library if all you really want to do is send a flat, raw struct. > And by sticking with the serialization libraries with higher performance, like flatbuffers and capnproto, you also lose a little bit of future flexibility, since its typically harder to change the format of these structures compared to higher level libraries. I disagree with this. Cap'n Proto's compatibility story is essentially the same as Protobuf and JSON: You can add new fields and old programs will ignore them. (I suspect Flatbuffers is also similar but I haven't looked at it in a long time.)
- a_t48 8y agoIt depends. It was common case for years in games to just cast your networked data to an array and send over the network. Nowadays it's much more in fashion to pick a serialization library to sit in between, as PCs are faster. But one might choose not to depending on the problem space. Embedded devices might not have the spare cycles or the space for a full serialization library. So...you're not wrong, but it's basically the same reason people use C over something with more ergonomics. Edit: the last time I wrote a networking library, I also just cast to raw structures, mainly because it was less work (didn't want to integrate a serialization library on both sides of the link). This was fine for a while (and very fast), but also hid some bugs in the protocol, and made sending arrays of data very ugly. Eventually we put protobuf on top, and things ended up much nicer.
- DaiPlusPlus 8y agoHow does sending raw memory to the socket work if you have a struct filled with pointers?
- taneq 8y agoThat's what I do these days for simple stuff, or for stuff which needs to talk to nonstandard hardware (eg. talking to a PLC, where I need to know the exact bit layout of the packet in order to unpack it manually on the other side). If you're doing a large project on supported platforms it's probably worth using a pre-existing serialisation library like protobuffers or something.
- CJefferson 8y agoChanging compiler or flags won't change structs, unless you do something crazy. If it did, you couldn't link C libraries compiled with one compiler with programs compiled on another. You can't be sure you can send such structures to different OSes or CPUs however.
- spc476 8y agoIt depends upon the architecture and compiler. One C compiler for the MC68000 might have int as 16 bits (since externally, the MC68000 had a 16-bit external bus) but another one have ints as 32 bits (since the MC68000 can handle 32-bits internally). This was more of an issue during the 80s and 90s when you had more commercial C compiler available than today.
- Gibbon1 8y agoThat is what stdint is for.
- saagarjha 8y ago> If it did, you couldn't link C libraries compiled with one compiler with programs compiled on another. You cannot do this unless the compilers have the same ABI, which is not part of the standard.
- anyfoo 8y agoThere is an attribute that lets you specify alignment per-struct, which means you will know the exact layout.
- hermitdev 8y agoBut it is still potentially problematic if not all of your clients have the same endianess. Memcpy and exact layout still fails you there.
- v_lisivka 8y agoRDMA (Infiniband) does DMA to/from network. Why we should not use RDMA?
- catern 8y agoHmm, that concern is not just limited to RDMA; any kind of shared memory communication across a trust boundary is vulnerable to info leaks in the padding of the struct. That does seem like a real issue. Not sure what's the appropriate mitigation. Having to memset is a drag, and it's easy to forget to do it in some place or another. I'm tempted to say that you should only use packed structs for shared memory communication across trust boundaries.
- gmueckl 8y agoWell, Infiniband is intended mainly for supercomputers. So security is of no real concern within the Imfiniband network as that is implicitly a trusted environment. Access comtrol to that network is the main security boundary. Edit: Also, lots of simulation codes I am aware of tend to send flat arrays anyway. So I'd say that your concerns are mostly theoretical in nature.
- shaklee3 8y agoRDMA, both roce and infiniband have some significant limitations, specifically at higher rates. But they're the best we have right now for direct dma.
- 0x8BADF00D 8y agoSerialization is a non-trivial problem. Especially when dealing with different OS, network stacks, and architectures. protobuf alleviates this quite a bit.
- johncolanduoni 8y agoThis kind of issue happens all the time with structs that a kernel, driver etc. expose to userspace: https://j00ru.vexillium.org/papers/2018/bochspwn_reloaded.pdf https://j00ru.vexillium.org/papers/2018/bochspwn_reloaded.pd... Not sending the structure over the network is insufficient, and rigorous memset is a clear solution to this problem. Also, C structure layout for a specific ABI is strictly determined; if that weren’t the case you couldn’t compile two C compilation units with different flags/compilers and expect them to work.
- blattimwind 8y agoAlso don't expect memset() to actually be a library call at all.
- rurban 8y agoAnd most importantly dont expect memset to be executed at all. Most -O2 compilers happily optimize the memset away if if doesn't care about the sideeffects. Therefore there exists the "secure" memset variants, which are the insecure variants adding a simple compiler barrier, so it isn't optimized away. And there exists the very few really secure memset_s variants with a real memory barrier which guarantees memset to be executed in load and store order, flushing the cache, so that Spectre attacks are mitigated. security-relevant crypto libraries, like openssl and variants don't care about that, because memory barriers are "too slow". only the Linux kernel does it properly. It's still a mess.
- FrozenVoid 8y agoyou can write your own memset that bypasses all of that. Its just 2 lines of code.
- gmueckl 8y agoAnd that will be slower than calling memset() unless you have a really smart compiler.
- 8y ago