5 ms·
I write a lot of low level C++ network/disk stuff. I need minimal decoding costs and rely on the type system to catch message differences asap. The following
by CoolGuySteve 7y ago
I write a lot of low level C++ network/disk stuff. I need minimal decoding costs and rely on the type system to catch message differences asap. The following is an example of an append only format that uses the sizeof as a kind of version.
In practice, not only is this infinitely faster performance-wise, but I find maintaining this is about the same amount of work as Protobufs or JSON but with less fussy tooling issues.
By using X macros or BOOST_FUSION for the struct definitions, you can infer the type definition of message fields and automatically write serializers for JSON, SQL, CSV, Python, or whatever schema format is your favourite. Unfortunately, all that macro/variadic magic is too verbose and fiddly for a hnews comment.
The important thing is that the drudgery of schema maintenance and conversion can be eliminated using C++'s (limited) reflection capabilities. Your code is the schema.
In WhateverMessages.h:
#pragma pack(1)
template<typename Message>
struct Header
{
uint32_t type = Message::MessageType;
uint16_t size = sizeof(Message);
};
struct Msg : Header<Msg>
{
static constexpr uint32_t MessageType = 'msg1'; // Can constexpr bswap this to appear correctly in GDB and hex dumps
uint32_t somedata;
char someText[32];
};
#pragma pop(pack)
In EventHandler.cpp:
void processMessage(const Msg& msg)
{
// Everything is type safe now, do real stuff
}
void processMessage(char* data, uint32_t len)
{
Header* hdr = reinterpret_cast<Header*>(data);
// should check the packet length is sufficient, return length processed, process multiple messages, etc
switch(hdr->type)
{
#define DISPATCH(m) case m ::MessageType: assert(hdr->size == sizeof(m) /*or warn, whatever */); \
processMessage(*reinterpret_cast<m*>(hdr)); break;
DISPATCH(Msg);
DISPATCH(OtherMsgDeprecated);
DISPATCH(OtherMsgButForRealThisTime);
#undef DISPATCH
default:
err("Unknown message type: %x (%s)\n", hdr->type, fourCCToString(hdr->type));
break;
}
}
Another important thing is that the raw binary is the first class format that requires no translation of any kind on x86-64 after the initial dynamic type inference in DISPATCH.
Python, JSON, SQL, etc are all slow anyways, so we can spend time massaging their serialization format afterwards.