6 ms·
Show HN: Serialize++ – Tiny Data Serialization Library for C++
- apankrat 7y agoUnder 500 lines, this probably barely qualifies as a library, but it still allows packing structs and containers to/from binary blobs with as little effort as possible. It has virtually no dependencies, requires no pre-processing (like ProtoBufs do) and it should be easy to understand and extend. It knows how to handle basic types and std containers and then uses a couple of C++ features to coerce the compiler into auto-generating all needed methods for custom types. All you need to do is to specify which struct/class members must be serialized and then call store() on an instance to produce a blob. To restore an instance - feed the blob into parse() and that's it. This was originally written to implement an IPC protocol for a pair of cooperating processes, but it can be readily reused for quickly storing data on disk and other things.
- cygaril 7y agoBeing only five hundred lines doesn't disqualify it from being a library. But being impossible to safely integrate into other projects because of illegal names and polluting the global namespace does. This is at best a rough proof of concept which could at some point be turned into a library.
- Jyaif 7y agoYou can add https://github.com/USCiLab/cereal https://github.com/USCiLab/cereal to your footnotes as it works in a similar way: you list the fields of your structure in a serialize function.
- apankrat 7y agoDone.
- zabzonk 7y agoNote that names containing double underscores are reserved in C++ for the C++ implementation, which your library is not part of. Using such names can lead to all sorts of difficult to diagnose problems.
- apankrat 7y agoNoted. From my perspective it's the same class of issues as with using _t suffix for custom C types - also reserved, so technically not good, but trivial to fix if it ever causes any trouble. PS. In practice, the use of underscores, both single and double, as a name prefix is wide-spread. From the use of _foo notation to pass arguments to functions, to using _bar() to designate member functions that should be called under some sort of lock, to using __xxx for macros - I mean, yes, all this doesn't align with the C++ standard, but it exists, it's actively used in live code and it also leads to the coding habits that are hard to change. Hence the use of __f() in this particular library.
- zabzonk 7y ago> he use of underscores, both single and double, as a name prefix is wide-spread. Leading single underscores are OK at class scope, so OK for member variables or functions. They are not OK at global scope, in either C++ or C. And, yes it is wide-spread, as are many other dangerous bad practices. It's something that is easy not to do, and which has no worth, so why do it?
- apankrat 7y agoBecause it's not a "dangerous bad practice". There are examples of actually bad and dangerous language misuse, but this is not one of them. It's mostly pedantry. Like using KiB instead of KB.
- zabzonk 7y agoIt is dangerous. If you (illegally) #define a reserved name in your own code and then include a standard header file that (legally) uses that name, the results can be almost impossible to diagnose. There is never any need to do it, so not doing it is not "pedantry", it is simple good practice. The rules about double-underscores exist for very good reasons.
- pubby 7y agoDoes it handle endianness?
- apankrat 7y agoNo. See caveat #4. The code stems for the same-machine IPC library. No point in doing any byte order normalization. Trivial to add though if needed.
- chrisseaton 7y ago> The code stems for the same-machine IPC library. No point in doing any byte order normalization. Some processors can run one process in one byte order, and another process in another byte order! For example Itanium and the UM.be bit which the user can set.
- apankrat 7y agoNice, didn't know that. Seems a bit unfair though they didn't cover middle-endian order. Seriously though, how many machines are configured like that in the real world and what's the overlap with our target installation base?
- chrisseaton 7y agoNo idea - but it's something to keep in mind if you have the assumption in your head that 'byte order is constant for a machine'.
- apankrat 7y agoIt almost universally is though, no? I'd expect a processor core with inverted endianness to be excluded from the general use and reserved for select processes that require that feature.
- chrisseaton 7y ago
- unlinked_dll 7y agoSome quick and dirty things to make this useful, put it all in one header and wrap your code in a namespace. No really viable as a dependency otherwise.
- hellofunk 7y agoI’m curious what problem this is trying to solve that hasn’t been already solved in many other ways. There are so many serialization libraries for C++. The API for this isn’t too far from what YAS offers: YAS_OBJECT_NVP("myobject" ,("a", aa) ,("b", bb) ) instead of: OBJECT_SPEC( foo ) __f( aaa ), __f( bbb ) END_OF_SPEC YAS also aims to do it with as little fuss as possible, similar goals, yet YAS binary archives are endian independent, and there are other advantages.
- apankrat 7y agoI take it's this one - https://github.com/niXman/yas https://github.com/niXman/yas ? I haven't seen it before, but there are bound to be similar solutions to the same problem. The library I posted is used internally for a number of our projects and it has evolved in a course of several years. It is not _trying_ to solve a problem, it _is_ solving a very specific need that our code had since the beginning. So if you are implying that this is a pointless duplication of some other existing solution, then you are off. There's more than one way to skin a cat. Re: endianess - to each (project) their own. We don't need normalized byte ordering, so with all other things being equal yas-like library will be marginally slower. Other advantages - I don't see any that would benefit our case, but I'm sure there are some that may come handy in other contexts.
- deleted 7y ago[deleted]
- kjgkjhfkjf 7y agoDoes this work if the two communicating processes use different versions of the struct?
- apankrat 7y agoDepends on the differences, but generally - no, of course, not.
- kjgkjhfkjf 7y agoIt'd be useful if you mentioned this in your caveats list, since version skew is a common gotcha that some serialization technologies (e.g. protobuf) are designed explicitly to address.
- apankrat 7y agoOK, will do. I assumed it was obvious that schemas on both ends must match.
- quietbritishjim 7y agoI would assume it's obvious that schemas on the two ends don't need to match because solving that problem is usually a basic requirement of a serialisation library. Of course they need to be compatible in some way though e.g. in protobuf, don't reuse old field numbers.
- apankrat 7y agoDepends on the project. Lots of cases when there's no need for schema versioning. For example, when two processes need to talk to each other and they both run from the same binary or share the exact same datamodel. Even if they aren't run from the same binary, it's often much simpler to just require both sides to use the same protocol revision.
- hoistbypetard 7y agoLooks nice to use, at least for the happy path. I'd also say it looks a lot like the non-intrusive pattern for the boost serialization library. That's my default tool for this problem just because I'm pretty much always pulling in a pile of boost libraries for any non-trivial C++ project anyway.
- jokoon 7y agoI want to store data as binary, and generally I would just lay everything in arrays, write the vector size as an int, the type id (arbitrary value) as a char, write data, done. I just have 2 function to do that, and it's enough. If you have fat data, binary is a wise solution. Generally flattening data in array is not so complicated, and you gain speed. SQLite is also a good solution, although I'm not sure it's good to store things like arrays of vec2, pictures... I tried protocol buffers once, I was really horrified by the size of the headers it required.