3 ms·

I'm working on a C++ library at work that lets you define a data model (a tree of objects, arrays and values) and run visitors through it. The data model is bot

by boricj 2y ago

I'm working on a C++ library at work that lets you define a data model (a tree of objects, arrays and values) and run visitors through it. The data model is both effectively a JSON schema in source code form and a binding to the underlying data/application through callbacks and whatnots. I can make it ingest or emit whatever data format I want by writing a deserializer or serializer visitor for it (either directly or through a library), I can perform various pipeline-like transformations by chaining visitors together. The core is header-only, templated, constexpr and suitable for usage on resource-constrained systems. I know that there are equivalents in managed or interpreted languages (like Java's Jackson library), but I haven't managed to find anything quite like it elsewhere on the Internet for compiled, unmanaged languages. Maybe I haven't looked hard enough or I just don't know what to search for, which is too bad because it handily beats writing serialization/deserlialization code by hand.

nly 2y ago
Here's something like that I wrote in C++ in 2017 for JSON using Boost.Fusion + simple function overloading of a single 'from_json' function for handling different types. It works for nested objects, it's runtime type checked, and all numeric conversions are checked for loss while being forgiving. https://gist.github.com/nlyan/045fbe075b4e51d83be0cf4513fecd68 https://gist.github.com/nlyan/045fbe075b4e51d83be0cf4513fecd... The DEFINE_JSON macro is a tiny wrapper around BOOST_FUSION_DEFINE_STRUCT so the whole parsing routine effectively gets unrolled at compile time. https://www.boost.org/doc/libs/1_87_0/libs/fusion/doc/html/fusion/adapted/define_struct.html https://www.boost.org/doc/libs/1_87_0/libs/fusion/doc/html/f... The code predates broad availability of std::optional (so uses boost::optional), [[unlikely]] and the existence of Boost.JSON, so if i were using this technique today that's what I'd use, but at the time I used the taocpp JSON library (which is still actively maintained 8 years later) https://github.com/taocpp/json https://github.com/taocpp/json Here's an article talking about the technique from a CppCon 2014 talk - "Implementing Wire Protocols with Boost Fusion --Thomas Rodgers" https://isocpp.org/blog/2015/01/cppcon-2014-implementing-wire-protocols-with-boost-fusion-thomas-rodgers https://isocpp.org/blog/2015/01/cppcon-2014-implementing-wir...
- boricj 2y ago
  It's fairly different than the approach inside the CppCon 2014 talk because the data model itself is not actually strongly tied to the underlying types of the data. That lack of strong integration does result in a bit more boiderplate code, but it allows more flexibility. In principle, the data model and the actual data can be quite different as long as you can bridge the two together with the boilderplate code. It's more like projecting a JSON schema over a bunch of various getters and setters and running visitors through them, with some helpers to automatically grok STL-like containers.
  - nly 2y ago
    Yeah, an extra layer of type indirection for expressing various implicit encodings is useful for e.g. dates My point overall is that there is tonnes of prior art for compile time reflection libraries in c++ for this kind of thing. Boost has MPL, Fusion and Hana for starters. All of these have visitor patterns and algorithms out of the box. You originally said you hadn't found anything quite like it for unmanaged languages...but people have been doing this for decades at this point. In general I'm no longer a fan of the "encode the schema directly in C++" approach if the data i'm interfacing with actually has a schema.
danhau 2y ago
What slightly annoys me about JSON is that the order of object properties is defined to be irrelevant. { “hello“: 123, “world“: “foo“ } is the same as { “world“: “foo“, “hello“: 123 } If these two were semantically different, writing deserializer would be easier and more efficient, since you can simply expect the next tokens to represent the currently visited class member, or error. Otherwise you need to construct an object tree and look up its properties by name / hash.
- nly 2y ago
  Apache Avro's C++ deserializer for the JSON serialization used to expect exactly that: keys to be in schema order. Trust me you don't want this. Usually the reason you want to use JSON in the first place is you want to support third party data access. Even if you constrained the field order you'd still have to deal with absent fields.
- jasonthorsness 2y ago
  Notably this is one of the main differences between JSON and the MongoDB BSON format. Though some client libraries just treat it as JSON and don’t preserve the order.
- inbx0 2y ago
  Does the JSON spec actually say that those objects should be "equal", or does it just leave that detail to implementations? In JavaScript at least, those two are not exactly "the same", in the sense that you can observe the difference if you want to. If you parse those JSON strings and then iterate the keys (e.g. with Object.keys), the ordering will be different.
  - reichstein 2y ago
    The JSON spec only defines the JSON text format. It doesn't say what the text means. There are obvious interpretations, but every program that reads or writes JSON can decide what it does with it. On the other hand, the thing that makes JSON actually useful is the interoperability, that JSON written by one program, on one platform, can be read by another preterm on another platform. Those programs have to agree on a protocol, what the JSON text must satisfy and what it means. It's usually not considered valuable to require object properties to be in a specific order, so they don't. But they could.
lmm 2y ago
> I know that there are equivalents in managed or interpreted languages (like Java's Jackson library), but I haven't managed to find anything quite like it elsewhere on the Internet for compiled, unmanaged languages. Like Rust's Frunk? Or like what Zig or D let you do with "compile time reflection"?
- boricj 2y ago
  It's definitively different than Frunk, the library is not a general-purpose functional toolkit. One could certainly implement it with Zig's compile-time reflection with ease (don't know much about D). Actually, it's superficially similar to refl-cpp's serialization example [1], but with far less templating magic underneath due to the restricted scope. [1] https://github.com/veselink1/refl-cpp/blob/master/examples/example-serialization.cpp https://github.com/veselink1/refl-cpp/blob/master/examples/e...
unrealhoang 2y ago
Isn’t rust’s serde what you are looking for? https://serde.rs/data-model.html https://serde.rs/data-model.html

by boricj 2y ago

nly 2y ago

Here's something like that I wrote in C++ in 2017 for JSON using Boost.Fusion + simple function overloading of a single 'from_json' function for handling different types. It works for nested objects, it's runtime type checked, and all numeric conversions are checked for loss while being forgiving. https://gist.github.com/nlyan/045fbe075b4e51d83be0cf4513fecd68 https://gist.github.com/nlyan/045fbe075b4e51d83be0cf4513fecd... The DEFINE_JSON macro is a tiny wrapper around BOOST_FUSION_DEFINE_STRUCT so the whole parsing routine effectively gets unrolled at compile time. https://www.boost.org/doc/libs/1_87_0/libs/fusion/doc/html/fusion/adapted/define_struct.html https://www.boost.org/doc/libs/1_87_0/libs/fusion/doc/html/f... The code predates broad availability of std::optional (so uses boost::optional), [[unlikely]] and the existence of Boost.JSON, so if i were using this technique today that's what I'd use, but at the time I used the taocpp JSON library (which is still actively maintained 8 years later) https://github.com/taocpp/json https://github.com/taocpp/json Here's an article talking about the technique from a CppCon 2014 talk - "Implementing Wire Protocols with Boost Fusion --Thomas Rodgers" https://isocpp.org/blog/2015/01/cppcon-2014-implementing-wire-protocols-with-boost-fusion-thomas-rodgers https://isocpp.org/blog/2015/01/cppcon-2014-implementing-wir...

boricj 2y ago
It's fairly different than the approach inside the CppCon 2014 talk because the data model itself is not actually strongly tied to the underlying types of the data. That lack of strong integration does result in a bit more boiderplate code, but it allows more flexibility. In principle, the data model and the actual data can be quite different as long as you can bridge the two together with the boilderplate code. It's more like projecting a JSON schema over a bunch of various getters and setters and running visitors through them, with some helpers to automatically grok STL-like containers.
- nly 2y ago
  Yeah, an extra layer of type indirection for expressing various implicit encodings is useful for e.g. dates My point overall is that there is tonnes of prior art for compile time reflection libraries in c++ for this kind of thing. Boost has MPL, Fusion and Hana for starters. All of these have visitor patterns and algorithms out of the box. You originally said you hadn't found anything quite like it for unmanaged languages...but people have been doing this for decades at this point. In general I'm no longer a fan of the "encode the schema directly in C++" approach if the data i'm interfacing with actually has a schema.

danhau 2y ago

What slightly annoys me about JSON is that the order of object properties is defined to be irrelevant. { “hello“: 123, “world“: “foo“ } is the same as { “world“: “foo“, “hello“: 123 } If these two were semantically different, writing deserializer would be easier and more efficient, since you can simply expect the next tokens to represent the currently visited class member, or error. Otherwise you need to construct an object tree and look up its properties by name / hash.

nly 2y ago
Apache Avro's C++ deserializer for the JSON serialization used to expect exactly that: keys to be in schema order. Trust me you don't want this. Usually the reason you want to use JSON in the first place is you want to support third party data access. Even if you constrained the field order you'd still have to deal with absent fields.
jasonthorsness 2y ago
Notably this is one of the main differences between JSON and the MongoDB BSON format. Though some client libraries just treat it as JSON and don’t preserve the order.
inbx0 2y ago
Does the JSON spec actually say that those objects should be "equal", or does it just leave that detail to implementations? In JavaScript at least, those two are not exactly "the same", in the sense that you can observe the difference if you want to. If you parse those JSON strings and then iterate the keys (e.g. with Object.keys), the ordering will be different.
- reichstein 2y ago
  The JSON spec only defines the JSON text format. It doesn't say what the text means. There are obvious interpretations, but every program that reads or writes JSON can decide what it does with it. On the other hand, the thing that makes JSON actually useful is the interoperability, that JSON written by one program, on one platform, can be read by another preterm on another platform. Those programs have to agree on a protocol, what the JSON text must satisfy and what it means. It's usually not considered valuable to require object properties to be in a specific order, so they don't. But they could.

lmm 2y ago

> I know that there are equivalents in managed or interpreted languages (like Java's Jackson library), but I haven't managed to find anything quite like it elsewhere on the Internet for compiled, unmanaged languages. Like Rust's Frunk? Or like what Zig or D let you do with "compile time reflection"?

boricj 2y ago
It's definitively different than Frunk, the library is not a general-purpose functional toolkit. One could certainly implement it with Zig's compile-time reflection with ease (don't know much about D). Actually, it's superficially similar to refl-cpp's serialization example [1], but with far less templating magic underneath due to the restricted scope. [1] https://github.com/veselink1/refl-cpp/blob/master/examples/example-serialization.cpp https://github.com/veselink1/refl-cpp/blob/master/examples/e...

unrealhoang 2y ago

Isn’t rust’s serde what you are looking for? https://serde.rs/data-model.html https://serde.rs/data-model.html