4 ms·
> But yes, the serialized dict-of-arrays-of-dicts type stuff can be approached in a few ways, none of which are particularly beautiful. For what it's worth, th
by haberman 5y ago
> But yes, the serialized dict-of-arrays-of-dicts type stuff can be approached in a few ways, none of which are particularly beautiful.
For what it's worth, this sounds somewhat similar to protobuf (which also supports dicts, arrays, etc).
After spending many years trying to figure out the smallest, fastest, and simplest way to implement protobuf in https://github.com/protocolbuffers/upb https://github.com/protocolbuffers/upb, the single best improvement I found was to make the entire memory management model arena-based.
When you parse an incoming request, all the little objects (messages, arrays, maps, etc) are allocated on the arena. When you are done with it, you just free the arena.
In my experience this results in code that is both simpler and faster than trying to memory-manage all of the sub-objects independently. It also integrates nicely with existing memory-management schemes: I've been able to adapt the arena model to both Ruby (tracing GC) and PHP (refcounting) runtimes. You just have to make sure that the arena itself outlives any reference to any of the objects within.
(Protobuf C++ also supports arenas, that's actually where the idea of using arenas for protobuf was first introduced. But Protobuf C++ also has to stay compatible with its pre-existing API based on unique ownership, so the whole API and implementation are complicated by the fact that it needs to support both memory management styles).
- megous 5y agoFor JSON I settled for no transformation to a different in-memory representation, just inline ondemand parsing of a JSON string buffer. Works nicely and you don't need to manage memory much at all. https://megous.com/git/megatools/tree/lib/sjson.h https://megous.com/git/megatools/tree/lib/sjson.h