5 ms·
Abusing C to implement JSON parsing with struct methods
- pjmlp 2y agoAlso known as how to manually do the work C++ compilers do for you.
- flykespice 2y agoMeta but I really appreciate how minimal static but styleful the website design is, specially the code snippets.
- xnacly 2y agoThanks, i tried to go for a mimimalist cyberpunk inspired vibe - thus far i like it :)
- oguz-ismail 2y agoWhat's with the lowercase I's though?
- johnisgood 2y agoThe first letter in a couple of sentences is also lowercase, so probably typos. Somewhere he uses "I" (correctly), somewhere he is not, so it is inconsistent, and most likely just typos.
- xnacly 2y agoI am not a native english speaker so I often forget to uppercase the 'I's
- bt1a 2y agolooks great on mobile, too. it's exceptional at organizing the content's contrast ratios and spacing, my only nitpick is slightly too much color
- montyanderson 2y agoyears ago i made a lib that tries to avoid the duplication in e.g. json->is_eof(json) https://github.com/montyanderson/foop https://github.com/montyanderson/foop
- hyperhello 2y agoGreat site, but the JSON looks like it's embedded in C as values but is really just a macro that expands it into a string and then parses it. So you can't put any local variables into the JSON. In my opinion, C really needs some officially supported reflection for the names of enums and such things. People have reinvented the wheel so many times and it never quite gets there.
- immibis 2y agoThat would be a C++ feature. C is very "batteries not included". In particular, it won't generate entire functions or data blocks that you didn't write or include from a library.
- deleted 2y ago[deleted]
- DeathArrow 2y ago>In my opinion, C really needs some officially supported reflection for the names of enums and such things. C is good as it is an someone has the chance to hold all its features in his memory. If you need or want to complicate things, there's C++ or Rust.
- pjmlp 2y agoI very much doubt that as any pub quiz on C will validate. People think they know C, in reality they never read ISO C document, aren't aware of the differences between ISO C standard library and POSIX, how each compiler handles ill formed no diagnostic required, implementation defined and UB parts of the standard, and to top that even better compiler specific extensions.
- virgilp 2y agoYou do not need how compiler handles UB parts of the standard; in fact you should NEVER rely on it, as it can change without warning. That's effectively "incorrect code", by definition.
- anonymousiam 2y agoI've used this trick (unions within structs) for decades. You can parse very quickly, but your code will not necessarily be portable. It's a perfect solution if your application will only be run on the hardware you're developing on.
- atiedebee 2y agoWhy are unions within structs not portable?
- cantrecallmypwd 2y agoUndefined padding and alignment.
- Keyframe 2y agoyou can do explicit padding and you can force alignment; latter might be compiler specific until C11 at least. You can always check for the struct layout as an assert.. it's doable.
- atiedebee 2y agoThat doesn't really matter for sum types as used in this blog, does it? As long as you're not serialising the struct or accessing it through pointers of different types it ought to work anywhere.
- cantrecallmypwd 2y agoTemporary, in-memory representation of an AST generally doesn't need to be serialized or passed outside the process in this use case.
- teo_zero 2y agoAnalyzing the json_value struct, I don't get why values, object_keys and length are fields outside the union. I expected something like: struct json_value { enum json_type type; union { bool boolean; char *string; double number; struct { char **keys; struct json_value *values; size_t length; } object; } value; }; Of course I would also use anonymous structs and unions to simplify json->value.object.length to json->length
- uecker 2y agoI haven't looked at this in detail, but I do not see an "abuse" here. This is just regular C code.
- DeathArrow 2y ago>Instead of using by itself functions: attach functions to a struct and use these as methods He could evolve this further: use SOLID and design patterns. If the goal is to bastardize C experience, why stop at objects?
- userbinator 2y agoOnce we hit the end we allocate a temporary string, copy the chars containing the number from the input string and terminate the string with \0. strtod is used to convert this string to a double. No need to allocate, just use strtod on the original string.
- tempodox 2y agoBut how do you know whether a JSON string is a pointer to static data (like in the example with the `main` function) or was dynamically allocated? The same goes for arrays and objects.
- threeducks 2y agoYou could tag dynamically allocated objects with some magic bytes. To then check whether a string is dynamically allocated, you can simply compare the first few bytes against the magic bytes. This is a probabilistic method, but if you use enough magic bytes, it becomes basically impossible to go wrong. See e.g. https://github.com/99991/dynamic/blob/a423a04061ee44bad0720fbd29f2321cc276564a/src/dynamic_gc.c#L21 https://github.com/99991/dynamic/blob/a423a04061ee44bad0720f... for an example (incidentally also a C JSON parser, but with automatic garbage collection).
- xnacly 2y agoHi, author here, i use the 'type' field of the json_value struct, strings, object keys, object values and array members are allocated, i use this info for freeing the memory
- UncleEntity 2y agoSo, umm, abusing the C switch statement to implement polymorphism? I kid, I kid... I am curious about how much this really saves over just using C++ classes and something like the curiously recurring template pattern while turning off runtime type checking.
- boricj 2y agoI'm working on a C++ library at work that lets you define a data model (a tree of objects, arrays and values) and run visitors through it. The data model is both effectively a JSON schema in source code form and a binding to the underlying data/application through callbacks and whatnots. I can make it ingest or emit whatever data format I want by writing a deserializer or serializer visitor for it (either directly or through a library), I can perform various pipeline-like transformations by chaining visitors together. The core is header-only, templated, constexpr and suitable for usage on resource-constrained systems. I know that there are equivalents in managed or interpreted languages (like Java's Jackson library), but I haven't managed to find anything quite like it elsewhere on the Internet for compiled, unmanaged languages. Maybe I haven't looked hard enough or I just don't know what to search for, which is too bad because it handily beats writing serialization/deserlialization code by hand.
- nly 2y agoHere's something like that I wrote in C++ in 2017 for JSON using Boost.Fusion + simple function overloading of a single 'from_json' function for handling different types. It works for nested objects, it's runtime type checked, and all numeric conversions are checked for loss while being forgiving. https://gist.github.com/nlyan/045fbe075b4e51d83be0cf4513fecd68 https://gist.github.com/nlyan/045fbe075b4e51d83be0cf4513fecd... The DEFINE_JSON macro is a tiny wrapper around BOOST_FUSION_DEFINE_STRUCT so the whole parsing routine effectively gets unrolled at compile time. https://www.boost.org/doc/libs/1_87_0/libs/fusion/doc/html/fusion/adapted/define_struct.html https://www.boost.org/doc/libs/1_87_0/libs/fusion/doc/html/f... The code predates broad availability of std::optional (so uses boost::optional), [[unlikely]] and the existence of Boost.JSON, so if i were using this technique today that's what I'd use, but at the time I used the taocpp JSON library (which is still actively maintained 8 years later) https://github.com/taocpp/json https://github.com/taocpp/json Here's an article talking about the technique from a CppCon 2014 talk - "Implementing Wire Protocols with Boost Fusion --Thomas Rodgers" https://isocpp.org/blog/2015/01/cppcon-2014-implementing-wire-protocols-with-boost-fusion-thomas-rodgers https://isocpp.org/blog/2015/01/cppcon-2014-implementing-wir...
- boricj 2y ago
- jbirer 2y agoThis is giving me GObject and Gdk flashbacks. If anyone wants to traumatize themselves, compile GTK2 and work with the aforementioned. That being said, it's still a very clever way to implement this.
- robinsonb5 2y agoI'm seeing reasonably clean use of function pointers without the usual tangle of opaque macros. This kind of pattern easily turns into a mess if you get too clever with it, but I quite like what I'm seeing here. (I suspect autotools will have traumatised anyone long before they manage to get GTK2 to compile on a modern system!)
- osmsucks 2y agoI'm not really seeing what's noteworthy about this article.
- flohofwoe 2y agoWould be good to get some words about performance. Looking at the code there are a lot of granular memory allocations, which IME with other JSON libraries which work like this is the number one performance killer and can add up to seconds when parsing JSON files with hundreds of thousands of nodes. Always going through a 'virtual method table' also can't be great for performance (not so much because of the indirection, but because it acts as an optimization barrier for the compiler). Peformance doesn't matter much of course when the code is ever only used to load tiny JSON files.
- xnacly 2y agoHi, author here, definitely, allocating elements while parsing is slow, especially calling realloc for each new element encountered, a growing array backed by an arena would be my go to if i had to overengineer this one.
- pwdisswordfishz 2y ago> CFLAGS := -std=c23 CFLAGS is for optional flags that don't change semantics or whether (correct) code compiles at all…
- MathMonkeyMan 2y agoCFLAGS are flags for the C compiler. It could totally change the interpretation of the semantics. C89 != C23.
- ingen0s 2y agothis is still the best read of the year for me
- _0ffh 2y agoI am sorry, I don't quite understand where the "abuse" comes in. Attaching function pointers to structs? I've done that in production twenty years ago (and others have probably done it in the 70s) and I see nothing wrong with it. C gives us function pointers for a reason, and it's not to not use them... to me, at least, they're actually one of the premier features of C! (GCC's goto *ptr on the other hand, now there's some potential for getting a little bit wild...)
- uecker 2y agoI assume it is because nowadays there are so many people running around claiming you need C++ or Rust or other more complicated languages to build abstractions, that people do not understand anymore that you can do this in C just fine.
- usrnm 2y agoI don't think I've ever seen a large C codebase that wasn't using structs with methods. It isn't by any measure a secret or lost art.
- uecker 2y agoAnd yet, here we are reading about it in the news.
- userbinator 2y agoThe earliest C++ compilers compiled to C, and I believe that's still possible to do even for the latest versions of the C++ standard.
- nextaccountic 2y agoIt's always possible to compile a turing complete language into another turing complete language, so this statement doesn't carry much meaning. The interesting thing with the original C++ (then called "C with classes") compiler, Cfront, which compiled to C, was that it was an 1:1 transpiler that offered only local transformations with no global analysis (you can transform one function at time without looking at the rest of the code). The structure of the generated C code closely matched the C++ source code. This is something that's not always possible when compiling a language into another, this only happens when a language offers essentially just syntactic sugar over another, like Coffeescript and Javascript. I guess that if you compile exceptions using setjmp/longjmp and expanded all templates (I don't think you can compile templates into C macros), and also expanded method overloading and type inference, then all other features from modern C++ are only local transformations. Well maybe with the exception of couroutines from C++20 (which maybe can be compiled into setjmp/longjmp as well, if it can interact well with exceptions? But the purpose of couroutines is to compile into a state machine, which is a non local transformation) The original Cfront compiler didn't implement even exceptions (they wanted to but Wikipedia said the code was a mess [0] so it was abandoned), but that another, later C++ compiler [1] compiled into C and implemented exceptions and other things [0] https://en.wikipedia.org/wiki/Cfront https://en.wikipedia.org/wiki/Cfront [1] https://en.wikipedia.org/wiki/Comeau_C/C%2B%2B https://en.wikipedia.org/wiki/Comeau_C/C%2B%2B
- jeffrallen 2y agoPlease don't write new code in nonsafe languages.
- McUsr 2y agoI look forward to the next blog post implementing the actual parser.
- eska 2y agoI don't think the use of structs to implement vtables in C is abuse, but in this case it's rather pointless. This idiom is used to have dynamic dispatch (e.g. a plugin system), an abstraction (e.g. same interface for file I/O and memory I/O in unit tests), or for ABI-stable vtables in libraries and C++ wrappers. Instead of `json_parse(&parser_state)` your code calls `parser_state.parse(&parser_state)` in main. Instead of `json_eof(parser_state)` your code calls `parser_state->eof(parser_state)` in the implementation. This is more to type, more irregular (switching from . to ->), and has worse performance. This interface is also not flexible enough to implement a parser for other file formats such as XML. So it doesn't make things easier, and you're not actually making use of dynamic dispatch, so I don't see the point in using this technique. In general I can not recommend this article for learning about C. I am being lenient about the things the author mentions explicitly, although I find even that mostly hand-waving away too much even for a prototype or short article. For example I have to say it is a bit of a head scratcher for me to hear that JSON is difficult to parse, since the ease of parsing is why JSON won to begin with. The linked article about JSON being a minefield is about underdefined semantics, not parser complexity. The biggest mistake, and this is sadly hand-waved away as "over-engineering", is the large amount of mallocs and frees, and necessity to track stack vs heap allocations. In general this style of C reminds me of what my CS professors taught us before they showed us Java GC and C++ RAII as our saviors. C has its issues, but this is far from professional grade C code, and is what C looks like when written by programmers who come from higher level languages with less control over memory allocation (or STL-using C++ programmers where RAII mixes allocation and initialization). A tiny bump allocator would simplify all the following code enormously. For example the freeing function is very complex and can be avoided completely. Whether memory is managed by the stack or heap doesn't need to be tracked anymore. It becomes possible to write code that never allocates at all. Consider this article for comparison: https://www.rfleury.com/p/untangling-lifetimes-the-arena-allocator https://www.rfleury.com/p/untangling-lifetimes-the-arena-all... The json struct mixes 2 responsibilities: providing data and parsing it. If we assume as in the article that all data is available from the start, then there's a lot of complexity in the parser that could've been avoided, since the parser is written as if it doesn't know where the EOF is and doesn't handle that error case correctly (especially in non-DEBUG build where ASSERT would be disabled). Returning EOF in-band in cur is bad practice also in C. I would avoid this whole function by providing higher level parsing functions like peek and expect which would provide or check for certain tokens (this is for non-optimized easy to write parsers). The check for null/false/true atoms should've been a code smell that this should be easier to implement somehow. In modern C null-terminated strings are mostly avoided, and string slices are used instead (begin pointer and size, or begin and end pointers). There are variables in the code called "slice", but they're C-strings. I'd suggest to the author to rewrite the code with slices and compare just how much more readable and efficient it becomes. The whole json_value struct can also be avoided by using X-macros to define the structs. It's the way how one implements a visitor pattern (with serializer and deserializer functions) without language support for reflection.
- variadix 2y agoUnless I’m missing something, I don’t really see the point of putting the functions in the struct instance since it doesn’t seem like you need to use any form of polymorphism/function overriding. If you have normal, non-virtual instance methods you are better off making it a normal function with some class centric naming pattern, e.g. JSON_Parse(JSON* self, const char* json). If you want to emulate virtual functions from C++ you are better off storing a single pointer to a statically allocated vtable of function pointers (1 instance per class), that way your instance structs are smaller at the cost of an extra indirection per virtual call. You could also go the Rust route and use a large pointer type (pointer to object + pointer to vtable) for polymorphic objects that avoids a longer dependency chain at the cost of larger object pointers.