5 ms·
JSON parser libraries in general is a black hole of suffering imo. They're either written with a different use case in mind, or a complex mess of abstractions;
by codr7 1y ago
JSON parser libraries in general is a black hole of suffering imo.
They're either written with a different use case in mind, or a complex mess of abstractions; often both.
It's not a very difficult problem to solve if you only write exactly what you need for your specific use case.
- flohofwoe 1y agoYou can't get much more 'opinion-less' than this library though. Iterate over keys and array items, identify the value type and return string-slices.
- IshKebab 1y agoIt also feels like only half the job to me. Reminds me of SAX "parsers" that were barely more than lexers.
- flohofwoe 1y agoI mean, what else is there to do when iterating over a JSON file? Delegating number parsing and UNICODE handling to the user can be considered a feature (since I can decide on my own how expensive/robust I want this to be).
- skydhash 1y agoThat is what I like Common Lisp libraries. They are mostly about the algorithms, leaving data structures up to the user. So you make sure you got those rights before calling the function.
- IshKebab 1y agoExtracting the data into objects. Libraries like Serde and Pydantic do this for you. Hell the original eval() JSON loading method did that too.
- meindnoch 1y agoThen you lose the ability to do streaming.
- IshKebab 1y agoTrue, but usually you only need that if your data is so large it can't fit in memory and in that case you shouldn't be using JSON anyway. (I was in this situation once where our JSON files grew to gigabytes and we switched to SQLite which worked extremely well.)
- meindnoch 1y agoActually, you'll hit the limits of DOM-style JSON parsers as soon as your data is larger than about half the available memory, since you'd most likely want to build your own model objects from the JSON, so at some point both of them must be present in memory (unless you're able to incrementally destroy those parts of the DOM that you're done with). Anyhow, IMO a proper JSON library should offer both, in a layered approach. That is, a lower level SAX-style parser, on top of which a DOM-style API is provided as a convenience.
- IshKebab 1y ago> since you'd most likely want to build your own model objects from the JSON, so at some point both of them must be present in memory Not really because the JSON library itself can stream the input. For example if you use `serde_json::from_reader()` it won't load the whole file into memory before parsing it into your objects: https://docs.rs/serde_json/latest/serde_json/fn.from_reader.html https://docs.rs/serde_json/latest/serde_json/fn.from_reader.... But that's kind of academic; half of all memory and all memory are in the same league.
- meindnoch 1y agoThat's only true if your model objects are serde structs, which is not desirable for a variety of reasons, most importantly because you don't want to tie your models to a particular on-disk format.
- nicce 1y agoThe project advertises that it has zero-allocations with minimal state. I don’t think it is fair or our problems are very different. Single string, (the most used type), and you need an allocation.
- mbac32768 1y agoIt's astonishing how involved a fucking modern JSON library becomes. The once "very simple" C++ single-header JSON library by nlohmann is now * 13 years old * is still actively merging PRs (last one 5 hours ago) * has 122 __million__ unit tests Despite all this, it's self-admittedly still not the fastest possible way to parse JSON in C++. For that you might want to look into simdjson. Don't start your own JSON parser library. Just don't. Yes you can whiteboard one that's 90% good enough in 45 minutes but that last 10% takes ten thousand man hours.
- EasyMark 1y agoYeah I use this and I think most of friends do too :)
- mbac32768 1y agoyeah it seems like every other C++ project uses it
- typpilol 1y ago122 million unit tests? What?
- fHr 1y agoholy shit
- codr7 1y agoYeah, but as long as I'm not releasing in public, I don't need to support 20 different ways of parsing. That's the thing with reinventing wheels, a wheel that fits every possible vehicle and runs well in any possible terrain is very difficult to build. But when you know exactly what you need it's a different story.
- vovavili 1y agoI am very surprised to hear the unit testing statistic. What kind of unholy edge cases would JSON parsing require to make it necessary to cover 122 million variations?
- forty 1y agoParsing JSON is a Minefield (2016) https://seriot.ch/projects/parsing_json.html https://seriot.ch/projects/parsing_json.html
- codr7 1y agoNot if I'm also the producer.
- president_zippy 1y agoFinally, I have found someone who understands the purpose of using someone else's tiny header-only C library; someone who sincerely thought about it before looking for an excuse to bitch and complain.
- TheRealPomax 1y agoAnyone who claims "it's not a very difficult problem" hasn't actually had to solve that problem.
- codr7 1y agoExcept I have, several times, with gopd results. So in this case you're wrong. General purpose is a different can of worms compared to solving a specific case.
- patrickmay 1y ago> JSON parser libraries in general is a black hole of suffering imo. Sexprs sitting over here, hoping for some love.
- codr7 1y agoI still mourn the timeline where we got a real Lisp in the browser instead of the current abomination.