18 ms·
On the complexity of JSON serialization (2020)
- alexvoda 6y agoI think the word he needs is "serdes" or "sedes". Just like codec is short for COder/DECoder an modem is short for MOdulator/DEModulator. The other word he is looking for is marshaling.
- lstamour 6y agoYou know, it never occurred to me to ask why Rust's popular serialization package was named https://serde.rs https://serde.rs but now I can't un-see it. :)
- navaati 6y agoWell, I don't know if the author is going to see this, but... I'm a young programmer and I think you've just, I don't know, "opened" something in me, something much bigger than the topic of JSON mapping. Not sure yet how to express it, about the more general topic of abstraction, maybe. You've made my world less simple but probably more correct. Thank you I guess...
- Waterluvian 6y agoI can't articulate it either but I know exactly what you mean. I've had these moments of epiphany before. It's wonderful.
- scubbo 6y agoTreasure those moments! They definitely get rarer over time (for many reasons: the amount of work (and so, time) required to reach another epiphany becomes greater, the amount of time you can dedicate to learning (rather than maintenance or leadership) decreases, and the likelihood that you'll be introduced to an interesting new problem in the course of your daily life becomes lower), but they're definitely worth it.
- geofft 6y agoYou might like this post, which is roughly on that topic: https://ideolalia.com/essays/composition-is-interpretation.html https://ideolalia.com/essays/composition-is-interpretation.h... (I thought of it when I first saw your comment but couldn't remember where, then I found it closing some few-day-old tabs :) )
- eyelidlessness 6y agoWelcome! You'll be glad for it (and cranky about it). This sensation is what many of us feel when we find the incidental complexities of things designed to be simple. Once you find enough you start to spot them yourself, and then you see them everywhere. The ugly part is... you see them everywhere. The beautiful part is you don't just get pattern recognition, you get a familiar solution mapping. I felt this way as a young programmer really getting my teeth in. I was also self-taught, so I didn't have familiarity with some things that would probably be considered basics/fundamentals. My advice regardless is: when you get this unsettling mind-expanding feeling go research prior art. Go find out how other people solve problems like it. Even if you come up wanting more/better, at least you have a lay of the land. And learn the terminology used describing the problem space to expand your hunt. You'll be amazed what you turn up! Edit: since this is on the topic of JSON (de)serialization, while I’d love to tout the very good pattern I see in my usual stack (TypeScript) where I’m working on an offering in the space, I’d actually recommend looking at prior art in a very different stack with very different goals: - Transit[1] which standardizes type metadata within JSON (but leaves type resolution up to producers/consumers). - EDN[2], which is the philosophical basis for Transit, written in Clojure syntax. It’s demonstrably worse for performance but syntactically a nicer format/DX if you have tooling to deal with it, and it’s nearly tooling-free if you use the stack. A lot of efforts to standardize rich data type representation in JSON unfortunately do it very haphazardly, so I wanted to include examples that come from the “pattern recognition/solution mapping” side as an example. Both have downsides, but they’re exceptionally well designed for what they are and deserve to be part of this discussion. [1]: https://github.com/cognitect/transit-format https://github.com/cognitect/transit-format [2]: https://github.com/edn-format/edn https://github.com/edn-format/edn
- einarwh 6y agoI did. Thank you for telling me about this experience. The world isn't simple, but at least it's interesting.
- tsimionescu 6y agoI think the article is missing the immense value of simply having a good default serialization. For most data models expressed in most languages, you can define a single canonical JSON representation that can be easily provided by a library. Then, there come the corner cases of course. And you do need to add a lot of code to handle them. But good defaults + some customizability is a very good strategy for many problems. It is true that you can encounter problems where your data model is so far away from JSON that it's a better idea to just transform it entirely by hand. But the very reason JSON caught on is that these cases are rare in the industry.
- cratermoon 6y agoHardly anyone has a system with written in only one language. Typically you'll have a mix of both languages, some statically typed, some dynamic, so strongly typed, some not. The problem is not about serializing data from language A and deserializing back to language A, although that comes with enough problems (e.g. what if you language supports functions as first-class types?), but going from language A, serializing the data, and then deserializing it to a language that may or may not support something the source language has built in: floating point, complex types, UTF8 strings, and so on.
- tsimionescu 6y agoPresumably, if you're exchanging data between languages A and B, you have already figured out how to translate you want to map all of the data structures between them. Then, what I would expect you do is that you have code in A that can translate Complex Data Structure -> Lowest Common Denominator Data Structure -> A built-in JSON serialization -> JSON -> B built-in JSON deserialization -> Lowest Common Denominator -> Complex Data Structure B. Of course, the language/library can't help with the complex -> lowest common denominator translation part, that one will have to get written by hand. But that shouldn't mean that the lowest common denominator has to be as basic as JSON itself.
- mjevans 6y agoMapping JSON to rigidly typed languages is also a bit of a problem. E.G. in golang you can use Interface, for some level of performance cost, to handle arbitrary maps of (string) keys to (arbitrary, Interface?) values. Usually I've encountered JSON blobs that start with a map (dictionary / object / whatever) of keys to values, but they might also be lists, or maybe even just a string in the case of errors. The values can also be of any type. For rapid prototyping or cases where performance is trivial, I would really like a language to have some kind of object tree type which can represent any of JSON's data types. It would only need to be converted / cast at the leaf level, when transforming data to/from JSON. The same data structure would also be useful for manipulating any other 'trivial within memory' documents, like XML or anything else that easily fits within memory. Extra bonus points if the interface type is also an option for external databases to wired up and mapped as an ORM. Golang has some kind of comments based decorator that does this for XML and JSON mapping of native objects, but it tends to be clunky and I generally end up hating myself for using it on anything not trivial. Offhand I'm not sure if there's a more proper way of implementing it via configuration files / scripts or programatically (build or runtime?)... It'd be interesting to see if any languages have an effective and easy solution to this issue; any suggestions from others?
- dnautics 6y agoI've been pretty happy with Elixir's Ecto. I wouldn't call it easy, though. There is a LOT of ceremony, but it's designed to be declarative and it sure puts me at ease. For database ingress, I use schemas, and put in aggressive validations in the changesets. For jsonb values, I use embedded_schemas (https://thoughtbot.com/blog/embedding-elixir-structs-in-ecto-models https://thoughtbot.com/blog/embedding-elixir-structs-in-ecto...), and cast their creation as a normal part of filling in the database object. For egress to a third party that accepts JSON, I use naked embedded_schemas (even better if they can be shared with the database) and use a protocol to build an custom encoder that can for example send the 3rd party a deeply structured JSON from a flat struct. Due to the way protocols work in elixir, when I send it to Oauth, Oauth will (indirectly) call my protocol code.
- waynesonfire 6y ago
- karavelov 6y agoNaming fields is a small pain, the PITA is that JSON does not have native encoding for enum/variant/sum types. So now there are multiple ways to encode them: internally tagged, externally tagged, adjacently tagged, implicit/untagged. And JSON schema does not solve that at all: you can write schemas for all of them, but they do not convey the same information in the end languages even if they have all the type capabilities.
- nly 6y agoEnum and variant/sum types are supported by a lot of serialization frameworks that have JSON serializations though. Thrift, Avro, Protocol Buffers, Flatbuffers etc. JSON is not a framework for serialization anymore than XML is.
- vemv 6y agoThis problem tends not to occur in Clojure codebases because Clojure is a data-centric (vs. model-centric) language. In other words, all our core data is expressed as json-like data (e.g. EDN), so the translation is quite trivial - basically from one format to the other. Building a codebase around plain old data is admittedly not for everyone, but it works! You certainly can still have a type-like system (in dev/test time) and validation (in production) using the various available mechanisms (Spec, Schema, Malli).
- m45t3r 6y agoThis is one of the reason I think FP is better than OOP for business development. You can still have entities (that are plain maps, possibly annotated with types so you can keep your sanity), but the business logic now works with simple data structures: you input some data and gets another more data back. Do this transformation a few more times and very complex business requirements can be expressed. Testing is also easy since most functions are pure. Now you need to transform the data? Ok, convert this data to another data format and pass it to serializer. Same thing for deserializing. Traditional OOP really makes this last step more complicated than it should, since now you have an object, and the object should be opaque so you don't leak implementation details. To serializer you possible have a complex hierarchy of objects so your serializing logic is more complex too. Of course OOP doesn't need to do this, but the FP model seems so much simpler to enforce this data processing part.
- monsieurbanana 6y agoI'm not so sure whether this is truly a benefit of FP. Is it the same in languages like haskell or ocaml? (haven't used them in any significant way) As for clojure, it's described as a "data-driven programming", which is almost as important as it's FP designation. I think it's that data-driven part that brings a lot of the benefits you're talking about.
- mrkeen 6y agoYes, smooth sailing in Haskell. Define the datatype and let GHC generics and the Aeson library handle the rest. The worst-case scenario is when keys in your JSON conflict with keywords or existing functions: 'id' and 'data' are pretty common. It means you need to rename your datatype fields to something like '_id' and '_data', and provide a mapping function to Aeson. Where the 'fully automatic' declaration looks like: instance FromJSON Coord The declaration where you remap to avoid name clashes could look like: instance FromJSON Coord where parseJSON = genericParseJSON defaultOptions { fieldLabelModifier = dropWhile (=='_') }
- deleted 6y ago[deleted]
- whalesalad 6y agoThis really has nothing at all to do with JSON. The author is really complaining about encoding and decoding domain objects (happening to use JSON as a representation) and being able to rehydrate them into full blown objects. This is actually a very trivial problem indeed! You gotta remember that JSON doesn’t support all of the types that many languages do. So you need to engineer around that by annotating your attributes with a type or relying on a naming convention on keys. You can’t get mad at JSON for not being good at this. It would like getting mad at a dog for barking at the mail man. It’s doing what it was designed to do. There are lots of tools and approaches for doing this. The native DynamoDB storage format comes to mind.
- Scarbutt 6y agoIt's not like everyone has the choice to not use JSON.
- whalesalad 6y ago??? There is absolutely nothing inherent to JSON that would cause the woes the author is experiencing. Any data can be serialized with JSON.
- Scarbutt 6y agoAny data can be serialized with JSON. Yep, by writing ad-hoc and context-dependent formats on top of JSON for every language on your system. JSON set of types are just too limited.
- blandflakes 6y agoCan't any data also be serialized as a string? Does that mean "string payload" is good enough for exchanging data between systems?
- whalesalad 6y agoBingo.
- resonantjacket5 6y agoThis sounds more about using openapi/protobuf/etc and language object mapping to some data format.
- fotta 6y ago> So what am I suggesting? I’m suggesting letting JSON serialization be about JSON only. Let JSON serializer libraries handle translating between text and a representation of the JSON object model. They can do that one job really well, quickly and robustly. Once you have that, you take over! You take direct control over the mapping from JSON to your own model. I’m mainly immersed in the Scala world so I’m not sure what solutions in other languages look like, but this is essentially what Circe[0] does, and very well IMO. You write Encoders/Decoders for your domain model which serialize your data to the JSON object model and the library takes care of serializing that to text. In most simple cases you can probably just use the (semi)auto-derived Encoder/Decoders so you don’t need to actually write them, but that’s besides the point which is that Circe does have this separation of concerns and makes working with JSON mostly painless (at least for me). [0] https://circe.github.io/circe/ https://circe.github.io/circe/
- hyperpallium2 6y ago"Databinding", though it also and now more commonly refers to UI-data mapping.
- PaulHoule 6y agoThose "floating point" numbers are part of the problem. It's shocking how many multiply-adds you can do in the time it takes to serialize and deserialize a single float. On top of that the numbers you are writing (say 0.1 or 0.2) don't really exist in the number system unless the number is a fraction over a power of two. So you get anomalies such as 0.1 + 0.2 != 0.3
- thedufer 6y agoJSON has no floating point numbers. It just has decimal numbers, which can be arbitrarily large and arbitrarily precise. If you deserialize them into floats and then are upset that they aren't exactly right, that's on you, not JSON. Deserialize them into some bigdecimal type if this is a problem for your domain.
- nly 6y agoBut almost all JSON libraries parse them to binary doubles and don't even do that correctly.
- ben509 6y agoIf JSON is transported through any intermediate processor, you have to assume they're translating them to floats and back again. If you treat them as arbitrary decimals, you have to expect that data is being silently corrupted. And this includes intermediate processors that are invoked manually, so if someone uses a command like jq to simply test something, numbers are being truncated and they're baffled as to why they're getting different results.
- PaulHoule 6y agoThe "decimal floats" specified in JSON are exactly what is wrong with it; Javascript approaches numerics like Applesoft basic did, you have floats that can often stand in for ints. The JSON specification promises one thing, but it's not supported by the 'reference implementation' that it is based on. Also it is a lose-lose situation. Not only are floats wrong in many ways (e.g. I think 0.1 + 0.2 = 0.300000000000002 makes many people decide "computing is not for me") but parsing floats (really any ASCII numbers) is astonishingly slow and people get "frog boiled" into accepting it. (e.g. there is no problem with the speed of parsing 10 floats, but when you are parsing a million floats you have a real problem, and a million floats is just 4 MB of core, well in the range that a web or other application could handle on anything bigger than an 8-bit micro-controller.) Like the numerous problems that cause programmers to not use the SIMD instructions in your Intel CPU, there are multiple problems with floats, each of which can be dismissed by apologists, but when you add them up it's a major drag on the industry.
- abhinuvpitale 6y agoGreat point about how encoding-decoding custom objects/models to JSON is the biggest pain point. Coming with requirements before starting out the Object->JSON encoding, really helps you solve the more complex bottlenecks you might hit in the later "serialization". Its super interesting especially when you are using floating points, dates, C-style unions (represented as nested lists/dicts).
- chestervonwinch 6y agoIn Python land, we use a library called marshmallow for mapping domain object attributes to JSON fields. In most cases, these mappings are a direct reflection of the associated domain objects, so the result is a bit verbose but still works nicely because you can easily version them as your model definition evolves over time. I was under the impression that having an interface between models and JSON serialization was commonplace, given the popularity of marshmallow library in Python. Is this not the case in other languages and ecosystems?
- asdfsfkwqe 6y agoOne example, Enums. How do you parse them? by number? or by text values? or even both?
- AlphaSite 6y agoPydantic is another great library. Golang also has the builtin json struct deserialiser.
- acjohnson55 6y agoIn Scala (and I assume many other languages), this is exactly how it works in the popular libraries. You design, or automatically derive, the mapping from your domain objects and a JSON AST. As the author mentions, the mapping between the AST and the string encoding is something you can consider to be a black box, unless you want to change settings like compact vs pretty printing. To me, the most interesting question is what the author alluded on how to manage the coupling between arbitrary business objects and JSON AST. Writing a ton of boilerplate code is not at all fun. But as he points out, attempting to automate the translation through metaprogramming results in a problematic coupling of internal system details and the external contract. An idea I like is something more like code generation. Macros where the expansion gets written to disk, so hand edits can be made. There should be automatic verification of any changes to the contract, which can be done by generating an OpenAPI spec and version controlling it. Also, it should be really easy to differentiate between the default generated code and hand edits. There shouldn't be any need to crawl through all of the mapping code if there are far fewer changes relative to the stuff that was autogenerated.
- draw_down 6y ago> we have no word that encompasses both “serde”
- austincheney 6y agoI imagine the complexity is bound to the language you are using. From JavaScript writing a JSON serialization library wasn’t that challenging. Compared to various other web technologies serializing JSON was almost trivial.
- phyrex 6y agoYou’re aware that json stands for “JavaScript object notation”, right? ;)
- alienspaces 6y agoEssentially identifying the usefulness of a layered architecture with abstractions between each layer.
- _ZeD_ 6y agoSometimes I still miss when I worked with Java and Jackson showed me that the databindibg was a solved problem. But it seems to me that no library writer for other languages still has learned it
- mrkeen 6y agoI assumed the author was writing from a Java/Jackson POV. What was your reading of it? What do you use now that makes Jackson look good?
- JaggerJo 6y agoThere is a beautiful json lib for F# called Thoth. Works on .NET and Fable. Encode.object [ "blockId", Encode.guid data.Id "blockName", Encode.string data.Name "nestedType", Nested.Encode data.Nested ] Decode.object (fun get -> { Block.Id = get.Required.Field "blockId" Decode.guid Block.Name = get.Required.Field "blockName" Decode.string Block.Nested = Nested.Decode get }
- kimburgess 6y agoThis is a problem space that crystal-lang handles with beautiful elegance [1]. Include a single module, get sane behaviour out-of-the-box, and specify anything domain specific with a type annotations. [1]: https://crystal-lang.org/api/0.35.1/JSON/Serializable.html https://crystal-lang.org/api/0.35.1/JSON/Serializable.html
- valenterry 6y agoHow does it look if you have two different json schemas/versions that are both valid (maybe an old one and a new one)?
- kimburgess 6y agoYou specify the type to deserialise to. This lets you be very concrete about this if you have versioned API endpoint or different data source. Alternatively if you deserialise into a type union, the first compatible type will be used. Finally, you can also use a discriminator field [1] to select an appropriate subtype. [1]: https://crystal-lang.org/api/0.35.1/JSON/Serializable.html#use_json_discriminator(field,mapping)-macro https://crystal-lang.org/api/0.35.1/JSON/Serializable.html#u...
- valenterry 6y agoI mean, what if the type I deserialize into is the same (my current domain object type), but I have to deserialize two (or more) different versions of json into it. I think what you linked is the case where my domain object type is either A or B - but that's not what I meant.
- anonymoushn 6y agoIs this a language issue? Users of dynamic languages can have almost fully general "Data model mapping" in like 40 LOC. That is, the tasks of translating between json dicts and class instances and handling graphs that have multiple in-edges to the same object can just be done once. Are there other issues?
- ben509 6y agoI've tried this approach, and it's not great. You either have to pack a lot of noisy information in the JSON for the deserializer to work reliably, or you kludge it to try and guess, and then you find yourself designing your data structures around all the kludges. You inevitably run the risk that your deserializer can, effectively, make arbitrary function calls. And your data is tied not only to your language, but also needs to find classes in specific modules.
- anonymoushn 6y agoI don't think I need to run arbitrary code to take a dict and make it into an instance of a specific class whose members are that dict, but this may depend on your choice of dynamic language! Edit: I guess you mean the deserializer is running arbitrary code to import the necessary classes if they are not already imported. I would prefer that it fail in this case. I think it's not too much trouble to ask users who insist on using classes to have them around before instantiating them.
- ben509 6y agoI mean that fundamentally the deserializer has lookup function and it's going to lookup anything it's told to. You're right, some languages are worse about this, e.g. Python's pickle will perform arbitrary code executions. But it's easy enough to restrict it to a set of known good constructors. You still have the classic late-binding dilemma: it's flexibility we don't actually want. Why should my find_matching_socks function ever have to worry that it might be passed basketballs? The usual answer is, "it's fine, it'll just crash if they do that." It might crash, or it might hang, or silently corrupt something. An attacker can stick the wrong kind of object into your code anywhere they like.
- ben509 6y agoThe author understates the problem. Many half-baked solutions handle serializing, but don't consider how you're going to deserialize data. And if you do anything with numbers, you may need to guard against intermediate processors that truncate all numbers to 64-bit float accuracy. I took a shot at it with https://pypi.org/project/json-syntax/ https://pypi.org/project/json-syntax/ after doing some ad hoc solutions. My approach was: 1. Don't serialize directly to JSON, but convert to the "jsonic" types. 2. Allow users to pick the rules they need, and make it reasonably easy to create new rules. 3. Generate picklable encoding and decoding functions. It's worked pretty well for a project that has a very complex set of objects that need to be transmitted via JSON. The main difficulty is that it depends on interrogating Python's type annotations, which were pretty sketchy in earlier versions.
- deepakarora3 6y agoCreating Java object models to map the JSON documents into was the problem that I was really fed up of. In my work, we have hundreds of JSON documents to manage and many a time the structure of the JSON document also changes. Managing the JSON object model classes in Java is all boiler plate which adds very limited value in my opinion. To solve this, I wrote unify-jdocs. You can completely eliminate the use of domain specific object model classes and work directly on the JSON document (an intermediate construct similar to what the author is pointing to). You can read more about it here at https://github.com/americanexpress/unify-jdocs https://github.com/americanexpress/unify-jdocs