8 ms·

> The fundamental advice on Unicode is decode and encode on system boundaries. That is, you should never be working on non-unicode strings within your business

by wtbob 10y ago

> The fundamental advice on Unicode is decode and encode on system boundaries. That is, you should never be working on non-unicode strings within your business logic. The same should apply to JSON. Decode it into business logic objects on entry into system, rejecting invalid data. Instead of relying on key errors and membership lookups, leave the orthogonal business of type validity to object instantiation. This right here is the correct approach. Serialisation formats should be serialisation formats, whether they be JSON, S-expressions, protobufs, XML, Thrift or what-have-you; application data should be application data. There are cases where it makes sense to operate on the serialised data directly, for performance or because it makes sense in context, but in the general case operate on typed application values.

Cthulhu_ 10y ago
When using parsers like e.g. Jackson or Gson for Java, this process is completely transparent and does not require any active thought from the developer - well, maybe if there's very specific formats that don't map 1:1 with the class that should be instantiated or generated from the json object. It's a bit more tricky in JS, both client-side and node. You can't work with the json string there, but after that you work directly with the json object. They're not OOP languages, really. I wouldn't want to work with too much untyped / unstructured json in back-end land myself to be fair.
- rattray 10y ago
  I think this might be where tools like Flow or TypeScript can become useful, since you can have typed javascript objects. On the other hand, the fact that there's no runtime typecheck renders static analysis somewhat impotent when it comes to the result of a network call.
- adrianratnapala 10y ago
  I've never had good experiences with automated serialisation -- even though it sounds like other people do it with success. What's the secret? To give you a flavour of the kind of poblem: In C# (or rather .net) json.net reads JSON and calls setters from the target class. That means the setters have to be public, and you don't know what order they will be called in, and you have no real signal about when it is all done. The constructor is no longer enough to guarantee the object's invariants are met. Most awkward.
  - PaulHoule 10y ago
    Automated serialization has gotten much better than it was in the bad old days of RPC and COM!
    - mack73 10y ago
      I have nothing good to say about COM, but I'm seriously thinking about gRPC [0] to get away from the sloppy json endpoints we code around today, at work. Before I dive in I would love to hear, what it is that makes that architecture a bad one. [0] http://www.grpc.io/ http://www.grpc.io/
  - eropple 10y ago
    JSON.NET can use private setters. They just have to exist. You do have to use thin constructors, but in JSON.NET there's a way to call a method post-deserialization.
  - lukasLansky 10y ago
    Yes, the automatic serialization is not a solution for the most pressing problems presented by the article -- it's just the first thing from all the things that has to be done at the boundary. You have some DTO class that is your system typed idea about the structure of the JSON -- this class is quite useful as an implicit documentation, but it really has to stay internal to the boundary. You will use an autodeserializer to such class and then you will continue by constructing real object from deserialized data that can be presented to the rest of the application. During such construction you can validate state and return errros. This step can be eased by some validating attributes on the boundary DTO properties, but there is always some custom logic that describes what is acceptable and what is not.
  - matwood 10y ago
    In Java land with Jackson/Gson they can use the getters/setters or reflection and find the private fields. The only time it is not completely automatic is when some json object is mixed cased myField1 and my_field1. Even then, just adding an annotation fixes it. For any special formats, for example iso8601 dates, you can quickly define a serializer/deserializer and be done. Is it really that hard in c#? It is not something I ever think about in Java.
- solatic 10y ago
  Automated serialization is the devil. Gson and Jackson require you to write EJB-style objects to get automatic serialization - default constructors, with getters and setters for each field - to achieve automatic serialization. The problem with this approach is that you've completely abdicated the power of the type system to ensure that your objects are valid. What happens if a field is missing from the JSON? Well, that field just becomes null. So now you have one of two options: 1) Write highly defensive code with null-checks everywhere. This is a pain to write, a pain to read, and almost impossible to get right and actually prevent null pointer exceptions. This is a nightmare. Switching to a null-safe language like Kotlin doesn't really help you beyond making sure that you actually code in all the null checks - the code is still ugly and a pain to maintain. 2) Call a (potentially) expensive verification method at the beginning of each method call for your object. This is less error prone than having null checks everywhere, but it's not much of an improvement. Because verification happens not at object creation time but rather when it's used, you'll find yourself with a verification exception at the entrance to some business logic where the JSON was passed to your system a week ago, immediately stored in a schema-less ORM, retrieved now, so you kind of have an idea that you have some client which didn't populate the field, but you have no idea which of the many, myriad versions of the client is responsible. So now you're fucked, and you're doubly fucked if you're losing data because of it. Or you could just take advantage of type safety and write immutable object factories which refuse to instantiate invalid objects. Then you can write clean code using objects which you know must be valid because of type system guarantees. Libraries like immutables.github.io make this a piece of cake.
  - kodablah 10y ago
    Automatic serialization is not the devil, overly forgiving automatic serialization is the devil. I use JSON serialization libs all the time in Scala which properly support optional vs required fields. For Java devs, Gson has bad required/optional support [0] but Jackson does have it for creator properties [1]. It is important to qualify statements like your initial one to include the specific situation which it is bad instead of using a broad brush. 0 - https://github.com/google/gson/issues/61 https://github.com/google/gson/issues/61 1 - http://static.javadoc.io/com.fasterxml.jackson.core/jackson-annotations/2.8.1/com/fasterxml/jackson/annotation/JsonProperty.html#required() http://static.javadoc.io/com.fasterxml.jackson.core/jackson-...
VikingCoder 10y ago
...unless your application is doing an in-place edit. For instance, if your image compression application throws out my EXIF data that it doesn't understand, I'm going to be pissed. (Unless you give me an option to preserve it.)
qwertyuiop924 10y ago
Part of the problem is that JSON and Sexprs aren't that they AREN'T serialization formats. They've been pressed into service as such, but they are actually notation for datastructures: In python, it may not be idiomatic to crawl dicts like this, but in JS, those aren't dicts, they're objects. If they've been de-serialized to some degree, they may even have their own methods. By the same token, in Lisp, Sexprs aren't a serialization format. They're a notation for the linked cons cells that Lisp data is made of. In Lisp, that Sexpr will be crawled for data, or maybe even executed. So while in Python, both may seem to be serialization formats, they aren't. Either way, if the application programmer has any sense, they'll abstract away the format of their data. In a lisp app, you won't be cdring down a sexpr, you'll be calling a function to grab the necessary data for you, usually from a set of functions that abstract away the underlying sexpr implementation, and treat whatever it is as a separate datatype. Of course, the sexpr might have been fed to an object constructor. Heck, it might be an object constructor, or a struct constructor. All of those types typically provide O(1) access, and autogenerated access functions, so it's the same story.
- parenthephobia 10y ago
  What are serialization formats, then? What makes them different from notations for data structures?
  - mattnewton 10y ago
    Not gp, but one difference is you can have a notation that can't capture some state: think of the date class in JavaScript. JSON can't serialize this without resorting to string encoding, whereas something like a protobuf or pickle could.
    - steveklabnik 10y ago
      That means JSON is missing support for some kinds of data, that doesn't mean that it's not a serialization format at all. Just one with less descriptive power.
Goladus 10y ago
This right here is the correct approach. Serialisation formats should be serialisation formats, ... application data should be application data. True, although the OP seems to be advocating having your app pretty much ignore serialization altogether in favor of object-oriented design. In particular the author objects to use of dictionaries and lists instead of objects. It is true that if you're designing an application with a json api in mind, you're likely to stick with the data structures that are easiest to serialize. Personally, I started writing programs that way before json became so common. I did it simply to take full advantage of the native data structures and to avoid prematurely confining myself into an object hierarchy that wasn't a good fit for the problem domain. It also winds up making code more generic and easier to rewrite in a different language if necessary (for example, moving server-side code to client javascript).
spc476 10y ago
That's the approach I took with my DNS library (https://github.com/spc476/SPCDNS)---extract https://github.com/spc476/SPCDNS)---extract the DNS packet into a C structure that's easier to deal with (for instance, the A RR structure: https://github.com/spc476/SPCDNS/blob/master/src/dns.h#L270 https://github.com/spc476/SPCDNS/blob/master/src/dns.h#L270).

by wtbob 10y ago

Cthulhu_ 10y ago

When using parsers like e.g. Jackson or Gson for Java, this process is completely transparent and does not require any active thought from the developer - well, maybe if there's very specific formats that don't map 1:1 with the class that should be instantiated or generated from the json object. It's a bit more tricky in JS, both client-side and node. You can't work with the json string there, but after that you work directly with the json object. They're not OOP languages, really. I wouldn't want to work with too much untyped / unstructured json in back-end land myself to be fair.

rattray 10y ago
I think this might be where tools like Flow or TypeScript can become useful, since you can have typed javascript objects. On the other hand, the fact that there's no runtime typecheck renders static analysis somewhat impotent when it comes to the result of a network call.
adrianratnapala 10y ago
I've never had good experiences with automated serialisation -- even though it sounds like other people do it with success. What's the secret? To give you a flavour of the kind of poblem: In C# (or rather .net) json.net reads JSON and calls setters from the target class. That means the setters have to be public, and you don't know what order they will be called in, and you have no real signal about when it is all done. The constructor is no longer enough to guarantee the object's invariants are met. Most awkward.
- PaulHoule 10y ago
  Automated serialization has gotten much better than it was in the bad old days of RPC and COM!
  - mack73 10y ago
    I have nothing good to say about COM, but I'm seriously thinking about gRPC [0] to get away from the sloppy json endpoints we code around today, at work. Before I dive in I would love to hear, what it is that makes that architecture a bad one. [0] http://www.grpc.io/ http://www.grpc.io/
- eropple 10y ago
  JSON.NET can use private setters. They just have to exist. You do have to use thin constructors, but in JSON.NET there's a way to call a method post-deserialization.
- lukasLansky 10y ago
  Yes, the automatic serialization is not a solution for the most pressing problems presented by the article -- it's just the first thing from all the things that has to be done at the boundary. You have some DTO class that is your system typed idea about the structure of the JSON -- this class is quite useful as an implicit documentation, but it really has to stay internal to the boundary. You will use an autodeserializer to such class and then you will continue by constructing real object from deserialized data that can be presented to the rest of the application. During such construction you can validate state and return errros. This step can be eased by some validating attributes on the boundary DTO properties, but there is always some custom logic that describes what is acceptable and what is not.
- matwood 10y ago
  In Java land with Jackson/Gson they can use the getters/setters or reflection and find the private fields. The only time it is not completely automatic is when some json object is mixed cased myField1 and my_field1. Even then, just adding an annotation fixes it. For any special formats, for example iso8601 dates, you can quickly define a serializer/deserializer and be done. Is it really that hard in c#? It is not something I ever think about in Java.
solatic 10y ago
Automated serialization is the devil. Gson and Jackson require you to write EJB-style objects to get automatic serialization - default constructors, with getters and setters for each field - to achieve automatic serialization. The problem with this approach is that you've completely abdicated the power of the type system to ensure that your objects are valid. What happens if a field is missing from the JSON? Well, that field just becomes null. So now you have one of two options: 1) Write highly defensive code with null-checks everywhere. This is a pain to write, a pain to read, and almost impossible to get right and actually prevent null pointer exceptions. This is a nightmare. Switching to a null-safe language like Kotlin doesn't really help you beyond making sure that you actually code in all the null checks - the code is still ugly and a pain to maintain. 2) Call a (potentially) expensive verification method at the beginning of each method call for your object. This is less error prone than having null checks everywhere, but it's not much of an improvement. Because verification happens not at object creation time but rather when it's used, you'll find yourself with a verification exception at the entrance to some business logic where the JSON was passed to your system a week ago, immediately stored in a schema-less ORM, retrieved now, so you kind of have an idea that you have some client which didn't populate the field, but you have no idea which of the many, myriad versions of the client is responsible. So now you're fucked, and you're doubly fucked if you're losing data because of it. Or you could just take advantage of type safety and write immutable object factories which refuse to instantiate invalid objects. Then you can write clean code using objects which you know must be valid because of type system guarantees. Libraries like immutables.github.io make this a piece of cake.
- kodablah 10y ago
  Automatic serialization is not the devil, overly forgiving automatic serialization is the devil. I use JSON serialization libs all the time in Scala which properly support optional vs required fields. For Java devs, Gson has bad required/optional support [0] but Jackson does have it for creator properties [1]. It is important to qualify statements like your initial one to include the specific situation which it is bad instead of using a broad brush. 0 - https://github.com/google/gson/issues/61 https://github.com/google/gson/issues/61 1 - http://static.javadoc.io/com.fasterxml.jackson.core/jackson-annotations/2.8.1/com/fasterxml/jackson/annotation/JsonProperty.html#required() http://static.javadoc.io/com.fasterxml.jackson.core/jackson-...

VikingCoder 10y ago

...unless your application is doing an in-place edit. For instance, if your image compression application throws out my EXIF data that it doesn't understand, I'm going to be pissed. (Unless you give me an option to preserve it.)

qwertyuiop924 10y ago

Part of the problem is that JSON and Sexprs aren't that they AREN'T serialization formats. They've been pressed into service as such, but they are actually notation for datastructures: In python, it may not be idiomatic to crawl dicts like this, but in JS, those aren't dicts, they're objects. If they've been de-serialized to some degree, they may even have their own methods. By the same token, in Lisp, Sexprs aren't a serialization format. They're a notation for the linked cons cells that Lisp data is made of. In Lisp, that Sexpr will be crawled for data, or maybe even executed. So while in Python, both may seem to be serialization formats, they aren't. Either way, if the application programmer has any sense, they'll abstract away the format of their data. In a lisp app, you won't be cdring down a sexpr, you'll be calling a function to grab the necessary data for you, usually from a set of functions that abstract away the underlying sexpr implementation, and treat whatever it is as a separate datatype. Of course, the sexpr might have been fed to an object constructor. Heck, it might be an object constructor, or a struct constructor. All of those types typically provide O(1) access, and autogenerated access functions, so it's the same story.

parenthephobia 10y ago
What are serialization formats, then? What makes them different from notations for data structures?
- mattnewton 10y ago
  Not gp, but one difference is you can have a notation that can't capture some state: think of the date class in JavaScript. JSON can't serialize this without resorting to string encoding, whereas something like a protobuf or pickle could.
  - steveklabnik 10y ago
    That means JSON is missing support for some kinds of data, that doesn't mean that it's not a serialization format at all. Just one with less descriptive power.

Goladus 10y ago

This right here is the correct approach. Serialisation formats should be serialisation formats, ... application data should be application data. True, although the OP seems to be advocating having your app pretty much ignore serialization altogether in favor of object-oriented design. In particular the author objects to use of dictionaries and lists instead of objects. It is true that if you're designing an application with a json api in mind, you're likely to stick with the data structures that are easiest to serialize. Personally, I started writing programs that way before json became so common. I did it simply to take full advantage of the native data structures and to avoid prematurely confining myself into an object hierarchy that wasn't a good fit for the problem domain. It also winds up making code more generic and easier to rewrite in a different language if necessary (for example, moving server-side code to client javascript).

spc476 10y ago

That's the approach I took with my DNS library (https://github.com/spc476/SPCDNS)---extract https://github.com/spc476/SPCDNS)---extract the DNS packet into a C structure that's easier to deal with (for instance, the A RR structure: https://github.com/spc476/SPCDNS/blob/master/src/dns.h#L270 https://github.com/spc476/SPCDNS/blob/master/src/dns.h#L270).