6 ms·

Part of the problem is that JSON and Sexprs aren't that they AREN'T serialization formats. They've been pressed into service as such, but they are actually nota

Part of the problem is that JSON and Sexprs aren't that they AREN'T serialization formats. They've been pressed into service as such, but they are actually notation for datastructures: In python, it may not be idiomatic to crawl dicts like this, but in JS, those aren't dicts, they're objects. If they've been de-serialized to some degree, they may even have their own methods. By the same token, in Lisp, Sexprs aren't a serialization format. They're a notation for the linked cons cells that Lisp data is made of. In Lisp, that Sexpr will be crawled for data, or maybe even executed. So while in Python, both may seem to be serialization formats, they aren't. Either way, if the application programmer has any sense, they'll abstract away the format of their data. In a lisp app, you won't be cdring down a sexpr, you'll be calling a function to grab the necessary data for you, usually from a set of functions that abstract away the underlying sexpr implementation, and treat whatever it is as a separate datatype. Of course, the sexpr might have been fed to an object constructor. Heck, it might be an object constructor, or a struct constructor. All of those types typically provide O(1) access, and autogenerated access functions, so it's the same story.

parenthephobia 10y ago
What are serialization formats, then? What makes them different from notations for data structures?
- mattnewton 10y ago
  Not gp, but one difference is you can have a notation that can't capture some state: think of the date class in JavaScript. JSON can't serialize this without resorting to string encoding, whereas something like a protobuf or pickle could.
  - steveklabnik 10y ago
    That means JSON is missing support for some kinds of data, that doesn't mean that it's not a serialization format at all. Just one with less descriptive power.
    - mattnewton 10y ago
      That's fair, I guess it's not really a different class but a different shade of the same thing.
  - parenthephobia 10y ago
    What's really being described there, I think, is the notion that you can serialize and unserialize some data and get, to some extent, the "same" data back. Whilst that's more possible with protocol buffers or pickling (or whatever your language calls it), I can't think of any languages offhand which can round-trip any data. It's generally not possible to serialize objects denoting external resources - such as open file handles or network sockets. It's also often not possible to serialize closures, weak references (without dereferencing them), and not necessarily possible to serialize self-referencing objects: e.g. a list which contains itself - pickle can handle it, but I don't believe protocol buffers can do it.
    - qwertyuiop924 10y ago
      Some extended sexpr notations (particularly those used in Common Lisp, although many Schemes also support it, as do other lisps) support self-referential data structures. fds and sockets CAN, IIRC, to some degree, be sent to other processes on the same machine, but it's fairly limited. As for serializing closures, CHICKEN Scheme's s11n egg is the most prominent, although not the only, example. It's fairly limited, once again, to avoid sending the forest with the bannana, as Joe Armstrong would put it. This has nothing to do with our discussion, I just thought it was cool.
- qwertyuiop924 10y ago
  First off, make no mistake, JSON and sexprs ARE serialization formats. I didn't establish this well in my orignal comment. However, they're designed be a notation for specific structures: dicts and arrays in json's case, and cons cells, specifically Lisp code in the case of sexprs. This is unlike say, XML, which defines a tree hierarchy, but NOT what the underlying structures are. That's the parser's job.
recursive 10y ago
From the official source: "JSON (JavaScript Object Notation) is a lightweight data-interchange format. " http://json.org/ http://json.org/ It's nothing about pressing into service. This is the authoritative source.
- betenoire 10y ago
  "lightweight data interchange format" is not the same as "serialization formats". For one, serialization usually handles binary data as well. And types, lots of types. Serialization includes class definitions, etc, which are used to create a (in this case) python object. JSON is literally javascript object notation, and has nothing to do with pickling python.
  - kazinator 10y ago
    Yes it is. Serialization doesn't have to handle a particular type system in its entirety to be serialization. A serialization scheme can dictate its own type system; which can be smaller than that of the programming languages which support that serialization scheme. JSON has a simple type system; it serializes that system. (Might you be confusing serialization for other concepts like object store databases, or image saving?)
- dspillett 10y ago
  From the official source: "Democratic People's Republic of Korea" http://www.korea-dpr.com/ http://www.korea-dpr.com/ It's democratic, and for the people. It says nothing about being a totalitarian dictatorship. Not everything is/does what it says on the tin. You don't have to agree with official or otherwise authoritative sources without question. (not that I wish in any way to compare Mr Crockford or whoever runs json.org with DPRK or its leadership - I'm just using a deliberatly extreme example to highlight that what is written may not be what is, at least not from absolutely everyone's point of view)
  - oldmanjay 10y ago
    I hope you feel clever, because in context, this is far too inapplicable to the discussed reality of JSON to be an actual point.
kazinator 10y ago
A notation for a data structure is a serialization format. > In Lisp, that Sexpr will be crawled for data, or maybe even executed. A s-expression cannot be executed; it's just text. The object which it denotes can be walked or executed via eval. Before that happens, the s-expression must be converted to that object. In other words, deserialized by the reader.
- qwertyuiop924 10y ago
  Good point kaz. I meant that the notation signifies a specific set of general data structures, unlike XML, which doesn't specify the data-structure used in memory afaik, only the actual tree structure of the data itself.
- ohyes 10y ago
  In that s-expressions are a notation for computation, they can be executed by an interpreter. This is what we refer to as execution. Even assembly is like this, there's no other reasonable way for it to work right now. This feels like you're hairsplitting to no obvious benefit other than increasing confusion. I could as easily say "mathematical notation isn't math it's just text, you can't evaluate '1 + 2' without a human being because otherwise those are just marks on the page otherwise." This is true(ish) (with the correct escaping), but it's difficult for me to see how it's relevant to the discussion? We could imagine the situation where I've created the Texpr, which has slightly different notation but the same properties. I don't know that we would necessarily classify it differently or treat it differently. This leads to the alternate conclusion that maybe the s-expression is the underlying set of objects in the interpreter/compiler. (S-expressions are a special type of linked list, in that case). This rings true(er) to me, because of the way that we talk about s-expression manipulation in lisps. We most certainly are not using string operations to generate them. In which case the thing with the many parenthesis is merely standard lisp syntax, not s-expressions. This is further justified by the existence of different syntax in Dylan or Clojure, and availability of reader macro manipulation as its own entity. Tldr; it most certainly is not text! Nor can it be executed.
  - qwertyuiop924 10y ago
jerf 10y ago
JSON is a serialization format that was based on the data structure notation for Javascript. It is, however, a serialization format. Javascript objects are a superset of JSON, as they can contain arbitrary objects and functions, which JSON can not, and "true" Javascript object notation can elide quote marks or use apostrophes for keys, whereas JSON strictly specifies double-quotes around keys. The problem that arises in Python and other dynamically-typed languages is that there exists a default deserialization that is so good it very strongly tempts the programmer to use that exclusively. However, as good as the default serialization may be, it's also quite dangerous, for the reasons you mention and more. In strongly-typed languages there's a stronger focus on parsing the JSON instead, which has the advantage of producing objects with stronger guarantees (which isn't quite the same as pointing out there's stronger typing here, you could theoretically get the same guarantees in Python with some sort of JSON schema library or something), but has the disadvantage of generally being more challenging, since it's hard to beat the conciseness of "json.loads(s)" in Python. There generally is a default serialization in strongly-typed languages, but it's far more likely to become inconvenient if you need anything beyond simple numbers and strings, and people generally learn to prefer true deserialization in my experience, unless they, alas, start their program out from day 1 inputting and outputting JSON and accidentally structure their entire program around the default JSON structures and end up with the exact same problems as you'd get in Python. But as long as JSON isn't the very first thing to go in you're generally in better shape. (I have witnessed a Java program primarily written as a map of string to map of string to map of string to string. It was unsalvagable. And for all the cynicism I may occasionally muster, I don't say that often, because refactoring can be pretty powerful in Java, but this was beyond help. It actually had no JSON in site, but the same fundamental forces were in play.) Personally, despite preferring the more strongly typed approach in most ways, I must confess that when I'm working in Perl I am generally unable to resist the temptation to just JSON::XS::decode_json, and cover over the differences with unit testing rather than dealing with "true" deserialization. I make myself feel better by also telling myself that if I do anything else, I will confuse my fellow programmers who don't generally expect to see fancy deserialization routines when dealing with JSON, which is true enough, but in my heart I still know guilt.
- czardoz 10y ago

parenthephobia 10y ago

What are serialization formats, then? What makes them different from notations for data structures?

mattnewton 10y ago
Not gp, but one difference is you can have a notation that can't capture some state: think of the date class in JavaScript. JSON can't serialize this without resorting to string encoding, whereas something like a protobuf or pickle could.
- steveklabnik 10y ago
  That means JSON is missing support for some kinds of data, that doesn't mean that it's not a serialization format at all. Just one with less descriptive power.
  - mattnewton 10y ago
    That's fair, I guess it's not really a different class but a different shade of the same thing.
- parenthephobia 10y ago
  What's really being described there, I think, is the notion that you can serialize and unserialize some data and get, to some extent, the "same" data back. Whilst that's more possible with protocol buffers or pickling (or whatever your language calls it), I can't think of any languages offhand which can round-trip any data. It's generally not possible to serialize objects denoting external resources - such as open file handles or network sockets. It's also often not possible to serialize closures, weak references (without dereferencing them), and not necessarily possible to serialize self-referencing objects: e.g. a list which contains itself - pickle can handle it, but I don't believe protocol buffers can do it.
  - qwertyuiop924 10y ago
    Some extended sexpr notations (particularly those used in Common Lisp, although many Schemes also support it, as do other lisps) support self-referential data structures. fds and sockets CAN, IIRC, to some degree, be sent to other processes on the same machine, but it's fairly limited. As for serializing closures, CHICKEN Scheme's s11n egg is the most prominent, although not the only, example. It's fairly limited, once again, to avoid sending the forest with the bannana, as Joe Armstrong would put it. This has nothing to do with our discussion, I just thought it was cool.
qwertyuiop924 10y ago
First off, make no mistake, JSON and sexprs ARE serialization formats. I didn't establish this well in my orignal comment. However, they're designed be a notation for specific structures: dicts and arrays in json's case, and cons cells, specifically Lisp code in the case of sexprs. This is unlike say, XML, which defines a tree hierarchy, but NOT what the underlying structures are. That's the parser's job.

recursive 10y ago

From the official source: "JSON (JavaScript Object Notation) is a lightweight data-interchange format. " http://json.org/ http://json.org/ It's nothing about pressing into service. This is the authoritative source.

betenoire 10y ago
"lightweight data interchange format" is not the same as "serialization formats". For one, serialization usually handles binary data as well. And types, lots of types. Serialization includes class definitions, etc, which are used to create a (in this case) python object. JSON is literally javascript object notation, and has nothing to do with pickling python.
- kazinator 10y ago
  Yes it is. Serialization doesn't have to handle a particular type system in its entirety to be serialization. A serialization scheme can dictate its own type system; which can be smaller than that of the programming languages which support that serialization scheme. JSON has a simple type system; it serializes that system. (Might you be confusing serialization for other concepts like object store databases, or image saving?)
dspillett 10y ago
From the official source: "Democratic People's Republic of Korea" http://www.korea-dpr.com/ http://www.korea-dpr.com/ It's democratic, and for the people. It says nothing about being a totalitarian dictatorship. Not everything is/does what it says on the tin. You don't have to agree with official or otherwise authoritative sources without question. (not that I wish in any way to compare Mr Crockford or whoever runs json.org with DPRK or its leadership - I'm just using a deliberatly extreme example to highlight that what is written may not be what is, at least not from absolutely everyone's point of view)
- oldmanjay 10y ago
  I hope you feel clever, because in context, this is far too inapplicable to the discussed reality of JSON to be an actual point.

kazinator 10y ago

A notation for a data structure is a serialization format. > In Lisp, that Sexpr will be crawled for data, or maybe even executed. A s-expression cannot be executed; it's just text. The object which it denotes can be walked or executed via eval. Before that happens, the s-expression must be converted to that object. In other words, deserialized by the reader.

qwertyuiop924 10y ago
Good point kaz. I meant that the notation signifies a specific set of general data structures, unlike XML, which doesn't specify the data-structure used in memory afaik, only the actual tree structure of the data itself.
ohyes 10y ago
In that s-expressions are a notation for computation, they can be executed by an interpreter. This is what we refer to as execution. Even assembly is like this, there's no other reasonable way for it to work right now. This feels like you're hairsplitting to no obvious benefit other than increasing confusion. I could as easily say "mathematical notation isn't math it's just text, you can't evaluate '1 + 2' without a human being because otherwise those are just marks on the page otherwise." This is true(ish) (with the correct escaping), but it's difficult for me to see how it's relevant to the discussion? We could imagine the situation where I've created the Texpr, which has slightly different notation but the same properties. I don't know that we would necessarily classify it differently or treat it differently. This leads to the alternate conclusion that maybe the s-expression is the underlying set of objects in the interpreter/compiler. (S-expressions are a special type of linked list, in that case). This rings true(er) to me, because of the way that we talk about s-expression manipulation in lisps. We most certainly are not using string operations to generate them. In which case the thing with the many parenthesis is merely standard lisp syntax, not s-expressions. This is further justified by the existence of different syntax in Dylan or Clojure, and availability of reader macro manipulation as its own entity. Tldr; it most certainly is not text! Nor can it be executed.
- qwertyuiop924 10y ago

jerf 10y ago

JSON is a serialization format that was based on the data structure notation for Javascript. It is, however, a serialization format. Javascript objects are a superset of JSON, as they can contain arbitrary objects and functions, which JSON can not, and "true" Javascript object notation can elide quote marks or use apostrophes for keys, whereas JSON strictly specifies double-quotes around keys. The problem that arises in Python and other dynamically-typed languages is that there exists a default deserialization that is so good it very strongly tempts the programmer to use that exclusively. However, as good as the default serialization may be, it's also quite dangerous, for the reasons you mention and more. In strongly-typed languages there's a stronger focus on parsing the JSON instead, which has the advantage of producing objects with stronger guarantees (which isn't quite the same as pointing out there's stronger typing here, you could theoretically get the same guarantees in Python with some sort of JSON schema library or something), but has the disadvantage of generally being more challenging, since it's hard to beat the conciseness of "json.loads(s)" in Python. There generally is a default serialization in strongly-typed languages, but it's far more likely to become inconvenient if you need anything beyond simple numbers and strings, and people generally learn to prefer true deserialization in my experience, unless they, alas, start their program out from day 1 inputting and outputting JSON and accidentally structure their entire program around the default JSON structures and end up with the exact same problems as you'd get in Python. But as long as JSON isn't the very first thing to go in you're generally in better shape. (I have witnessed a Java program primarily written as a map of string to map of string to map of string to string. It was unsalvagable. And for all the cynicism I may occasionally muster, I don't say that often, because refactoring can be pretty powerful in Java, but this was beyond help. It actually had no JSON in site, but the same fundamental forces were in play.) Personally, despite preferring the more strongly typed approach in most ways, I must confess that when I'm working in Perl I am generally unable to resist the temptation to just JSON::XS::decode_json, and cover over the differences with unit testing rather than dealing with "true" deserialization. I make myself feel better by also telling myself that if I do anything else, I will confuse my fellow programmers who don't generally expect to see fancy deserialization routines when dealing with JSON, which is true enough, but in my heart I still know guilt.

czardoz 10y ago