8 ms·

Rich Hickey: extensible data notation

219 points by Peteris 14y ago

tree_of_item 14y ago
Could the blessed days of the s-expression interchange format finally be here?
ChuckMcM 14y ago
http://tools.ietf.org/html/rfc4506 http://tools.ietf.org/html/rfc4506 That is 'XDR' (eXtensible Data Representation) which has similar goals and is reasonably mature. Of course JSON (http://www.ietf.org/rfc/rfc4627.txt http://www.ietf.org/rfc/rfc4627.txt) does this as well but using character code points. Not to mention XML and ASN.1. In perl there is YaML (http://search.cpan.org/dist/YAML/ http://search.cpan.org/dist/YAML/) too. I'm wondering what this one brings to the table. The readme file doesn't say.
- guns 14y ago
  JSON is not extensible and syntactically more verbose. However, the syntax corresponds exactly to many modern dynamic languages and is now nearly ubiquitous. > Of course JSON (http://www.ietf.org/rfc/rfc4627.txt http://www.ietf.org/rfc/rfc4627.txt) does this as well but using character code points. I hope Rich will define "alphanumeric" as Unicode letter and number classes. Clojure itself is ambiguous on valid symbol identifiers. http://clojure.org/reader http://clojure.org/reader has the same "alphanumeric plus some punctuation" rule, but the actual reader implementation [1] is extremely permissive: static Pattern symbolPat = Pattern.compile("[:]?([\\D&&[^/]].*/)?([\\D&&[^/]][^/]*)"); [1]: https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/LispReader.java#L61 https://github.com/clojure/clojure/blob/master/src/jvm/cloju...
- deleted 14y ago
  [deleted]
- prakashk 14y ago
  > In perl there is YaML ... YAML is not just limited for use in Perl, nor is it as closely related to Perl as JSON is to Javascript. One of its creators, Ingy döt Net, was (and still is) an active member of the Perl community, so Perl had excellent support for YAML from the very beginning. From http://www.yaml.org/about.html http://www.yaml.org/about.html: The founding members of YAML are Ingy döt Net (author of the Perl module Data::Denter), Clark Evans, and Oren Ben-Kiki. YAML emerged from the union of two efforts. The first was Ingy döt Net's need for a serialization format for Inline, this resulted in his Data::Denter module. The second, was the joint work of Oren Ben-Kiki Clark Evans on simplifying XML within the sml-dev group. YAML was first publicized with a <?xmlhack?> article on 12 May 2001. Oren and Clark's vision for YAML was very similar to Ingy's Data::Denter, and vice versa, thus a few days later they teamed up and YAML was born.
- andrewflnr 14y ago
  Your link says "external", not "extensible", and in fact seems to make no mention of extensibility.
  - ChuckMcM 14y ago
    Absolutely correct, brain fart. The description language was designed to represent any data structure you could represent in C, in XDR but more importantly insure that moving those structures across a network between disparate architectures would return them the natively 'correct' format when received.
falcolas 14y ago
So, XML, YAML, JSON, and the dozen plus other markup notations were not suitable because they used C style delimiters instead of Lisp style delimiters? Snark aside, what does this buy us that these existing markup languages don't? I appreciate all that Rich Hickey does for programmers, but aside from his name, this just adds to the noise that already exists in this domain.
- mattdeboard 14y ago
  I suspect it's just a tool, and the Datomic folks thought they'd make it available. No one said it was going to change the world, right? What does it buy "us"? Well, it doesn't buy "us" anything. They needed a data interchange format and I assume it made more sense to use Clojure's primitives than parsing up JSON. Why would a project written using a homoiconic language use anything but that language to exchange data between its components? edit: Said best by this tweet by fogus: "Clojure devs have been using #Clojure data as an interchange format all along. ..." https://twitter.com/fogus/status/243913831242399744 https://twitter.com/fogus/status/243913831242399744
  - d0m 14y ago
    I suppose the "buy us" part was meant because it was the first rank on HN. (I.e. As if it's a big deal.)
    - dmix 14y ago
      Plenty of language / tookit specific articles reach the top of HN. There are quite a few people here interested in clojure and Rich Hickey's work.
  - lloeki 14y ago
    > They needed a data interchange format and I assume it made more sense to use Clojure's primitives than parsing up JSON. Even in JavaScript it's better to parse JSON instead of eval'ing, because you don't want to execute stuff that would "happen" to be contained in the data interchange format.
jacobolus 14y ago
Nice advantages over JSON: more compact, easier to pretty print, includes an integer type, non-string map keys, has a nice built-in extension mechanism (which is much more elegant than any ad-hoc thing that JSON can support). Things that probably make sense coming from Clojure, but seem somewhat unnecessary for a general purpose data interchange format: explicit character type (as compared to length 1 strings (which could optionally use the extension mechanism if necessary)), separate types for vectors and lists (seems like the extension mechanism could handle this if it’s ever necessary; to some extent this criticism holds for sets too, but those are also more independently useful). One type not included that I find useful: some kind of "raw" string wherein backslashes are interpreted literally, and double escapes aren’t required all over the place. Possible point of confusion that should be spelled out more explicitly: by the grammar provided, a floating point number requires an explicit leading digit. That is, '0.5' cannot be spelled '.5'. (Should an implementation accept '.5' as a number, or reject it as badly formed?) Also, does "a floating-point number may have the suffix M to indicate that exact precision is desired" mean that it should be interpreted as a decimal number? Might be worth saying that directly. It would be nice to see a bit more guidance about "Symbols are used to represent identifiers, and should map to something other than strings, if possible." Perhaps this could include examples of what Rich Hickey & al. think would be useful interpretations in JavaScript and Python (to pick two obvious popular examples). Most of all, it would be nice to see a clear explicit treatment of Unicode for strings/symbols (I’d recommend utf-8 here), including possible ways of escaping code points in strings. Confusions about Unicode are one of the main points of incompatibility between JSON implementations, and the JSON spec has more to say about the subject than this current spec does. One nice built-in tag to add: base64 raw data, using one of the typical base64 encodings, which should be described in the spec to avoid any confusion. Question: if a tagged element is considered a unit, can another tag be put in front of one? That is, something along the lines of '#outer_tag #inner_tag "built-in element"', where in the code which interprets the file, whatever object is produced by the inner_tag's extension is sent as input to the outer_tag's? It’s worth clarifying this so that implementors make sure to add the case to their test suites.
- alexchamberlain 14y ago
  .5 cannot be a symbol, so it would make sense to add it to the grammar.
snprbob86 14y ago
I added a spot on the wiki for links to implementations: https://github.com/richhickey/edn/wiki/Implementations https://github.com/richhickey/edn/wiki/Implementations A few months ago, I started doing an implementation of a Clojure reader in C and a Ruby extension, but I ran out of time to work on it. If anyone wants to use my code as a starting point, please do! I'd love to see more usage of Clojure forms... errr... "EDN" in the wild!
michaelsbradley 14y ago
It seems one immediate application is in Datomic's new REST API: Datomic gets a REST API http://news.ycombinator.com/item?id=4487467 http://news.ycombinator.com/item?id=4487467 I'm developing with Clojure on a daily basis at present, and it's cool to see such a slick data notation take on new life outside of my REPLs and .clj files. Now, what would be really interesting is if edn (or an official superset of it) could be formalized with "hypermedia controls". See: Hypermedia-Oriented Design http://amundsen.com/articles/hypermedia-oriented-design/ http://amundsen.com/articles/hypermedia-oriented-design/ One of the shortcomings of JSON is that as a standardized hypermedia type, it doesn't offer any hypermedia controls. There are efforts to standardize JSON-derived types which do provide such controls: HAL http://tools.ietf.org/html/draft-kelly-json-hal-03 http://tools.ietf.org/html/draft-kelly-json-hal-03 JSON-LD http://json-ld.org/ http://json-ld.org/ Collection+JSON http://www.amundsen.com/media-types/collection/ http://www.amundsen.com/media-types/collection/ It would be great to see edn take on that challenge in its early days as an extra-clojure, general purpose data notation. I'm convinced that Fielding is right about REST implying HATEOAS (others argue for "practical REST"), but you can't robustly implement REST/HATEOAS APIs with a media type that outright lacks hypermedia controls.
- jacobolus 14y ago
  Would you mind suggesting some possible formats for this, and explaining a bit what’s necessary for “hypermedia controls”? Or is there a link somewhere that defines this more clearly? The hypermedia-oriented design link was kind of abstract.
  - michaelsbradley 14y ago
    I'm not sure about a format just yet, but I'm fairly certain it would be a superset of edn (as opposed to imposing it on "plain old edn"). The Document Format spec for Collection+JSON is probably not a bad starting point: http://amundsen.com/media-types/collection/format/ http://amundsen.com/media-types/collection/format/ Seems like it could readily be adapted to edn syntax. But I find myself drawn to the RDF concepts underlying JSON-LD, and wonder if the C+J and LD ideas could be melded together. A solid exploration of the larger concepts involved in hypermedia controls can be found in Mike Amundsen's recent book: Building Hypermedia APIs with HTML5 and Node http://shop.oreilly.com/product/0636920020530.do http://shop.oreilly.com/product/0636920020530.do The basic idea is that, per Fielding, HATEOAS is an aspect of REST that shouldn't be ignored. Standardized media types with well-defined hypermedia controls allow clients that understand those standards to make state transitions without having to "know other rules" that aren't contained in the representations themselves. Our web browsers do this all the time when we click on links and submit forms that we find in web pages. Based on the X/HTML standard, ours browsers (hypermedia clients) know what to do with links and forms, i.e. how to initiate the appropriate state transitions (build and run GET/POST requests) based on the markup semantics. Now, if a client is driven programmatically (i.e. not by a human reasoning about the appearance and purpose of a form-button labeled "login", etc.) then you need a bit more besides links and forms. This is where attributes like "rel" (for `a` and `link`) become important. There's been a lot of work in that area, e.g. Microformats and RDFa, among others.
6ren 14y ago
Extreme expressiveness makes a data-format harder to understand at a glance; and data is generally dumb enough not to need it. Examples are helpful. An example of an example: http://www.sinatrarb.com/ http://www.sinatrarb.com/
- loqi 14y ago
  {"follows": ["ben", "billy-jo", "anon"] ,"message-counts": {"billy-jo-ben": 2}} At a glance, is the order of follows relevant? Should it be preserved? Is "anon" the same kind of thing as "ben" or "billy-jo"? Is the key in message-counts another name, or some collection thereof?
- josteink 14y ago
  Extreme expressiveness makes a data-format harder to understand I would argue that extreme expressiveness through the most simplistic rules makes a data-format easier to understand than some data-format which is only mediocre-ly expressive through more complex and inconsistent rules. For instance, Lisp/S-expressions typically has a much simpler data-format than C and is much easier to learn from end to end. The complexities associated with Lisp-code can not and should not be attributed to it's syntax, but rather that most Lisp-code is written to be purely functional and non-procedural. While procedural and/or stateful C/C++/Java/C#-code might be easier to understand for a C/C++/Java/C#-programmer, I don't think you would find any of those programmers arguing that S-expression syntax is harder to grasp and master than the complex syntax of C-based languages.
skrebbel 14y ago
i think that one of the reasons why JSON got so popular is that it was artificially restricted to a smaller language than the actual object notation in JavaScript. This meant that it was easy to write a parser for which, in turn, happened a lot. I see lots of optional extras here that might make humans a little happier but may increase the chance that different implementations are incompatible. YAML had this problem, too.
Groxx 14y ago
<redacted> I'll stick with JSON for one simple reason: you can have anything as a key (as long as it's escaped). The fact that this can't means you either a) can't use it if you have more-complex keys for some reason, or b) you have to use some non-standardized packing format to encode your illegal keys. </redacted> edit: thanks repliers, I missed part of the spec. Ignore!
- lmkg 14y ago
  False. You can use anything as a key. > Note that keys and values can be elements of any type. The sample code also uses a vector as a key.
  - Groxx 14y ago
    Ah, I missed that part. I was going off the keyword == symbol rules, which are waaaaay more restrictive. Thanks!
- andrewflnr 14y ago
  Did you read the spec? "... keys and values can be elements of any type".
- zcam 14y ago
  Well if by anything you mean Strings, that is very limited. 2.2. Objects An object structure is represented as a pair of curly brackets surrounding zero or more name/value pairs (or members). A name is a string. A single colon comes after each name, separating the name from the value. A single comma separates a value from a following name. The names within an object SHOULD be unique. http://www.ietf.org/rfc/rfc4627.txt http://www.ietf.org/rfc/rfc4627.txt
andrewflnr 14y ago
I kind of like the tagging syntax. It almost seems like they should really be lists with the first tag in function position, but I don't know think that would interop with real Clojure. I tried to come up with something similar, but somehow couldn't generalize to tagging any random object. When in doubt, generalize... PS: I'm just learning Clojure.
- threedaymonk 14y ago
  If you mean something like this, it would interoperate just fine: (list 1 2 3) (vector 1 2 3) (hash-set :a :b :c) (hash-map :a 1 :b 2) I suspect that typing hash-map all over the place might get old fast, though.
  - andrewflnr 14y ago
    I mean `(#myapp/person {:first "Fred" :last "Mertz"})' or `(#seconds 330)'. In this case it's probably the extra parentheses that get old fast. Even so, having semantically connected items not be stuck together makes me a bit queasy.
Flow 14y ago
I get that it's a richer format than JSON, with the introduction of sets, dicts and so on, but what I felt wasn't explained was the "extensible" part. Can someone give an example of this?
- martinwnet 14y ago
  It's extensible in the same way that XML (Extensible Markup Language) is extensible.
- yusefnapora 14y ago
  I think it refers to the use of tags, which allow you to register tag handlers to add custom semantics to whatever data structure follows the tag.
  - Flow 14y ago
    Ok, as a trigger. But isn't it backwards to have the generator suggest where to put trigger points instead of the consumer?
    - richhickey 14y ago
      > But isn't it backwards to have the generator suggest where to put trigger points instead of the consumer? No. edn is self-describing, and all descriptions bottom out on built-in types. Having each date/instant or extended type proclaim it is an #inst or #whatever means a single handler will ensure they all become proper types on the consumer side, with no knowledge of the application or document structure at all. And if no specific handler is installed, a generic handler can at least ensure that the value returned keeps track that the tagged element was tagged, and how, rather than just silently yielding a string with some encoded cruft in it. The world you are describing is that 'context-sensitive' world mentioned in the edn rationale, where the consuming app must know that the cruft inside the dob: string inside a particular map in a particular context is actually a date or :id is a uuid or whatever, and also know how they represented. Ditto the lastEdited: field, startDate: and foo: fields etc. Are they all the same? Handling any particular document means having complete knowledge of such details. Context-sensitivity greatly complicates applications and thwarts generic processing.
dons 14y ago
As with JSON, no support for algebraic data types (so tagged unions or recursive data types)?
- mattdeboard 14y ago
  How would a union data type work?
- pufuwozu 14y ago
  In a dynamically typed language, every value is a tagged union.
- richhickey 14y ago
  edn is a mechanism for the conveyance of values. It is not a system for defining their semantics, types nor schemas. That said, there should be no problem using edn to convey the values of types whose definitions are algebraic or recursive. In particular: "A tag may specify more than one format for the tagged element, e.g. both a string and a vector representation." admits unions. And the system itself nests arbitrarily, i.e. a tagged element can be defined in terms of others. Values of this type: data Tree = Empty | Leaf Int | Node Tree Tree could be conveyed like this: #tree/Tree Empty #tree/Tree [Leaf 42] #tree/Tree [Node #tree/Tree Empty, #tree/Tree [Leaf 42]]
  - dons 14y ago
    Thanks for the info. Having written wire serialization from typed to untyped formats many times over the years, the limited expressivness of such formats has been an ongoing source of annoyance.
  - dschiptsov 14y ago
    It is only me who has a strong feeling that this vector is an ugly construct here - [Node #tree/Tree Empty, #tree/Tree [Leaf 42]] and traditional S-Expression, which just describes an abstract structure - a list, not a particular data-structure - vector, is much better way? And [Leaf 42]? What is this? Vector? Of elements of arbitrary type? But a list of arbitrary elements is a quite natural and human-mind-friendly concept, and the Lisp notation, which intentionally omits any description of underlying representation, is a great idea.
shaunxcode 14y ago
https://github.com/shaunxcode/jsedn https://github.com/shaunxcode/jsedn This is my attempt at a js version. parse is working - now working on encode. There is really nothing stopping it from working in browser other than I am currently working on it as an npm/node js package. I can successfully load datomic schemas and .dtm files which I think is a pretty good test. Need a lot more tests around int/float wrt the arbitrary precision. I would love to have some pull requests if anyone wants to write valid tests which break it.
icrbow 14y ago
https://bitbucket.org/dpwiz/hedn https://bitbucket.org/dpwiz/hedn My stab at haskell parser/encoder and type converter. Would be glad for a friendly hug^W code review and packaging suggestions.

219 points by Peteris 14y ago

Could the blessed days of the s-expression interchange format finally be here?

http://tools.ietf.org/html/rfc4506 http://tools.ietf.org/html/rfc4506 That is 'XDR' (eXtensible Data Representation) which has similar goals and is reasonably mature. Of course JSON (http://www.ietf.org/rfc/rfc4627.txt http://www.ietf.org/rfc/rfc4627.txt) does this as well but using character code points. Not to mention XML and ASN.1. In perl there is YaML (http://search.cpan.org/dist/YAML/ http://search.cpan.org/dist/YAML/) too. I'm wondering what this one brings to the table. The readme file doesn't say.

guns 14y ago
JSON is not extensible and syntactically more verbose. However, the syntax corresponds exactly to many modern dynamic languages and is now nearly ubiquitous. > Of course JSON (http://www.ietf.org/rfc/rfc4627.txt http://www.ietf.org/rfc/rfc4627.txt) does this as well but using character code points. I hope Rich will define "alphanumeric" as Unicode letter and number classes. Clojure itself is ambiguous on valid symbol identifiers. http://clojure.org/reader http://clojure.org/reader has the same "alphanumeric plus some punctuation" rule, but the actual reader implementation [1] is extremely permissive: static Pattern symbolPat = Pattern.compile("[:]?([\\D&&[^/]].*/)?([\\D&&[^/]][^/]*)"); [1]: https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/LispReader.java#L61 https://github.com/clojure/clojure/blob/master/src/jvm/cloju...
deleted 14y ago
[deleted]
prakashk 14y ago
> In perl there is YaML ... YAML is not just limited for use in Perl, nor is it as closely related to Perl as JSON is to Javascript. One of its creators, Ingy döt Net, was (and still is) an active member of the Perl community, so Perl had excellent support for YAML from the very beginning. From http://www.yaml.org/about.html http://www.yaml.org/about.html: The founding members of YAML are Ingy döt Net (author of the Perl module Data::Denter), Clark Evans, and Oren Ben-Kiki. YAML emerged from the union of two efforts. The first was Ingy döt Net's need for a serialization format for Inline, this resulted in his Data::Denter module. The second, was the joint work of Oren Ben-Kiki Clark Evans on simplifying XML within the sml-dev group. YAML was first publicized with a <?xmlhack?> article on 12 May 2001. Oren and Clark's vision for YAML was very similar to Ingy's Data::Denter, and vice versa, thus a few days later they teamed up and YAML was born.
andrewflnr 14y ago
Your link says "external", not "extensible", and in fact seems to make no mention of extensibility.
- ChuckMcM 14y ago
  Absolutely correct, brain fart. The description language was designed to represent any data structure you could represent in C, in XDR but more importantly insure that moving those structures across a network between disparate architectures would return them the natively 'correct' format when received.

falcolas 14y ago

So, XML, YAML, JSON, and the dozen plus other markup notations were not suitable because they used C style delimiters instead of Lisp style delimiters? Snark aside, what does this buy us that these existing markup languages don't? I appreciate all that Rich Hickey does for programmers, but aside from his name, this just adds to the noise that already exists in this domain.

mattdeboard 14y ago
I suspect it's just a tool, and the Datomic folks thought they'd make it available. No one said it was going to change the world, right? What does it buy "us"? Well, it doesn't buy "us" anything. They needed a data interchange format and I assume it made more sense to use Clojure's primitives than parsing up JSON. Why would a project written using a homoiconic language use anything but that language to exchange data between its components? edit: Said best by this tweet by fogus: "Clojure devs have been using #Clojure data as an interchange format all along. ..." https://twitter.com/fogus/status/243913831242399744 https://twitter.com/fogus/status/243913831242399744
- d0m 14y ago
  I suppose the "buy us" part was meant because it was the first rank on HN. (I.e. As if it's a big deal.)
  - dmix 14y ago
    Plenty of language / tookit specific articles reach the top of HN. There are quite a few people here interested in clojure and Rich Hickey's work.
- lloeki 14y ago
  > They needed a data interchange format and I assume it made more sense to use Clojure's primitives than parsing up JSON. Even in JavaScript it's better to parse JSON instead of eval'ing, because you don't want to execute stuff that would "happen" to be contained in the data interchange format.

jacobolus 14y ago

Nice advantages over JSON: more compact, easier to pretty print, includes an integer type, non-string map keys, has a nice built-in extension mechanism (which is much more elegant than any ad-hoc thing that JSON can support). Things that probably make sense coming from Clojure, but seem somewhat unnecessary for a general purpose data interchange format: explicit character type (as compared to length 1 strings (which could optionally use the extension mechanism if necessary)), separate types for vectors and lists (seems like the extension mechanism could handle this if it’s ever necessary; to some extent this criticism holds for sets too, but those are also more independently useful). One type not included that I find useful: some kind of "raw" string wherein backslashes are interpreted literally, and double escapes aren’t required all over the place. Possible point of confusion that should be spelled out more explicitly: by the grammar provided, a floating point number requires an explicit leading digit. That is, '0.5' cannot be spelled '.5'. (Should an implementation accept '.5' as a number, or reject it as badly formed?) Also, does "a floating-point number may have the suffix M to indicate that exact precision is desired" mean that it should be interpreted as a decimal number? Might be worth saying that directly. It would be nice to see a bit more guidance about "Symbols are used to represent identifiers, and should map to something other than strings, if possible." Perhaps this could include examples of what Rich Hickey & al. think would be useful interpretations in JavaScript and Python (to pick two obvious popular examples). Most of all, it would be nice to see a clear explicit treatment of Unicode for strings/symbols (I’d recommend utf-8 here), including possible ways of escaping code points in strings. Confusions about Unicode are one of the main points of incompatibility between JSON implementations, and the JSON spec has more to say about the subject than this current spec does. One nice built-in tag to add: base64 raw data, using one of the typical base64 encodings, which should be described in the spec to avoid any confusion. Question: if a tagged element is considered a unit, can another tag be put in front of one? That is, something along the lines of '#outer_tag #inner_tag "built-in element"', where in the code which interprets the file, whatever object is produced by the inner_tag's extension is sent as input to the outer_tag's? It’s worth clarifying this so that implementors make sure to add the case to their test suites.

alexchamberlain 14y ago
.5 cannot be a symbol, so it would make sense to add it to the grammar.

snprbob86 14y ago

I added a spot on the wiki for links to implementations: https://github.com/richhickey/edn/wiki/Implementations https://github.com/richhickey/edn/wiki/Implementations A few months ago, I started doing an implementation of a Clojure reader in C and a Ruby extension, but I ran out of time to work on it. If anyone wants to use my code as a starting point, please do! I'd love to see more usage of Clojure forms... errr... "EDN" in the wild!

michaelsbradley 14y ago

It seems one immediate application is in Datomic's new REST API: Datomic gets a REST API http://news.ycombinator.com/item?id=4487467 http://news.ycombinator.com/item?id=4487467 I'm developing with Clojure on a daily basis at present, and it's cool to see such a slick data notation take on new life outside of my REPLs and .clj files. Now, what would be really interesting is if edn (or an official superset of it) could be formalized with "hypermedia controls". See: Hypermedia-Oriented Design http://amundsen.com/articles/hypermedia-oriented-design/ http://amundsen.com/articles/hypermedia-oriented-design/ One of the shortcomings of JSON is that as a standardized hypermedia type, it doesn't offer any hypermedia controls. There are efforts to standardize JSON-derived types which do provide such controls: HAL http://tools.ietf.org/html/draft-kelly-json-hal-03 http://tools.ietf.org/html/draft-kelly-json-hal-03 JSON-LD http://json-ld.org/ http://json-ld.org/ Collection+JSON http://www.amundsen.com/media-types/collection/ http://www.amundsen.com/media-types/collection/ It would be great to see edn take on that challenge in its early days as an extra-clojure, general purpose data notation. I'm convinced that Fielding is right about REST implying HATEOAS (others argue for "practical REST"), but you can't robustly implement REST/HATEOAS APIs with a media type that outright lacks hypermedia controls.

jacobolus 14y ago
Would you mind suggesting some possible formats for this, and explaining a bit what’s necessary for “hypermedia controls”? Or is there a link somewhere that defines this more clearly? The hypermedia-oriented design link was kind of abstract.
- michaelsbradley 14y ago
  I'm not sure about a format just yet, but I'm fairly certain it would be a superset of edn (as opposed to imposing it on "plain old edn"). The Document Format spec for Collection+JSON is probably not a bad starting point: http://amundsen.com/media-types/collection/format/ http://amundsen.com/media-types/collection/format/ Seems like it could readily be adapted to edn syntax. But I find myself drawn to the RDF concepts underlying JSON-LD, and wonder if the C+J and LD ideas could be melded together. A solid exploration of the larger concepts involved in hypermedia controls can be found in Mike Amundsen's recent book: Building Hypermedia APIs with HTML5 and Node http://shop.oreilly.com/product/0636920020530.do http://shop.oreilly.com/product/0636920020530.do The basic idea is that, per Fielding, HATEOAS is an aspect of REST that shouldn't be ignored. Standardized media types with well-defined hypermedia controls allow clients that understand those standards to make state transitions without having to "know other rules" that aren't contained in the representations themselves. Our web browsers do this all the time when we click on links and submit forms that we find in web pages. Based on the X/HTML standard, ours browsers (hypermedia clients) know what to do with links and forms, i.e. how to initiate the appropriate state transitions (build and run GET/POST requests) based on the markup semantics. Now, if a client is driven programmatically (i.e. not by a human reasoning about the appearance and purpose of a form-button labeled "login", etc.) then you need a bit more besides links and forms. This is where attributes like "rel" (for `a` and `link`) become important. There's been a lot of work in that area, e.g. Microformats and RDFa, among others.

6ren 14y ago

Extreme expressiveness makes a data-format harder to understand at a glance; and data is generally dumb enough not to need it. Examples are helpful. An example of an example: http://www.sinatrarb.com/ http://www.sinatrarb.com/

loqi 14y ago
{"follows": ["ben", "billy-jo", "anon"] ,"message-counts": {"billy-jo-ben": 2}} At a glance, is the order of follows relevant? Should it be preserved? Is "anon" the same kind of thing as "ben" or "billy-jo"? Is the key in message-counts another name, or some collection thereof?
josteink 14y ago
Extreme expressiveness makes a data-format harder to understand I would argue that extreme expressiveness through the most simplistic rules makes a data-format easier to understand than some data-format which is only mediocre-ly expressive through more complex and inconsistent rules. For instance, Lisp/S-expressions typically has a much simpler data-format than C and is much easier to learn from end to end. The complexities associated with Lisp-code can not and should not be attributed to it's syntax, but rather that most Lisp-code is written to be purely functional and non-procedural. While procedural and/or stateful C/C++/Java/C#-code might be easier to understand for a C/C++/Java/C#-programmer, I don't think you would find any of those programmers arguing that S-expression syntax is harder to grasp and master than the complex syntax of C-based languages.

skrebbel 14y ago

i think that one of the reasons why JSON got so popular is that it was artificially restricted to a smaller language than the actual object notation in JavaScript. This meant that it was easy to write a parser for which, in turn, happened a lot. I see lots of optional extras here that might make humans a little happier but may increase the chance that different implementations are incompatible. YAML had this problem, too.

Groxx 14y ago

<redacted> I'll stick with JSON for one simple reason: you can have anything as a key (as long as it's escaped). The fact that this can't means you either a) can't use it if you have more-complex keys for some reason, or b) you have to use some non-standardized packing format to encode your illegal keys. </redacted> edit: thanks repliers, I missed part of the spec. Ignore!

lmkg 14y ago
False. You can use anything as a key. > Note that keys and values can be elements of any type. The sample code also uses a vector as a key.
- Groxx 14y ago
  Ah, I missed that part. I was going off the keyword == symbol rules, which are waaaaay more restrictive. Thanks!
andrewflnr 14y ago
Did you read the spec? "... keys and values can be elements of any type".
zcam 14y ago
Well if by anything you mean Strings, that is very limited. 2.2. Objects An object structure is represented as a pair of curly brackets surrounding zero or more name/value pairs (or members). A name is a string. A single colon comes after each name, separating the name from the value. A single comma separates a value from a following name. The names within an object SHOULD be unique. http://www.ietf.org/rfc/rfc4627.txt http://www.ietf.org/rfc/rfc4627.txt

andrewflnr 14y ago

I kind of like the tagging syntax. It almost seems like they should really be lists with the first tag in function position, but I don't know think that would interop with real Clojure. I tried to come up with something similar, but somehow couldn't generalize to tagging any random object. When in doubt, generalize... PS: I'm just learning Clojure.

threedaymonk 14y ago
If you mean something like this, it would interoperate just fine: (list 1 2 3) (vector 1 2 3) (hash-set :a :b :c) (hash-map :a 1 :b 2) I suspect that typing hash-map all over the place might get old fast, though.
- andrewflnr 14y ago
  I mean `(#myapp/person {:first "Fred" :last "Mertz"})' or `(#seconds 330)'. In this case it's probably the extra parentheses that get old fast. Even so, having semantically connected items not be stuck together makes me a bit queasy.

Flow 14y ago

I get that it's a richer format than JSON, with the introduction of sets, dicts and so on, but what I felt wasn't explained was the "extensible" part. Can someone give an example of this?

martinwnet 14y ago
It's extensible in the same way that XML (Extensible Markup Language) is extensible.
yusefnapora 14y ago
I think it refers to the use of tags, which allow you to register tag handlers to add custom semantics to whatever data structure follows the tag.
- Flow 14y ago
  Ok, as a trigger. But isn't it backwards to have the generator suggest where to put trigger points instead of the consumer?
  - richhickey 14y ago
    > But isn't it backwards to have the generator suggest where to put trigger points instead of the consumer? No. edn is self-describing, and all descriptions bottom out on built-in types. Having each date/instant or extended type proclaim it is an #inst or #whatever means a single handler will ensure they all become proper types on the consumer side, with no knowledge of the application or document structure at all. And if no specific handler is installed, a generic handler can at least ensure that the value returned keeps track that the tagged element was tagged, and how, rather than just silently yielding a string with some encoded cruft in it. The world you are describing is that 'context-sensitive' world mentioned in the edn rationale, where the consuming app must know that the cruft inside the dob: string inside a particular map in a particular context is actually a date or :id is a uuid or whatever, and also know how they represented. Ditto the lastEdited: field, startDate: and foo: fields etc. Are they all the same? Handling any particular document means having complete knowledge of such details. Context-sensitivity greatly complicates applications and thwarts generic processing.

dons 14y ago

As with JSON, no support for algebraic data types (so tagged unions or recursive data types)?

mattdeboard 14y ago
How would a union data type work?
pufuwozu 14y ago
In a dynamically typed language, every value is a tagged union.
richhickey 14y ago
edn is a mechanism for the conveyance of values. It is not a system for defining their semantics, types nor schemas. That said, there should be no problem using edn to convey the values of types whose definitions are algebraic or recursive. In particular: "A tag may specify more than one format for the tagged element, e.g. both a string and a vector representation." admits unions. And the system itself nests arbitrarily, i.e. a tagged element can be defined in terms of others. Values of this type: data Tree = Empty | Leaf Int | Node Tree Tree could be conveyed like this: #tree/Tree Empty #tree/Tree [Leaf 42] #tree/Tree [Node #tree/Tree Empty, #tree/Tree [Leaf 42]]
- dons 14y ago
  Thanks for the info. Having written wire serialization from typed to untyped formats many times over the years, the limited expressivness of such formats has been an ongoing source of annoyance.
- dschiptsov 14y ago
  It is only me who has a strong feeling that this vector is an ugly construct here - [Node #tree/Tree Empty, #tree/Tree [Leaf 42]] and traditional S-Expression, which just describes an abstract structure - a list, not a particular data-structure - vector, is much better way? And [Leaf 42]? What is this? Vector? Of elements of arbitrary type? But a list of arbitrary elements is a quite natural and human-mind-friendly concept, and the Lisp notation, which intentionally omits any description of underlying representation, is a great idea.

shaunxcode 14y ago

https://github.com/shaunxcode/jsedn https://github.com/shaunxcode/jsedn This is my attempt at a js version. parse is working - now working on encode. There is really nothing stopping it from working in browser other than I am currently working on it as an npm/node js package. I can successfully load datomic schemas and .dtm files which I think is a pretty good test. Need a lot more tests around int/float wrt the arbitrary precision. I would love to have some pull requests if anyone wants to write valid tests which break it.

icrbow 14y ago

https://bitbucket.org/dpwiz/hedn https://bitbucket.org/dpwiz/hedn My stab at haskell parser/encoder and type converter. Would be glad for a friendly hug^W code review and packaging suggestions.