7 ms·

Let's stop pretending JSON isn't popular because it lets us be loosey-goosey with our specifications. Writing specifications is hard and time consuming, and get

• jonathanaird 6y ago
  I really want something that’s strongly typed but doesn’t require code generation like protobufs do. Yaml doesn’t do it for me. The closest I can get is putting the type guarantees in the database and using GraphQL.
  • speedgoose 6y ago
    I use JSON Schema to validate JSON documents. https://json-schema.org/ https://json-schema.org/
    • nly 6y ago
      Imho, statically typed languages are the ones that benefit most from schema. The current schema version is 12 but the implementations for Go, Rust, C++ and Java are all listed as draft 7. None of them support codegen either, just validation, so not exactly compelling.
      • lloeki 6y ago
        I benefit greatly from schema validation in Ruby, ensuring that ingress-processing code does not receive e.g a String or Hash instead of an Array which would have things blow up way after the ingress edge when a call to an Array method fails, or worse, produce silently broken behaviour that may or may not blow up even farther down the road because both String and Hash respond to e.g #[](Integer).
        • nly 6y ago
          Yeah but Ruby is a dynamically typed language. There's not much benefit to codegen since nothing is checked at compile time anyway.
          • lloeki 6y ago
            I found code generation to be useful in Ruby with protobuf. This: https://github.com/lloeki/ruby-skyjam/blob/master/defs/skyjam/music_manager/export_tracks_request.proto https://github.com/lloeki/ruby-skyjam/blob/master/defs/skyja... gives that: https://github.com/lloeki/ruby-skyjam/blob/master/lib/skyjam/music_manager/export_tracks_request.pb.rb https://github.com/lloeki/ruby-skyjam/blob/master/lib/skyjam... I would certainly enjoy having a DSL to write descriptive code to validate using JSON schema, but it would be even better if the Ruby definitions could be generated and persisted in Ruby files using that DSL. Also, storing things in basic hash/array types works, but having dedicated types is useful, so that one can ensure not shoving one kind of hash in place of another unrelated kind of hash. As for types themselves in general, there's RBS and Sorbet. One could have type definition generation as well for even deeper static and runtime checks.
      • speedgoose 6y ago
        Do you really want generated code to manipulate JSON? I'm not sure there is a demand for that.
        • nly 6y ago
          Manipulating anything dynamic in a statically typed language is generally tedious and not type safe, so yes.
  • nly 6y ago
    Strongly typed + no code generation is obviously doable in any dynamically typed language. Apache Avro has support for parsing and utilizing schemas at runtime, even in C++. For Apache Thrift you have things like thriftpy: https://thriftpy.readthedocs.io/en/latest/ https://thriftpy.readthedocs.io/en/latest/ I'm not aware of a type-safe mechanism for Flatbuffers or Protocol Buffers.
  • kstenerud 6y ago
    I made https://concise-encoding.org/ https://concise-encoding.org/ to deal with this: - strongly typed - ad-hoc or schema (your choice) - no code generation step - edit in text, send in binary
    • PixyMisa 6y ago
      That looks pretty good, actually.
    • fanf2 6y ago
      Why would someone choose this rather than msgpack or CBOR or protobuf or any of the other existing things in that space?
      • kstenerud 6y ago
        Because there isn't anything else in this space that: - supports ad-hoc data structures or schemas per your preference - supports all common types natively (doesn't require special string encoding like base64 or such nonsense) - supports comments, metadata, references (for recursive/cyclical data), custom types - doesn't require an extra compilation step or special definition files - Has parallel binary and textual forms so that you're not wasting CPU and bandwidth serializing/deserializing text. Everything stays in binary except in the rare cases where humans want to look or edit.
  • devj 6y ago
    And there's Amazon Ion too - https://amzn.github.io/ion-docs/ https://amzn.github.io/ion-docs/ "Amazon Ion is a richly-typed, self-describing, hierarchical data serialization format offering interchangeable binary and text representations. The text format (a superset of JSON) is easy to read and author, supporting rapid prototyping. The binary representation is efficient to store, transmit, and skip-scan parse. The rich type system provides unambiguous semantics for long-term preservation of data which can survive multiple generations of software evolution."
  • flohofwoe 6y ago
    There are protobuf libraries without code generation, for instance: https://github.com/cloudwu/pbc https://github.com/cloudwu/pbc (you lose the connection to the language's type system though).
• bawolff 6y ago
  I think JSON demonstrates that most of the time you don't need it, and doing that sort of thing badly is worse than not doing it at all. E.g. XML has all that, and yet the average use of xml is much more fragile thsn the average use of json. Similarly asn.1 has all that, but does anyone actually like the fact the x509 certs use it? (To be clear, json would definitely not be appropriate for that case either).
  • vidarh 6y ago
    There are both JSON [1] and XML [2] encoding rules specified for ASN.1... I have never seen it used in the wild, but realising it's a thing is horrifying enough. [1] https://www.itu.int/rec/T-REC-X.697/en https://www.itu.int/rec/T-REC-X.697/en [2] https://www.itu.int/rec/T-REC-X.693/en https://www.itu.int/rec/T-REC-X.693/en
    • nly 6y ago
      I've used BER encoded ASN.1 at work quite a bit, and the reason you won't see it much are pretty much: - HORRIBLE schema format. - Very few FOSS libraries, most of which are terrible. I used Lev Walkins asn1c[0], which I patched slightly and coupled with some semi-generic code that walks the generated data structures to emit JSON (for debugging purposes). [0] https://github.com/vlm/asn1c https://github.com/vlm/asn1c
      • vidarh 6y ago
        I've implemented (a subset of) ASN.1 for an SNMP server I wrote for an embedded monitoring system, so I know the pain, but it was one thing to implement it for a compact binary format. Inflicting it on a text-based format just seems like it'd multiply the pain...
        • nly 6y ago
          The resulting JSON was pretty clean, no worse than you'd get out of Flatbuffers or Protocol Buffers JSON encoding. It was very handy to be able to use jq on the data.
• secondcoming 6y ago
  Json does have schemas
  • q3k 6y ago
    The JSON schema ecosystem is a mess. json-schema.org is one thing, openapi is another (it uses an 'extended subset' of json-schema.org, which is another way of saying 'incompatible'), there’s also typeschema, and a bunch of other pet projects. There is no official JSON schema RFC, there is no official linking between json-schema.org and the JSON RFC. There is no push to make every language that supports JSON to also support a JSON schema system. Finally, there isn't even a guarantee that all human-readable JSON document specifications can be expressed within a JSON schema in any sensible way.
    • devj 6y ago
      OpenAPI 3.1 which was released recently, uses JSONSchema 2020-12 as the primary schema format. The incompatibilities has been completely resolved as of today.
    • majkinetor 6y ago
      XML schema is joy ?
  • jillesvangurp 6y ago
    Sure, they are just not that widely used. As in, I've not really encountered much json with schema in the wild in my career. I know it exists (in several forms). Support for schemas in mainstream parsers is pretty much non existent making their use very optional as you pretty much have to jump through hoops to do anything with them. Most people seem to opt not to do that. I never got anything out of xml validation either. It just lead to this ridiculously verbose garbage xml that was even less readable with all the namespace, namespace declarations, etc. Very tedious to write manually as well. Also lots of weirdness doing e.g. xpath or xsl transformations against that. The whole SOAP / web services bubble eventually imploded when people figured that you could send tiny json objects instead via REST. I have many issues. People sending invalid json to my APIs is not one of them. You get a bad request if you do. End of story. Don't don that if you don't like bad requests. It's not a problem that justifies a lot of over engineering. Json, yaml, toml, hocon, etc. are basically all just variants of attempting to send human editable blobs of information. Fine for apis where all of the requests are created by programs. Unfortunately people also abuse them for things like DSLs that are authored by humans. The real problem there is not using a strongly and statically typed language that simply does not allow illegal things. A schema is just a stop gap solution when you don't have that. Kotlin is actually great for creating proper DSLs. You get IDE auto-completion and red squiggly lines when you do it wrong. I think Rust also has some nice syntactical constructs for creating DSLs. Typescript might also emerge as a language that is very suitable for that (with maybe a few more features borrowed from Kotlin). Using a non compiled language with weak typing kind of defeats the purpose. Hence the endless ruby but not quite ruby like DSLs for things like puppet.
• cout 6y ago
  I think the reason JSON caught on outside of the javascript community is because it provided a succinct way to represent tree structures of common types (numbers, strings, dictionaries, and lists), using a syntax familiar to anyone who has used C, Perl, Python, Ruby, or Java. S-expressions might have caught on for this purpose, but they lacked a killer app (jquery, and later rails). Like JSON, they are easy to parse and easy to generate, but being more loosely-specified than JSON, it's less clear how to map a given S-expression to a native type. As a glue format, JSON feels just right, even if it's a bit picky sometimes (e.g. trailing commas). For anyone who wants extra features like comments, YAML is the oldest popular format I am aware of that is a superset of JSON. For any new format to succeed, IMO it needs to sufficiently distinguish itself from both YAML and JSON, and not just support a feature set that happens to lie somewhere between the two.
  • biztos 6y ago
    I think the reason JSON caught on outside the Javascript community is because it caught on so massively inside the Javascript community, at a time when there was a lot more crossover between people working on the javascript layer of web apps and people working on the server side (back then, almost always in a different language). I was doing mostly Perl and Javascript when it caught on, and to this day I have very mixed feelings: it's wasteful of space but still doesn't allow comments; its type system is basically a technical-debt generator; and for all that people still get it wrong pretty often. On the other hand, it's more or less human readable for simple data structures and it's more or less everywhere. My hunch is that for a new format to take off it wouldn't so much need to not fall between YAML and JSON: it would need to be the default format of something with such super exponential growth that even us oldies would have to use it.
  • rualca 6y ago
    > I think the reason JSON caught on outside of the javascript community is because it provided a succinct way to represent tree structures of common types (numbers, strings, dictionaries, and lists), using a syntax familiar to anyone who has used C, Perl, Python, Ruby, or Java. I believe it's hard to explain JSON's popularity and wide adoption without talking about javascript. With javascript, JSON was right from the start an ‘eval’ away from being parsed. The barrier to entry to adopt it simply was never there. Once you start to expose JSON APIs to clients, other servers also start to need to consume data from those servers. Rinse and repeat until you reach mass adoption.
• flohofwoe 6y ago
  What programming languages and platforms will your SDK be compatible with, will you also be around to port the SDK to new platforms in twenty or thirty years? Those are good reasons to agree on a standardized data format instead of a specific implementation. Pretty much every language on the planet has JSON parser libraries, and if not, it's fairly trivial to write your own (or convert the JSON data to something else).
  • q3k 6y ago
    > Pretty much every language on the planet has JSON parser libraries. Yes, but they all implement the JSON RFC slightly differently, or implement it in a totally non-compliant way.
    • johannes1234321 6y ago
      As long as you set a limit on integer sizes (in most cases an arbitrary low limit should be fine) and don't expect any precision on float (each conversion between text and binary representation there is problematic) and don't use extensions (like comments or trailing commas) you should be quite fine with most JSON libraries. This won't cover all things, but a lot.
      • q3k 6y ago
        It's not a great standard if every program/library using it must explicitly state its limits for such basic things as number values. It's very easy to do JSON wrong, to not even be aware of doing it wrong, and do it wrong in a way that limits its interoperability (see: the default behaviour of the json library in Python). This is not theoretical either, just google for 'json nan github' to see the hundreds of production programs accidentally emitting JSON that cannot be ingested by RFC-compliant parsers.
        • johannes1234321 6y ago
          I agree that as a standard it isn't great. However the fact that it is so successful shows that the minimalistic design has benefits for adoption. And regarding integer ranges: Any user has restrictions on top of the generic format. Some fields have to be present for the application to work, some fields have to be a string, others an array. Some integer has to be between 0 and 100, some array has to have 5 elements. Given that 99% of integers in practice fit in a signed int32 there is little problem (97% are probably 0, 1% are 1, 0.5% are -1 and only the rest other values ...) If you are on the edge you have to know and work-around ...