4 ms·

All this effort on building a better JSON parser for language 'X' should have been better spent on inventing a statically-typed JSON. (Type inference seems like

All this effort on building a better JSON parser for language 'X' should have been better spent on inventing a statically-typed JSON. (Type inference seems like a thing that should work well in this space.)

alexzender 6y ago
True, miss it also sometimes. It will not be as lightweight though - reminds me about SOAP/WSDL times. GraphQL goes in this direction by enforcing type checks, but is designed for web frontends...
jcelerier 6y ago
> All this effort on building a better JSON parser for language 'X' should have been better spent on inventing a statically-typed JSON how does that help when you want to implement a JSON-based protocol ?
- otabdeveloper4 6y ago
  I'd guess that 99.9% of those protocols are actually statically-typed. E.g., an object's "foobar" field is always an array of int, and won't suddenly become a string from one invocation to the next. It seems incredibly strange that we're somehow not leveraging this.
  - fnord123 6y ago
    What you're saying exists. Protobuf, flatbuffers, capnproto, xdr, parquet, arrow, avro, orc, sqlite. Indeed if you use these formats you will be able to load the data into memory more quickly than using JSON. But this article isn't about those protocols. It's about JSON.
    - chrismorgan 6y ago
      You can do it with JSON too. Various statically-typed languages do JSON parsing straight into structs all the time. (e.g. in the Rust ecosystem, Serde is the most popular implementation of this concept.)
  - redmorphium 6y ago
    Because it's relatively easy to make JSON-serialization fast if you know the shape ahead-of-time. It's a solved problem. Here's a pure-JS solution: https://github.com/fastify/fast-json-stringify https://github.com/fastify/fast-json-stringify
  - Semaphor 6y ago
    > E.g., an object's "foobar" field is always an array of int, and won't suddenly become a string from one invocation to the next. If only… I had to figure out how to deserialize a JSON API where a field could be: a) an object X, b) an array of X, or c) a string (mapping to X.Y)
    - chrismorgan 6y ago
      That doesn’t need to be a problem. In Rust, the most popular thing in this space is Serde, with the serde_json crate. It can cope with such things just fine; define a data type that is capable of representing all the possibilities (perhaps as simple as an enum with a variant for each style, or perhaps it normalises them into one), and then… well, you probably have to write a visitor to help deserialise, which isn’t trivial, especially when you’re used to just deriving those things and so you’re not so familiar with Serde internals. But it’s possible. https://serde.rs/string-or-struct.html https://serde.rs/string-or-struct.html gives one example. The length of the example may be daunting, but it’s actually not too bad.
      - Semaphor 6y ago
        It was C#. I ended up writing a custom converter that converted all 3 cases into X[]
masklinn 6y ago
> a statically-typed JSON Do you mean "JSON with a schema" (there are multiple such) or "JSON decoding to static types" (without reflection).
- otabdeveloper4 6y ago
  The latter. JSON is popular because it's schemaless; storing and versioning schemas is its own nightmare of inconsistent and inefficient hacks. However, in practice, JSON messages almost always have an implied schema. Figuring it out on the fly shouldn't be too hard - just apply the JIT and type inference technologies that compilers have already used for decades. (Not a problem with an immediately obvious solution, but that's why we're here, no?)
  - setr 6y ago
    > However, in practice, JSON messages almost always have an implied schema. Typed json already exists in various attempts eg tjson Really I think the problem is that JSON is great when your schema is still in flux -- once it's public/stable, it makes sense to want to strictly encode the current schema. The solution then is that you really want a way to trivially transition from "implied schema" to "strict schema" -- though a typed json won't help you too much because the larger trouble is figuring out what your schema is in practice. I guess you really want either a code analysis tool to determine it, or more likely an automated tool to take eg a list of json responses (probably based on a test suite/code coverage) and produce a typed json schema that you can simply drop into your public API
jcrites 6y ago
Hasn't that been done? Efforts like Amazon's Ion come to mind, or Google's Protocol Buffers & FlatBuffers. http://amzn.github.io/ion-docs/ http://amzn.github.io/ion-docs/ https://developers.google.com/protocol-buffers https://developers.google.com/protocol-buffers https://google.github.io/flatbuffers/ https://google.github.io/flatbuffers/ ProtoBuf requires a schema for the document to be known ahead of time and involves deserialization; Ion is more like JSON in that it's self-describing, but has a rich set of data types as well as annotations on fields to further describe meaning. FlatBuffers requires a schema and provides access to the without parsing/unpacking. On the other hand, if you're building a service and specifying its interface, rather than just specifying a data format, then there are tools like Smithy, gRPC, and Thrift. https://github.com/awslabs/smithy https://github.com/awslabs/smithy https://grpc.io/ https://grpc.io/ https://thrift.apache.org/ https://thrift.apache.org/

alexzender 6y ago

True, miss it also sometimes. It will not be as lightweight though - reminds me about SOAP/WSDL times. GraphQL goes in this direction by enforcing type checks, but is designed for web frontends...

jcelerier 6y ago

> All this effort on building a better JSON parser for language 'X' should have been better spent on inventing a statically-typed JSON how does that help when you want to implement a JSON-based protocol ?

otabdeveloper4 6y ago
I'd guess that 99.9% of those protocols are actually statically-typed. E.g., an object's "foobar" field is always an array of int, and won't suddenly become a string from one invocation to the next. It seems incredibly strange that we're somehow not leveraging this.
- fnord123 6y ago
  What you're saying exists. Protobuf, flatbuffers, capnproto, xdr, parquet, arrow, avro, orc, sqlite. Indeed if you use these formats you will be able to load the data into memory more quickly than using JSON. But this article isn't about those protocols. It's about JSON.
  - chrismorgan 6y ago
    You can do it with JSON too. Various statically-typed languages do JSON parsing straight into structs all the time. (e.g. in the Rust ecosystem, Serde is the most popular implementation of this concept.)
- redmorphium 6y ago
  Because it's relatively easy to make JSON-serialization fast if you know the shape ahead-of-time. It's a solved problem. Here's a pure-JS solution: https://github.com/fastify/fast-json-stringify https://github.com/fastify/fast-json-stringify
- Semaphor 6y ago
  > E.g., an object's "foobar" field is always an array of int, and won't suddenly become a string from one invocation to the next. If only… I had to figure out how to deserialize a JSON API where a field could be: a) an object X, b) an array of X, or c) a string (mapping to X.Y)
  - chrismorgan 6y ago
    That doesn’t need to be a problem. In Rust, the most popular thing in this space is Serde, with the serde_json crate. It can cope with such things just fine; define a data type that is capable of representing all the possibilities (perhaps as simple as an enum with a variant for each style, or perhaps it normalises them into one), and then… well, you probably have to write a visitor to help deserialise, which isn’t trivial, especially when you’re used to just deriving those things and so you’re not so familiar with Serde internals. But it’s possible. https://serde.rs/string-or-struct.html https://serde.rs/string-or-struct.html gives one example. The length of the example may be daunting, but it’s actually not too bad.
    - Semaphor 6y ago
      It was C#. I ended up writing a custom converter that converted all 3 cases into X[]

masklinn 6y ago

> a statically-typed JSON Do you mean "JSON with a schema" (there are multiple such) or "JSON decoding to static types" (without reflection).

otabdeveloper4 6y ago
The latter. JSON is popular because it's schemaless; storing and versioning schemas is its own nightmare of inconsistent and inefficient hacks. However, in practice, JSON messages almost always have an implied schema. Figuring it out on the fly shouldn't be too hard - just apply the JIT and type inference technologies that compilers have already used for decades. (Not a problem with an immediately obvious solution, but that's why we're here, no?)
- setr 6y ago
  > However, in practice, JSON messages almost always have an implied schema. Typed json already exists in various attempts eg tjson Really I think the problem is that JSON is great when your schema is still in flux -- once it's public/stable, it makes sense to want to strictly encode the current schema. The solution then is that you really want a way to trivially transition from "implied schema" to "strict schema" -- though a typed json won't help you too much because the larger trouble is figuring out what your schema is in practice. I guess you really want either a code analysis tool to determine it, or more likely an automated tool to take eg a list of json responses (probably based on a test suite/code coverage) and produce a typed json schema that you can simply drop into your public API

jcrites 6y ago

Hasn't that been done? Efforts like Amazon's Ion come to mind, or Google's Protocol Buffers & FlatBuffers. http://amzn.github.io/ion-docs/ http://amzn.github.io/ion-docs/ https://developers.google.com/protocol-buffers https://developers.google.com/protocol-buffers https://google.github.io/flatbuffers/ https://google.github.io/flatbuffers/ ProtoBuf requires a schema for the document to be known ahead of time and involves deserialization; Ion is more like JSON in that it's self-describing, but has a rich set of data types as well as annotations on fields to further describe meaning. FlatBuffers requires a schema and provides access to the without parsing/unpacking. On the other hand, if you're building a service and specifying its interface, rather than just specifying a data format, then there are tools like Smithy, gRPC, and Thrift. https://github.com/awslabs/smithy https://github.com/awslabs/smithy https://grpc.io/ https://grpc.io/ https://thrift.apache.org/ https://thrift.apache.org/