6 ms·

> Polymorphic JSON. The protocol elements have different data types that convey additional contextual meaning, allowing us to avoid mutually exclusive protocol

> Polymorphic JSON. The protocol elements have different data types that convey additional contextual meaning, allowing us to avoid mutually exclusive protocol elements and design a more succinct and readable protocol. Yeah... let's not please.

thosakwe 6y ago
Yeah... This is the same sort of thing you see with, say, ActivityPub, that makes it a massive pain, if not totally impossible to implement it in a statically-typed language.
oblio 6y ago
I'm not sure I get this, does the data type change depending on context? Is that what they mean?
- golergka 6y ago
  > XYZ’s protocol is not just based on JSON, but it’s based on a particular technique known as polymorphic JSON. In this, a single field could have a different data type in different circumstances. For example, the resources field can be an array when requesting a single access token, or it can be an object when requesting multiple named access tokens. The difference in type indicates a difference in processing from the server. Within the resources array, each element can be either an object, representing a rich resource request description, or a string, representing a resource handle (scope). This is horrible.
  - pwinnski 6y ago
    This is definitely not a protocol dreamed up by, say, Java developers. Obviously it's doable in Java, but I'm hard-pressed to imagine a Java developer would think of such a thing.
    - tasogare 6y ago
      This is a few magnitude even more infuriating than a spec I have to work with at work, where some keys are required to be URI...
    - mumblemumble 6y ago
      The only time I've ever encountered an API that used it extensively, it was done by a company that does all their implementation in Java. My best guess at what happened, based on the shape of the API, is that they implemented it by taking their pre-existing domain model, which had a fairly deep subclassing hierarchy, liberally sprinkled some annotations from com.fasterxml.jackson.annotation, and dumped the result straight onto the wire. You could absolutely do an object-oriented codebase where two different subclasses have fields with the same name and different types, and, depending on how you structure your code, it might not be too painful. And it's fairly easy to imagine someone serializing a structure like that to JSON without ever meditating on the fact that JSON won't retain the all the type information. Ironically, the end result was an API that is nigh-impossible to consume from Java. I ended up writing a façade in Python. I've also seen little bits of this happen in the internal APIs at my current company. Also a Java shop, also a result of trying to directly connect an internal object model to the API. I've never seen it done in an API implemented in a dynamic or functional language.
      - pwinnski 6y ago
        I hadn't considered the one-way nature of dumping Java classes to JSON. Mainly because I work with classes that are serialized both to and from JSON, rather than having Jackson annotations added much later. I can see it now. Horrifying! I'm sorry for your troubles.
        mumblemumble 6y ago
        My take-away is this: Protocols should always be defined independently of any existing code. The code-first approach only works well when you're doing something self-contained. Which, an API, almost by definition, is not.
  - alyandon 6y ago
    Lovely - I can't wait to see how horrible something like that is to implement in a language like Go.
    - nullandvoid 6y ago
      interface{} to the rescue
    - meddlepal 6y ago
      I hope you enjoy working with interface{} _sigh_
    - CameronNemo 6y ago
      Sounds like a golang problem. Rust can easily handle it with union types from what I have seen.
    - dullgiulio 6y ago
      You have to implement UnmarshalJSON. Between each attempt to deserialize into a possible struct, be careful to return on errors that are not JSON serialisation errors (for example caused by reading from the underlying Reader, etc.) It's ugly and verbose but there is no need to use empty interface.
- q3k 6y ago
  It means sometimes you'll get: { foo: { bar: "baz" }} and other times you'll get { foo: "something else" } Good luck!
  - maerF0x0 6y ago
    Polymorphic JSON is such a PITA for strongly typed languages. var data map[string]interface{} //.. switch t := data["foo"].(type) { case string: case interface{} } imagine that for every key...
    - speedgoose 6y ago
      In my humble opinion, Golang is pretty verbose in a bad way when it comes to JSON. Rust and the crate serde_json are also strongly typed and it's a lot better.
      - q3k 6y ago
        How would you express deserializing these 'polymorphic JSON' objects using serde/Rust?
        speedgoose 6y ago
        Using a rust enum I guess.
        q3k 6y ago
        You're right, this does seem to work [1]. I wasn't aware that serde would attempt to deserialize multiple enum variants until something matches. [1] - https://serde.rs/enum-representations.html#untagged https://serde.rs/enum-representations.html#untagged
        masklinn 6y ago
        Yeah that's a rather common scheme out there so Serde does provide built-in support for this deserialisation. Probably better for deserialisation performances to use properly tagged enums, but if you don't have a choice Serde's got your back.
- mumblemumble 6y ago
  Yeah, pretty much exactly that. The only time I've had to deal with it in the wild was a terrible experience. As a consumer, you couldn't make any decisions with confidence without first making a careful study of the documentation. For every. single. decision.
sk5t 6y ago
I'm in vigorous agreement here--polymorphic JSON is far more difficult to deserialize safely, and every instance I've seen of this in the wild has been the output of careless or deeply ignorant makers.
kevincox 6y ago
Yeah, put a `type` tag in there and call it a day.
samprotas 6y ago
The number of comments here specifically upset with this part of the current design is a bit discouraging, but not necessarily surprising. Yes, many mainstream languages have near-zero support for Tagged/Discriminated Unions or Enums with Associated Data or Algebraic Data Types (pick your favorite name for the same concept). This is a limitation of those languages, which should not force a language-agnostic protocol to adopt the lowest common denominator of expressiveness. Consider the problem they're avoiding of mutually exclusive keys in a struct/object. What do you do if you receive more than one? Is that behavior undefined? If it is defined, how sure are you that the implementation your package manager installed for you doesn't just pick one key arbitrarily in the name of "developer friendliness" leading to security bugs? This seems like a much more bug-ridden problem to solve than having to write verbose type/switching golang/java. Implementing more verbose deserialization code in languages with no support for Tagged Unions seems like a small price to pay for making a protocol that leaves no room for undefined behavior. To be clear, _many_ statically typed languages have perfect support for this concept (Rust/Swift/Scala/Haskell, to name a few).
- GordonS 6y ago
  It's not the same as union types, but you can also often achieve polymorphic serialisation with any OO language, through the use of interfaces.
- parhamn 6y ago
  > To be clear, _many_ statically typed languages have perfect support for this concept (Rust/Swift/Scala/Haskell, to name a few). No they don't, at least in the way you're selling it. The "limitation" here is JSON which doesn't attach type information. You're going to have to implement some typing protocol on top of the JSON anyway which will face similar problems to the ones you raised (unless you do some trait based inference which could be ambiguous and dangerous). If they were Enums/Unions over a serialization protocol like protobuf, maybe your case makes sense. Even then, Im guessing a large % of the OAuth 3 requests will go through Java/Golang libraries, so on a practical level this is a bad idea too.
  - samprotas 6y ago
nyanpasu64 6y ago
YAML has !Tag syntax for expressing polymorphism. Shame that nobody uses YAML because the standard is so complex, and shame that AppVeyor uses a single-element key-value mapping instead of a tag for this purpose...

thosakwe 6y ago

Yeah... This is the same sort of thing you see with, say, ActivityPub, that makes it a massive pain, if not totally impossible to implement it in a statically-typed language.

oblio 6y ago

I'm not sure I get this, does the data type change depending on context? Is that what they mean?

golergka 6y ago
> XYZ’s protocol is not just based on JSON, but it’s based on a particular technique known as polymorphic JSON. In this, a single field could have a different data type in different circumstances. For example, the resources field can be an array when requesting a single access token, or it can be an object when requesting multiple named access tokens. The difference in type indicates a difference in processing from the server. Within the resources array, each element can be either an object, representing a rich resource request description, or a string, representing a resource handle (scope). This is horrible.
- pwinnski 6y ago
  This is definitely not a protocol dreamed up by, say, Java developers. Obviously it's doable in Java, but I'm hard-pressed to imagine a Java developer would think of such a thing.
  - tasogare 6y ago
    This is a few magnitude even more infuriating than a spec I have to work with at work, where some keys are required to be URI...
  - mumblemumble 6y ago
    The only time I've ever encountered an API that used it extensively, it was done by a company that does all their implementation in Java. My best guess at what happened, based on the shape of the API, is that they implemented it by taking their pre-existing domain model, which had a fairly deep subclassing hierarchy, liberally sprinkled some annotations from com.fasterxml.jackson.annotation, and dumped the result straight onto the wire. You could absolutely do an object-oriented codebase where two different subclasses have fields with the same name and different types, and, depending on how you structure your code, it might not be too painful. And it's fairly easy to imagine someone serializing a structure like that to JSON without ever meditating on the fact that JSON won't retain the all the type information. Ironically, the end result was an API that is nigh-impossible to consume from Java. I ended up writing a façade in Python. I've also seen little bits of this happen in the internal APIs at my current company. Also a Java shop, also a result of trying to directly connect an internal object model to the API. I've never seen it done in an API implemented in a dynamic or functional language.
    - pwinnski 6y ago
      I hadn't considered the one-way nature of dumping Java classes to JSON. Mainly because I work with classes that are serialized both to and from JSON, rather than having Jackson annotations added much later. I can see it now. Horrifying! I'm sorry for your troubles.
      - mumblemumble 6y ago
        My take-away is this: Protocols should always be defined independently of any existing code. The code-first approach only works well when you're doing something self-contained. Which, an API, almost by definition, is not.
- alyandon 6y ago
  Lovely - I can't wait to see how horrible something like that is to implement in a language like Go.
  - nullandvoid 6y ago
    interface{} to the rescue
  - meddlepal 6y ago
    I hope you enjoy working with interface{} _sigh_
  - CameronNemo 6y ago
    Sounds like a golang problem. Rust can easily handle it with union types from what I have seen.
  - dullgiulio 6y ago
    You have to implement UnmarshalJSON. Between each attempt to deserialize into a possible struct, be careful to return on errors that are not JSON serialisation errors (for example caused by reading from the underlying Reader, etc.) It's ugly and verbose but there is no need to use empty interface.
q3k 6y ago
It means sometimes you'll get: { foo: { bar: "baz" }} and other times you'll get { foo: "something else" } Good luck!
- maerF0x0 6y ago
  Polymorphic JSON is such a PITA for strongly typed languages. var data map[string]interface{} //.. switch t := data["foo"].(type) { case string: case interface{} } imagine that for every key...
  - speedgoose 6y ago
    In my humble opinion, Golang is pretty verbose in a bad way when it comes to JSON. Rust and the crate serde_json are also strongly typed and it's a lot better.
    - q3k 6y ago
      How would you express deserializing these 'polymorphic JSON' objects using serde/Rust?
      - speedgoose 6y ago
        Using a rust enum I guess.
        q3k 6y ago
        You're right, this does seem to work [1]. I wasn't aware that serde would attempt to deserialize multiple enum variants until something matches. [1] - https://serde.rs/enum-representations.html#untagged https://serde.rs/enum-representations.html#untagged
        masklinn 6y ago
        Yeah that's a rather common scheme out there so Serde does provide built-in support for this deserialisation. Probably better for deserialisation performances to use properly tagged enums, but if you don't have a choice Serde's got your back.
mumblemumble 6y ago
Yeah, pretty much exactly that. The only time I've had to deal with it in the wild was a terrible experience. As a consumer, you couldn't make any decisions with confidence without first making a careful study of the documentation. For every. single. decision.

sk5t 6y ago

I'm in vigorous agreement here--polymorphic JSON is far more difficult to deserialize safely, and every instance I've seen of this in the wild has been the output of careless or deeply ignorant makers.

kevincox 6y ago

Yeah, put a `type` tag in there and call it a day.

samprotas 6y ago

The number of comments here specifically upset with this part of the current design is a bit discouraging, but not necessarily surprising. Yes, many mainstream languages have near-zero support for Tagged/Discriminated Unions or Enums with Associated Data or Algebraic Data Types (pick your favorite name for the same concept). This is a limitation of those languages, which should not force a language-agnostic protocol to adopt the lowest common denominator of expressiveness. Consider the problem they're avoiding of mutually exclusive keys in a struct/object. What do you do if you receive more than one? Is that behavior undefined? If it is defined, how sure are you that the implementation your package manager installed for you doesn't just pick one key arbitrarily in the name of "developer friendliness" leading to security bugs? This seems like a much more bug-ridden problem to solve than having to write verbose type/switching golang/java. Implementing more verbose deserialization code in languages with no support for Tagged Unions seems like a small price to pay for making a protocol that leaves no room for undefined behavior. To be clear, _many_ statically typed languages have perfect support for this concept (Rust/Swift/Scala/Haskell, to name a few).

GordonS 6y ago
It's not the same as union types, but you can also often achieve polymorphic serialisation with any OO language, through the use of interfaces.
parhamn 6y ago
> To be clear, _many_ statically typed languages have perfect support for this concept (Rust/Swift/Scala/Haskell, to name a few). No they don't, at least in the way you're selling it. The "limitation" here is JSON which doesn't attach type information. You're going to have to implement some typing protocol on top of the JSON anyway which will face similar problems to the ones you raised (unless you do some trait based inference which could be ambiguous and dangerous). If they were Enums/Unions over a serialization protocol like protobuf, maybe your case makes sense. Even then, Im guessing a large % of the OAuth 3 requests will go through Java/Golang libraries, so on a practical level this is a bad idea too.
- samprotas 6y ago

nyanpasu64 6y ago

YAML has !Tag syntax for expressing polymorphism. Shame that nobody uses YAML because the standard is so complex, and shame that AppVeyor uses a single-element key-value mapping instead of a tag for this purpose...