5 ms·

You Are Doing JSON APIs Wrong (“foo”:{}, not “type”: “foo”)

armchairhacker 5y ago
this is how I usually write JSON. { “type”: foo, … } is more verbose and can lead to issues if you forget to check the type, and moreover it makes it easier to generate some complex types.
nly 5y ago
> The type of every JSON object field should be uniquely determined by its field name. Fields of variant/oneof/union type are really useful, so I don't agree with this. The problem regarding JSON serialization of type-tagged fields and streaming parsers is very well observed though. I suppose one nice thing about adding a sibling "type" key, is it allows you to have the option to make a non-variant field a variant later on (by allowing the type field to have a default if it's missing).
- FeepingCreature 5y ago
  You can do union types with the same mechanism: `{"foo": FooTypedValue}` or `{"bar": BarTypedValue}`. No variant, admittedly, but there's no making open-ended JSON objects fast no matter what you do. The problematic thing with `"type"` is that it forces the parser to treat your entire message as variant.
  - nly 5y ago
    That's not a variant, it's just 2 optional fields: one called "foo" and one called "bar". A validating parser then has to know that they are semantically linked. Admittedly this is what protocol buffers does in the binary wire format, because oneof's are just a frontend hack (iirc the latest value decoded gets kept), but it's not the same.
hyperhello 5y ago
It sounds like the point is to type members before their data in the stream. But that can’t be a huge drain on serializers. JSON isn’t supposed to be incredibly performant, it’s supposed to be human readable. And with multi core I would expect serializers to read ahead anyway. There was a thing called Reverse Hungarian that put an abbreviation for the type in the variable name. That had some value to work on very small screens without much scrolling.
- FeepingCreature 5y ago
  It's even more human-readable this way though (IMO), because it saves you from having to pull a type field from somewhere of a soup of a log message. With the key-identified format, the type is guaranteed to be directly before the data.
eatonphil 5y ago
Pretty sure TypeScript discriminated unions don't work unless you have a type field you can switch on.
- hyuuu 5y ago
  why cant you have: type Message { contentA?: ContentAStructure, contentB?: ContentBStructure, } on read, you are already accessing the property data as typed right?
  - matt_kantor 5y ago
    That's not a union. Both `{}` and `{ contentA: a, contentB: b }` inhabit that type, so it doesn't represent "a value of type `Message` is either a value of `ContentAStructure` or a value of `ContentBStructure`".
  - eatonphil 5y ago
    You can but that's not type safe in TypeScript.
stickfigure 5y ago
This is premature optimization. A very small percentage of APIs return so much data that you need to worry about stream parsing. The vast majority of APIs in the world, including probably yours, returns structures that are easy to manage in memory. If you render this `{"MyMessageType":{...}}` structure, people who live in statically typed languages will hate you. At best you've created a union type with a truly vast number of possible cases. In practice you've forced this to be read as `Map<String, JsonNode>` and dynamically interpreted. Unless your problem domain routinely streams gigabytes of data across an API, don't do this. And if it does stream gigabytes of data across an API, figure out how to chunk the data and still don't do this.
- Zababa 5y ago
  How is this premature optimization? Since it's equivalent to the alternative, it should be known that this way has an advantage.
  - stickfigure 5y ago
    Adding a ton of structural noise is not "equivalent". This pattern will be repeated throughout your hierarchy, and your clients will have to deal with this: GET /assets/123 { "Asset": { "owner": { "Person": { "name": "Bob", "contact": { "Address": { "street1": "blah", ... } } } } } } On the other hand, the traditional approach: GET /assets/123 { "owner": { "type": "Person", "name": "Bob", "contact": { "type": "Address", "street1": "blah", ... } } } } This is much more sane to navigate. And when you end up making some previously-static node polymorphic (as happens), the structure does not change.
    - FeepingCreature 5y ago
      I think making a previously-static node polymorphic in a way that doesn't drop any required field is actually quite rare. But also, I just think the first one is better.
henrydark 5y ago
I came in knowing I'd be against it, and came out convinced. The argument is just solid: it's easier to make things correct as well as faster this way.
__ryan__ 5y ago
While it's easier to use discriminated unions with some "type" field (easier to use switch/case) here's the reality: 1. There's many ways for the compiler to narrow a union type [0] 2. There's a better solution than depending on "easy" narrowing for excessively large union types: type guards, callback registry, etc. [0]: https://www.typescriptlang.org/play?#code/JYOwLgpgTgZghgYwgAgILIN4ChnLgWQgGci4BzCALmRAFcBbAI2gG4sBfLLUSWRFAEKYcyRoRLkqyImCigybTt3DR4SZAGFhuBONIVqjAPZGANhDghFXMAE8ADij2TkAXjTIAPsiHeNbLBhaEAQwYCMQZEgZAAp6Yn0pZwoASm1kYBhkGIAiAgTJHIzI+IlU9J0ImWQANzhTWik6Jmg3ZFLEgDp8sog2XEqQIjMITtMjMlyexOocgBpa+saU-uR2ZAhTIhRM7JyxAooi0HbDiDTsAeQEKrBFhqkZORAyNo7JToPe1cHh8zGJrkvjNkPN7stVutNtsKtdbuCpMYRpY3mdOrozj84UMRgDJjkMb1Zgs6g8ViJOEogA https://www.typescriptlang.org/play?#code/JYOwLgpgTgZghgYwgA...
jurschreuder 5y ago
I may be weird but I do it like this: { "type": "a", "version": "1", "payloadJson": "{\"a\": 10}" }
- __ryan__ 5y ago
  Why?
- Otek 5y ago
  That is kinda weird, why would you store json in a json as a string?
- arinlen 5y ago
  > { "type": "a", "version": "1", "payloadJson": "{\"a\": 10}" } What's wrong with REST and leaving stuff like versioning and resource type to HATEOAS?
- selfhoster11 5y ago
  It is weird. There aren't many reasons that I can think of for encoding things this way.
- remram 5y ago
  It sounds like you are reinventing HTTP headers and trying to put them in the body for whatever reason. I've also seen this before, but I can't say I didn't laugh: HTTP/1.1 200 Ok Content-type: application/json {"status": 404, "type": "xml", "body": "<entry><name>...</name><author>...</author></entry>"}
benibela 5y ago
I am doing ["foo", {} ]

this is how I usually write JSON. { “type”: foo, … } is more verbose and can lead to issues if you forget to check the type, and moreover it makes it easier to generate some complex types.

nly 5y ago

> The type of every JSON object field should be uniquely determined by its field name. Fields of variant/oneof/union type are really useful, so I don't agree with this. The problem regarding JSON serialization of type-tagged fields and streaming parsers is very well observed though. I suppose one nice thing about adding a sibling "type" key, is it allows you to have the option to make a non-variant field a variant later on (by allowing the type field to have a default if it's missing).

FeepingCreature 5y ago
You can do union types with the same mechanism: `{"foo": FooTypedValue}` or `{"bar": BarTypedValue}`. No variant, admittedly, but there's no making open-ended JSON objects fast no matter what you do. The problematic thing with `"type"` is that it forces the parser to treat your entire message as variant.
- nly 5y ago
  That's not a variant, it's just 2 optional fields: one called "foo" and one called "bar". A validating parser then has to know that they are semantically linked. Admittedly this is what protocol buffers does in the binary wire format, because oneof's are just a frontend hack (iirc the latest value decoded gets kept), but it's not the same.

hyperhello 5y ago

It sounds like the point is to type members before their data in the stream. But that can’t be a huge drain on serializers. JSON isn’t supposed to be incredibly performant, it’s supposed to be human readable. And with multi core I would expect serializers to read ahead anyway. There was a thing called Reverse Hungarian that put an abbreviation for the type in the variable name. That had some value to work on very small screens without much scrolling.

FeepingCreature 5y ago
It's even more human-readable this way though (IMO), because it saves you from having to pull a type field from somewhere of a soup of a log message. With the key-identified format, the type is guaranteed to be directly before the data.

eatonphil 5y ago

Pretty sure TypeScript discriminated unions don't work unless you have a type field you can switch on.

hyuuu 5y ago
why cant you have: type Message { contentA?: ContentAStructure, contentB?: ContentBStructure, } on read, you are already accessing the property data as typed right?
- matt_kantor 5y ago
  That's not a union. Both `{}` and `{ contentA: a, contentB: b }` inhabit that type, so it doesn't represent "a value of type `Message` is either a value of `ContentAStructure` or a value of `ContentBStructure`".
- eatonphil 5y ago
  You can but that's not type safe in TypeScript.

stickfigure 5y ago

This is premature optimization. A very small percentage of APIs return so much data that you need to worry about stream parsing. The vast majority of APIs in the world, including probably yours, returns structures that are easy to manage in memory. If you render this `{"MyMessageType":{...}}` structure, people who live in statically typed languages will hate you. At best you've created a union type with a truly vast number of possible cases. In practice you've forced this to be read as `Map<String, JsonNode>` and dynamically interpreted. Unless your problem domain routinely streams gigabytes of data across an API, don't do this. And if it does stream gigabytes of data across an API, figure out how to chunk the data and still don't do this.

Zababa 5y ago
How is this premature optimization? Since it's equivalent to the alternative, it should be known that this way has an advantage.
- stickfigure 5y ago
  Adding a ton of structural noise is not "equivalent". This pattern will be repeated throughout your hierarchy, and your clients will have to deal with this: GET /assets/123 { "Asset": { "owner": { "Person": { "name": "Bob", "contact": { "Address": { "street1": "blah", ... } } } } } } On the other hand, the traditional approach: GET /assets/123 { "owner": { "type": "Person", "name": "Bob", "contact": { "type": "Address", "street1": "blah", ... } } } } This is much more sane to navigate. And when you end up making some previously-static node polymorphic (as happens), the structure does not change.
  - FeepingCreature 5y ago
    I think making a previously-static node polymorphic in a way that doesn't drop any required field is actually quite rare. But also, I just think the first one is better.

henrydark 5y ago

I came in knowing I'd be against it, and came out convinced. The argument is just solid: it's easier to make things correct as well as faster this way.

__ryan__ 5y ago

While it's easier to use discriminated unions with some "type" field (easier to use switch/case) here's the reality: 1. There's many ways for the compiler to narrow a union type [0] 2. There's a better solution than depending on "easy" narrowing for excessively large union types: type guards, callback registry, etc. [0]: https://www.typescriptlang.org/play?#code/JYOwLgpgTgZghgYwgAgILIN4ChnLgWQgGci4BzCALmRAFcBbAI2gG4sBfLLUSWRFAEKYcyRoRLkqyImCigybTt3DR4SZAGFhuBONIVqjAPZGANhDghFXMAE8ADij2TkAXjTIAPsiHeNbLBhaEAQwYCMQZEgZAAp6Yn0pZwoASm1kYBhkGIAiAgTJHIzI+IlU9J0ImWQANzhTWik6Jmg3ZFLEgDp8sog2XEqQIjMITtMjMlyexOocgBpa+saU-uR2ZAhTIhRM7JyxAooi0HbDiDTsAeQEKrBFhqkZORAyNo7JToPe1cHh8zGJrkvjNkPN7stVutNtsKtdbuCpMYRpY3mdOrozj84UMRgDJjkMb1Zgs6g8ViJOEogA https://www.typescriptlang.org/play?#code/JYOwLgpgTgZghgYwgA...

jurschreuder 5y ago

I may be weird but I do it like this: { "type": "a", "version": "1", "payloadJson": "{\"a\": 10}" }

__ryan__ 5y ago
Why?
Otek 5y ago
That is kinda weird, why would you store json in a json as a string?
arinlen 5y ago
> { "type": "a", "version": "1", "payloadJson": "{\"a\": 10}" } What's wrong with REST and leaving stuff like versioning and resource type to HATEOAS?
selfhoster11 5y ago
It is weird. There aren't many reasons that I can think of for encoding things this way.
remram 5y ago
It sounds like you are reinventing HTTP headers and trying to put them in the body for whatever reason. I've also seen this before, but I can't say I didn't laugh: HTTP/1.1 200 Ok Content-type: application/json {"status": 404, "type": "xml", "body": "<entry><name>...</name><author>...</author></entry>"}

benibela 5y ago

I am doing ["foo", {} ]