4 ms·

> When you start having polyglot microservices this all will make sense. I'll be surprised if you use a language that has a good protobuf implementation that d

> When you start having polyglot microservices this all will make sense. I'll be surprised if you use a language that has a good protobuf implementation that doesn't also have a good json/yaml/xml implementation. Incidentally, as far as I can tell, protobuf also neglects performance. For high performance you want as much of your message to be predictably positioned in the bytestream as possible, but protobuf uses a key-value style system which means you always have to look at each key-value pair to determine what it is and how big it is. This makes good sense for compatibility, but it's a definite trade off that limits how fast deserialization can be.

artursapek 10y ago
If you use a type safe language, Protobuf gives you type safety. This helps a lot as a system grows complex. You don't get that with JSON/XML/etc.
- JBReefer 10y ago
  Huh? All (?) of the XML parsers/unmarshallers I've ever used have been type safe - certainly the built in ones in C#, Java, and Go.
  - scottbez1 10y ago
    I think there's two distinct notions of type safety being discussed here: most deserializers in typed languages provide type safety once successfully deserialized, but the IDL/shared-schema approach helps guarantee that you'll be able to successfully deserialize the data into a type-safe representation in the first place. I can't speak to XML, but can say that even with "type safe" json serialization, I've seen several bugs due to some json libraries treating all numbers as doubles internally, meaning bad things happen when you're actually serializing integers larger than 52 bits (say, a nano timestamp). Sure, maybe it's a bug in the json library because the json spec doesn't establish any max precision for numbers, but it's not hard to run into it when different components/languages with independently implemented json libraries try to talk to each other. Shared schema approaches like proto also make it really hard to accidentally typo the name of the field, or accidentally try to read a field as a string rather than a list of strings. Just like I wouldn't consider code to be type safe if it passed data around as raw Objects and each usage cast it to the expected type, I wouldn't consider services that communicate with each other using independently (manually) implemented parsers/extractors as being type safe, even if the "meat" of the code is dealing with strongly typed data on either end.
    - rsynnott 10y ago
      Another numeric oddity; the spec says that ints should have from 1 to 9 digits (in theory limiting it to 35 bit numbers or so)! Few libraries actually honour this, but it's really pot luck what they'll do with anything greater than a 32 bit integer.
      - scottbez1 10y ago
        Is that actually the case? The reference I see for 1-9 in the spec is not talking about length of the number, but is referring to digits 1,2,3...,9 as a single token in the grammar. digit1-9 is a token that's used to disallow leading zeros: a number must start (ignoring the possibility of "-" prefix) with a `digit1-9` which may then be followed by any number of `digit` tokens (inclusive of the digit "0" this time), or else it must start with a "0" which may only be followed by "." etc. I don't see any reference to range/precision in the spec, other than the suggestion that numbers that can be represented as an IEEE double are likely to be interoperable, and integers within 53 bits of range will generally be interoperable due to exact agreement on their value.
- eberkund 10y ago
  In Golang you can have type safety with simple JSON marshaling between structures.
  - cookiecaper 10y ago
    It's been a while since I looked at it but back when I was trying to map a JSON wire format onto a Go structure, this was a PITA, as it typically is in strongly-typed languages. You say it's "simple". What's the best way to go about it?
    - placeybordeaux 10y ago
      I've found it very easy just using the Marshal and Unmarshal packages for golang: https://golang.org/pkg/encoding/json/#Unmarshal https://golang.org/pkg/encoding/json/#Unmarshal They have some reasonable examples there.
    - skybrian 10y ago
      The idea is that if you're inventing a new data format, you can write the Go types first, and the JSON schema is based on that. This works ok if you're starting in Go. Mapping arbitrary JSON to Go types may or may not be simple, depending on the data format.
    - eberkund 10y ago
      The other commenter posted a link to the official docs which explain it pretty well. I'm new to Go so I don't know what was there before but basically all I have to do is define a Go struct with the types that I want and then using struct field tags I can name the fields (for example if I want to use snakecase or something for JSON). Then when I encode to a string it automatically takes care of formatting it as JSON and I can decode a string and get back the typed struct (it will throw an error if it can't be done).
lmm 10y ago
json/yaml/xml have much less support for a schema-first workflow. XML offers it but only in a terrible schema language; JSON/YAML don't offer it at all as far as I know, and have difficulty expressing sum types, never mind anything more complicated.
cookiecaper 10y ago
The serialization format is an implementation detail. That's why there are similar schema/definition formats that ARE natively represented in JSON, like JSON-Schema. Personally I think it's easier to maintain a format defined in an IDL like Protobuf or Thrift than a JSON format. Everyone is always on about how JSON is a "human-readable" format, which is nice and probably 90% true. What we want is a human-writable format, which neither JSON nor YAML really are. It's easier to define a structure in Thrift than in JSON. I still gotta review this release of Cap'n Proto, but after struggling with some buggy language implementations from the upstream Thrift project and evaluating Protobuf 3.0 + grpc, I think that Protobuf 3.0 is going to be the way forward for now.

artursapek 10y ago

If you use a type safe language, Protobuf gives you type safety. This helps a lot as a system grows complex. You don't get that with JSON/XML/etc.

JBReefer 10y ago
Huh? All (?) of the XML parsers/unmarshallers I've ever used have been type safe - certainly the built in ones in C#, Java, and Go.
- scottbez1 10y ago
  I think there's two distinct notions of type safety being discussed here: most deserializers in typed languages provide type safety once successfully deserialized, but the IDL/shared-schema approach helps guarantee that you'll be able to successfully deserialize the data into a type-safe representation in the first place. I can't speak to XML, but can say that even with "type safe" json serialization, I've seen several bugs due to some json libraries treating all numbers as doubles internally, meaning bad things happen when you're actually serializing integers larger than 52 bits (say, a nano timestamp). Sure, maybe it's a bug in the json library because the json spec doesn't establish any max precision for numbers, but it's not hard to run into it when different components/languages with independently implemented json libraries try to talk to each other. Shared schema approaches like proto also make it really hard to accidentally typo the name of the field, or accidentally try to read a field as a string rather than a list of strings. Just like I wouldn't consider code to be type safe if it passed data around as raw Objects and each usage cast it to the expected type, I wouldn't consider services that communicate with each other using independently (manually) implemented parsers/extractors as being type safe, even if the "meat" of the code is dealing with strongly typed data on either end.
  - rsynnott 10y ago
    Another numeric oddity; the spec says that ints should have from 1 to 9 digits (in theory limiting it to 35 bit numbers or so)! Few libraries actually honour this, but it's really pot luck what they'll do with anything greater than a 32 bit integer.
    - scottbez1 10y ago
      Is that actually the case? The reference I see for 1-9 in the spec is not talking about length of the number, but is referring to digits 1,2,3...,9 as a single token in the grammar. digit1-9 is a token that's used to disallow leading zeros: a number must start (ignoring the possibility of "-" prefix) with a `digit1-9` which may then be followed by any number of `digit` tokens (inclusive of the digit "0" this time), or else it must start with a "0" which may only be followed by "." etc. I don't see any reference to range/precision in the spec, other than the suggestion that numbers that can be represented as an IEEE double are likely to be interoperable, and integers within 53 bits of range will generally be interoperable due to exact agreement on their value.
eberkund 10y ago
In Golang you can have type safety with simple JSON marshaling between structures.
- cookiecaper 10y ago
  It's been a while since I looked at it but back when I was trying to map a JSON wire format onto a Go structure, this was a PITA, as it typically is in strongly-typed languages. You say it's "simple". What's the best way to go about it?
  - placeybordeaux 10y ago
    I've found it very easy just using the Marshal and Unmarshal packages for golang: https://golang.org/pkg/encoding/json/#Unmarshal https://golang.org/pkg/encoding/json/#Unmarshal They have some reasonable examples there.
  - skybrian 10y ago
    The idea is that if you're inventing a new data format, you can write the Go types first, and the JSON schema is based on that. This works ok if you're starting in Go. Mapping arbitrary JSON to Go types may or may not be simple, depending on the data format.
  - eberkund 10y ago
    The other commenter posted a link to the official docs which explain it pretty well. I'm new to Go so I don't know what was there before but basically all I have to do is define a Go struct with the types that I want and then using struct field tags I can name the fields (for example if I want to use snakecase or something for JSON). Then when I encode to a string it automatically takes care of formatting it as JSON and I can decode a string and get back the typed struct (it will throw an error if it can't be done).

lmm 10y ago

json/yaml/xml have much less support for a schema-first workflow. XML offers it but only in a terrible schema language; JSON/YAML don't offer it at all as far as I know, and have difficulty expressing sum types, never mind anything more complicated.

cookiecaper 10y ago

The serialization format is an implementation detail. That's why there are similar schema/definition formats that ARE natively represented in JSON, like JSON-Schema. Personally I think it's easier to maintain a format defined in an IDL like Protobuf or Thrift than a JSON format. Everyone is always on about how JSON is a "human-readable" format, which is nice and probably 90% true. What we want is a human-writable format, which neither JSON nor YAML really are. It's easier to define a structure in Thrift than in JSON. I still gotta review this release of Cap'n Proto, but after struggling with some buggy language implementations from the upstream Thrift project and evaluating Protobuf 3.0 + grpc, I think that Protobuf 3.0 is going to be the way forward for now.