5 ms·

When you start having polyglot microservices this all will make sense. For example protobuf gives you versioning and compiles to a dozen languages...just becaus

by edem 9y ago

When you start having polyglot microservices this all will make sense. For example protobuf gives you versioning and compiles to a dozen languages...just because it is not useful to you don't dismiss the idea. And since I'm part of the minority I'm happy to have libs like this around. Just a sidenote: this behavior of neglecting performance alltogether leads to systems which take ages to build and collapse under pressure.

kybernetikos 9y ago
> When you start having polyglot microservices this all will make sense. I'll be surprised if you use a language that has a good protobuf implementation that doesn't also have a good json/yaml/xml implementation. Incidentally, as far as I can tell, protobuf also neglects performance. For high performance you want as much of your message to be predictably positioned in the bytestream as possible, but protobuf uses a key-value style system which means you always have to look at each key-value pair to determine what it is and how big it is. This makes good sense for compatibility, but it's a definite trade off that limits how fast deserialization can be.
- artursapek 9y ago
  If you use a type safe language, Protobuf gives you type safety. This helps a lot as a system grows complex. You don't get that with JSON/XML/etc.
  - JBReefer 9y ago
    Huh? All (?) of the XML parsers/unmarshallers I've ever used have been type safe - certainly the built in ones in C#, Java, and Go.
    - scottbez1 9y ago
      I think there's two distinct notions of type safety being discussed here: most deserializers in typed languages provide type safety once successfully deserialized, but the IDL/shared-schema approach helps guarantee that you'll be able to successfully deserialize the data into a type-safe representation in the first place. I can't speak to XML, but can say that even with "type safe" json serialization, I've seen several bugs due to some json libraries treating all numbers as doubles internally, meaning bad things happen when you're actually serializing integers larger than 52 bits (say, a nano timestamp). Sure, maybe it's a bug in the json library because the json spec doesn't establish any max precision for numbers, but it's not hard to run into it when different components/languages with independently implemented json libraries try to talk to each other. Shared schema approaches like proto also make it really hard to accidentally typo the name of the field, or accidentally try to read a field as a string rather than a list of strings. Just like I wouldn't consider code to be type safe if it passed data around as raw Objects and each usage cast it to the expected type, I wouldn't consider services that communicate with each other using independently (manually) implemented parsers/extractors as being type safe, even if the "meat" of the code is dealing with strongly typed data on either end.
      - rsynnott 9y ago
        Another numeric oddity; the spec says that ints should have from 1 to 9 digits (in theory limiting it to 35 bit numbers or so)! Few libraries actually honour this, but it's really pot luck what they'll do with anything greater than a 32 bit integer.
        scottbez1 9y ago
        Is that actually the case? The reference I see for 1-9 in the spec is not talking about length of the number, but is referring to digits 1,2,3...,9 as a single token in the grammar. digit1-9 is a token that's used to disallow leading zeros: a number must start (ignoring the possibility of "-" prefix) with a `digit1-9` which may then be followed by any number of `digit` tokens (inclusive of the digit "0" this time), or else it must start with a "0" which may only be followed by "." etc. I don't see any reference to range/precision in the spec, other than the suggestion that numbers that can be represented as an IEEE double are likely to be interoperable, and integers within 53 bits of range will generally be interoperable due to exact agreement on their value.
  - eberkund 9y ago
    In Golang you can have type safety with simple JSON marshaling between structures.
    - cookiecaper 9y ago
      It's been a while since I looked at it but back when I was trying to map a JSON wire format onto a Go structure, this was a PITA, as it typically is in strongly-typed languages. You say it's "simple". What's the best way to go about it?
      - placeybordeaux 9y ago
        I've found it very easy just using the Marshal and Unmarshal packages for golang: https://golang.org/pkg/encoding/json/#Unmarshal https://golang.org/pkg/encoding/json/#Unmarshal They have some reasonable examples there.
      - skybrian 9y ago
        The idea is that if you're inventing a new data format, you can write the Go types first, and the JSON schema is based on that. This works ok if you're starting in Go. Mapping arbitrary JSON to Go types may or may not be simple, depending on the data format.
      - eberkund 9y ago
        The other commenter posted a link to the official docs which explain it pretty well. I'm new to Go so I don't know what was there before but basically all I have to do is define a Go struct with the types that I want and then using struct field tags I can name the fields (for example if I want to use snakecase or something for JSON). Then when I encode to a string it automatically takes care of formatting it as JSON and I can decode a string and get back the typed struct (it will throw an error if it can't be done).
- lmm 9y ago
  json/yaml/xml have much less support for a schema-first workflow. XML offers it but only in a terrible schema language; JSON/YAML don't offer it at all as far as I know, and have difficulty expressing sum types, never mind anything more complicated.
- cookiecaper 9y ago
  The serialization format is an implementation detail. That's why there are similar schema/definition formats that ARE natively represented in JSON, like JSON-Schema. Personally I think it's easier to maintain a format defined in an IDL like Protobuf or Thrift than a JSON format. Everyone is always on about how JSON is a "human-readable" format, which is nice and probably 90% true. What we want is a human-writable format, which neither JSON nor YAML really are. It's easier to define a structure in Thrift than in JSON. I still gotta review this release of Cap'n Proto, but after struggling with some buggy language implementations from the upstream Thrift project and evaluating Protobuf 3.0 + grpc, I think that Protobuf 3.0 is going to be the way forward for now.
simias 9y ago
I work on such a system (multiple "microservices" written in a variety of languages) and we use JSON on top of zeromq. It's super easy to debug without any fancy tool. And as long as I'm consistent with my API design it's trivial to extend and keep everybody up to date. If performance ever becomes an issue (unlikely in my case) I could probably switch to some binary serialization protocol eventually.

by edem 9y ago

kybernetikos 9y ago

> When you start having polyglot microservices this all will make sense. I'll be surprised if you use a language that has a good protobuf implementation that doesn't also have a good json/yaml/xml implementation. Incidentally, as far as I can tell, protobuf also neglects performance. For high performance you want as much of your message to be predictably positioned in the bytestream as possible, but protobuf uses a key-value style system which means you always have to look at each key-value pair to determine what it is and how big it is. This makes good sense for compatibility, but it's a definite trade off that limits how fast deserialization can be.

artursapek 9y ago
If you use a type safe language, Protobuf gives you type safety. This helps a lot as a system grows complex. You don't get that with JSON/XML/etc.
- JBReefer 9y ago
  Huh? All (?) of the XML parsers/unmarshallers I've ever used have been type safe - certainly the built in ones in C#, Java, and Go.
  - scottbez1 9y ago
    I think there's two distinct notions of type safety being discussed here: most deserializers in typed languages provide type safety once successfully deserialized, but the IDL/shared-schema approach helps guarantee that you'll be able to successfully deserialize the data into a type-safe representation in the first place. I can't speak to XML, but can say that even with "type safe" json serialization, I've seen several bugs due to some json libraries treating all numbers as doubles internally, meaning bad things happen when you're actually serializing integers larger than 52 bits (say, a nano timestamp). Sure, maybe it's a bug in the json library because the json spec doesn't establish any max precision for numbers, but it's not hard to run into it when different components/languages with independently implemented json libraries try to talk to each other. Shared schema approaches like proto also make it really hard to accidentally typo the name of the field, or accidentally try to read a field as a string rather than a list of strings. Just like I wouldn't consider code to be type safe if it passed data around as raw Objects and each usage cast it to the expected type, I wouldn't consider services that communicate with each other using independently (manually) implemented parsers/extractors as being type safe, even if the "meat" of the code is dealing with strongly typed data on either end.
    - rsynnott 9y ago
      Another numeric oddity; the spec says that ints should have from 1 to 9 digits (in theory limiting it to 35 bit numbers or so)! Few libraries actually honour this, but it's really pot luck what they'll do with anything greater than a 32 bit integer.
      - scottbez1 9y ago
        Is that actually the case? The reference I see for 1-9 in the spec is not talking about length of the number, but is referring to digits 1,2,3...,9 as a single token in the grammar. digit1-9 is a token that's used to disallow leading zeros: a number must start (ignoring the possibility of "-" prefix) with a `digit1-9` which may then be followed by any number of `digit` tokens (inclusive of the digit "0" this time), or else it must start with a "0" which may only be followed by "." etc. I don't see any reference to range/precision in the spec, other than the suggestion that numbers that can be represented as an IEEE double are likely to be interoperable, and integers within 53 bits of range will generally be interoperable due to exact agreement on their value.
- eberkund 9y ago
  In Golang you can have type safety with simple JSON marshaling between structures.
  - cookiecaper 9y ago
    It's been a while since I looked at it but back when I was trying to map a JSON wire format onto a Go structure, this was a PITA, as it typically is in strongly-typed languages. You say it's "simple". What's the best way to go about it?
    - placeybordeaux 9y ago
      I've found it very easy just using the Marshal and Unmarshal packages for golang: https://golang.org/pkg/encoding/json/#Unmarshal https://golang.org/pkg/encoding/json/#Unmarshal They have some reasonable examples there.
    - skybrian 9y ago
      The idea is that if you're inventing a new data format, you can write the Go types first, and the JSON schema is based on that. This works ok if you're starting in Go. Mapping arbitrary JSON to Go types may or may not be simple, depending on the data format.
    - eberkund 9y ago
      The other commenter posted a link to the official docs which explain it pretty well. I'm new to Go so I don't know what was there before but basically all I have to do is define a Go struct with the types that I want and then using struct field tags I can name the fields (for example if I want to use snakecase or something for JSON). Then when I encode to a string it automatically takes care of formatting it as JSON and I can decode a string and get back the typed struct (it will throw an error if it can't be done).
lmm 9y ago
json/yaml/xml have much less support for a schema-first workflow. XML offers it but only in a terrible schema language; JSON/YAML don't offer it at all as far as I know, and have difficulty expressing sum types, never mind anything more complicated.
cookiecaper 9y ago
The serialization format is an implementation detail. That's why there are similar schema/definition formats that ARE natively represented in JSON, like JSON-Schema. Personally I think it's easier to maintain a format defined in an IDL like Protobuf or Thrift than a JSON format. Everyone is always on about how JSON is a "human-readable" format, which is nice and probably 90% true. What we want is a human-writable format, which neither JSON nor YAML really are. It's easier to define a structure in Thrift than in JSON. I still gotta review this release of Cap'n Proto, but after struggling with some buggy language implementations from the upstream Thrift project and evaluating Protobuf 3.0 + grpc, I think that Protobuf 3.0 is going to be the way forward for now.

simias 9y ago

I work on such a system (multiple "microservices" written in a variety of languages) and we use JSON on top of zeromq. It's super easy to debug without any fancy tool. And as long as I'm consistent with my API design it's trivial to extend and keep everybody up to date. If performance ever becomes an issue (unlikely in my case) I could probably switch to some binary serialization protocol eventually.