4 ms·

XML's bad, JSON is better. But Protocol Buffers are best. Like JSON, the data model for Protocol Buffers maps nicely onto simple data structures (unlike DOM).

XML's bad, JSON is better. But Protocol Buffers are best. Like JSON, the data model for Protocol Buffers maps nicely onto simple data structures (unlike DOM). But with Protocol Buffers you also get a schema for free, a wicked efficient binary format if you want it, default values, etc. And you can use JSON as a text format. In other words, you could take your JSON that you have sitting around, whip up a Protocol Buffers .proto file for it, and get nice generated C++ classes for it with full schema validation. Your JSON file: {field1="foo", field2=5} ...could be accessed from a C++ object as: my_obj.field1(); The only bummer about protocol buffers is that its support for high-level languages (PHP, Perl, Ruby, etc) is not very good. I've been working on a separate implementation of Protocol Buffers in C to address this (by making it easy to write bindings for) but this project has unfortunately been stalled as I've been busy with work and life. :(

alavrik 16y ago
What is it the tool that generates C++ codecs for JSON from .proto specifications? (I wrote a generic command-line Protobuf-JSON converter).
- haberman 16y ago
  Hmm, I just looked and apparently this code isn't part of the open-source protobuf release. :( This code exists inside Google, but apparently hasn't been disentangled from the rest of the Google source tree to be released outside.
prodigal_erik 16y ago
The protobuf encoding carries a ton of redundant crap around (if not as much as text formats). You can do a lot better by paying more attention to the schema and omitting facts that aren't allowed to change, like tags on self-delimiting required fields and arrays (ASN.1 PER does this well). The protobuf tradeoffs only make any sense if you think you can somehow do something useful with messages that are somewhat but not entirely corrupted, because you still have to solve the problem of finding intact field boundaries without being given any HDLC-style framing.
- haberman 16y ago
  > The protobuf encoding carries a ton of redundant crap around (if not as much as text formats). [citation needed] One of the features of protocol buffers is that they're backwards and forwards compatible. You can add and remove fields, change "required" to "optional" and back again, and still make sense of what comes to you on the wire. I don't think there's much of anything that can be eliminated from the protobuf binary format. Eliminating tags for a field (even if you consider it required) wouldn't be backward compatible with a previous version of the protocol that considered it optional. A message from a previous version of the schema is not "corrupted." Being able to make wire-compatible changes to the protocol is an extremely important feature.
  - prodigal_erik 16y ago
    You can't add or remove required fields, because outdated recipients will be dangerously wrong about whether they understand the intent of a message in the revised format. At some point meeting new requirements and keeping interoperability calls for deprecating and replacing the old format, because treating everything as optional (which some Googlers apparently do, avoiding "required" completely) is almost as bad as having 2^n mostly-untested formats.
    - haberman 16y ago
      This is the kind of over-engineering analysis that leads to overly complicated systems like XML schema. Just because a field says "optional" doesn't mean it's logically optional. You don't have to make your schema formalism complex enough that it can describe every last rule of what it takes for a message to be valid. In fact you definitely don't want to do that, because it's a horrible amount of complexity in the schema for little gain. Yes, it's true that some Googlers use "optional" instead of "required" everywhere in their .proto files. That doesn't mean that you can omit any field and expect your message to be processed by your peer without error. It just means that you won't get an error at the schema validation level. But the application could still throw an error. More complex rules about what fields must be specified or what values they must have can be described in comments, and enforced with custom validation if necessary. Also, since protobufs support default values, you can define what value will be returned for scalar fields if no value is explicitly sent. This can often be used to define useful default behavior for the case where a field is omitted.

alavrik 16y ago

What is it the tool that generates C++ codecs for JSON from .proto specifications? (I wrote a generic command-line Protobuf-JSON converter).

haberman 16y ago
Hmm, I just looked and apparently this code isn't part of the open-source protobuf release. :( This code exists inside Google, but apparently hasn't been disentangled from the rest of the Google source tree to be released outside.

prodigal_erik 16y ago

The protobuf encoding carries a ton of redundant crap around (if not as much as text formats). You can do a lot better by paying more attention to the schema and omitting facts that aren't allowed to change, like tags on self-delimiting required fields and arrays (ASN.1 PER does this well). The protobuf tradeoffs only make any sense if you think you can somehow do something useful with messages that are somewhat but not entirely corrupted, because you still have to solve the problem of finding intact field boundaries without being given any HDLC-style framing.

haberman 16y ago
> The protobuf encoding carries a ton of redundant crap around (if not as much as text formats). [citation needed] One of the features of protocol buffers is that they're backwards and forwards compatible. You can add and remove fields, change "required" to "optional" and back again, and still make sense of what comes to you on the wire. I don't think there's much of anything that can be eliminated from the protobuf binary format. Eliminating tags for a field (even if you consider it required) wouldn't be backward compatible with a previous version of the protocol that considered it optional. A message from a previous version of the schema is not "corrupted." Being able to make wire-compatible changes to the protocol is an extremely important feature.
- prodigal_erik 16y ago
  You can't add or remove required fields, because outdated recipients will be dangerously wrong about whether they understand the intent of a message in the revised format. At some point meeting new requirements and keeping interoperability calls for deprecating and replacing the old format, because treating everything as optional (which some Googlers apparently do, avoiding "required" completely) is almost as bad as having 2^n mostly-untested formats.
  - haberman 16y ago
    This is the kind of over-engineering analysis that leads to overly complicated systems like XML schema. Just because a field says "optional" doesn't mean it's logically optional. You don't have to make your schema formalism complex enough that it can describe every last rule of what it takes for a message to be valid. In fact you definitely don't want to do that, because it's a horrible amount of complexity in the schema for little gain. Yes, it's true that some Googlers use "optional" instead of "required" everywhere in their .proto files. That doesn't mean that you can omit any field and expect your message to be processed by your peer without error. It just means that you won't get an error at the schema validation level. But the application could still throw an error. More complex rules about what fields must be specified or what values they must have can be described in comments, and enforced with custom validation if necessary. Also, since protobufs support default values, you can define what value will be returned for scalar fields if no value is explicitly sent. This can often be used to define useful default behavior for the case where a field is omitted.