3 ms·

I wrote a protobuf decoder once and found it to be remarkably pleasant. Getting the decoder working only took a few hours. The format was obviously designed to

I wrote a protobuf decoder once and found it to be remarkably pleasant. Getting the decoder working only took a few hours. The format was obviously designed to be straightforward -- no escaping, no backtracking, no ambiguity. I believe the grammar is LL(0), which is a nice touch. And because it's not meant to be human-readable, there's no incentive for people to make their parsers deviate from the strict grammar; e.g. there's no protobuf quirk analogous to JSON's parser-dependent handling of trailing commas, because why would anyone bother?

iainmerrick 9y ago
There is no such quirk. Trailing commas are simply not standard JSON (sadly). Even if they were, it would still be easy to parse!
- pjscott 9y ago
  You're right in the same technically-correct sense that parsing HTML 4 is easy: it's just SGML! And as long as you don't have to support any of the crazy deviations from the spec that some people have come to rely on, that's fine. There's a whole spectrum of unofficial parser "helpfulness" here, with HTML 4 being an extreme case of parsers filled with hacks to deal with existing broken data, protobufs being an extreme case of parsers doing the One and Only True Thing, and JSON mostly toward the same end of the spectrum as protobufs, but a bit less so.
  - iainmerrick 9y ago
    Yeah, that spectrum is a good way to think about it. JSON hits a sweet spot of being very easy for computers to deal with almost all the time, while also being reasonably easy for humans to read and write. I was going to add “if you started with that as the spec, it wouldn’t be hard to design something better than JSON” but real examples like YAML are pretty awkward, so probably it’s a harder problem than it seems.
    - Mikhail_Edoshin 9y ago
      I think it's the other way round: JSON mimics the syntax of JavaScript literals and the syntax was meant to be easy for humans to write and read. Not super-easy, because it has to be easily parseable as well, but still the original use case of this notation is to write relatively short pieces of code. This is why it seems easy to people and they assume it's easy for computers as well, while in fact JSON is hard for computers to emit because of the trailing comma issue (not super-hard, but harder than XML, which has no separators: elements are self-contained).
      - iainmerrick 9y ago
        Yes, I really wish trailing commas were standard. But they’re not, so we have to resist the temptation to use them!
- Mikhail_Edoshin 9y ago
  It would be much easier to emit if they were standard. For example, in XML each element is self-contained; I can pour it into the data stream without knowing if it is preceded or followed by a sibling. With JSON I have to manage the context.
haberman 9y ago
I'm glad you had a good experience with Protocol Buffers. (I work on the protobuf team at Google). But I would advise a bit of caution. Writing a fully-compliant protobuf implementation is trickier than it looks. I'd recommend running any new parser through the conformance tests; I created these to exercise some of the edge cases. https://github.com/google/protobuf/tree/master/conformance https://github.com/google/protobuf/tree/master/conformance Here are some of the quirks of protobuf: - non-repeated fields can occur multiple times on the wire -- the last value "wins". - you have to be able to handle unknown fields, including unknown groups that can be nested arbitrarily. - repeated numbers have two different wire formats (packed and non-packed), you have be able to handle both. - when serializing, all signed integers need to be sign- extended to 64 bits, to support interop between different integer types. - you have to bounds-check delimited fields to make sure they don't violate the bounds of submessages you are already in. I do think protobuf is a great technology overall. But it has some complexities too; I wouldn't want to oversell its simplicity and have people be unpleasantly surprised when they come across them later. :)
- zeveb 9y ago
  > when serializing, all signed integers need to be sign-extended to 64 bits, to support interop between different integer types FWIW, I think it'd be awesome if the only wire integer format were a bignum, in order to support full interoperability between integer types. Maybe even do that same for floats, too …

iainmerrick 9y ago

There is no such quirk. Trailing commas are simply not standard JSON (sadly). Even if they were, it would still be easy to parse!

pjscott 9y ago
You're right in the same technically-correct sense that parsing HTML 4 is easy: it's just SGML! And as long as you don't have to support any of the crazy deviations from the spec that some people have come to rely on, that's fine. There's a whole spectrum of unofficial parser "helpfulness" here, with HTML 4 being an extreme case of parsers filled with hacks to deal with existing broken data, protobufs being an extreme case of parsers doing the One and Only True Thing, and JSON mostly toward the same end of the spectrum as protobufs, but a bit less so.
- iainmerrick 9y ago
  Yeah, that spectrum is a good way to think about it. JSON hits a sweet spot of being very easy for computers to deal with almost all the time, while also being reasonably easy for humans to read and write. I was going to add “if you started with that as the spec, it wouldn’t be hard to design something better than JSON” but real examples like YAML are pretty awkward, so probably it’s a harder problem than it seems.
  - Mikhail_Edoshin 9y ago
    I think it's the other way round: JSON mimics the syntax of JavaScript literals and the syntax was meant to be easy for humans to write and read. Not super-easy, because it has to be easily parseable as well, but still the original use case of this notation is to write relatively short pieces of code. This is why it seems easy to people and they assume it's easy for computers as well, while in fact JSON is hard for computers to emit because of the trailing comma issue (not super-hard, but harder than XML, which has no separators: elements are self-contained).
    - iainmerrick 9y ago
      Yes, I really wish trailing commas were standard. But they’re not, so we have to resist the temptation to use them!
Mikhail_Edoshin 9y ago
It would be much easier to emit if they were standard. For example, in XML each element is self-contained; I can pour it into the data stream without knowing if it is preceded or followed by a sibling. With JSON I have to manage the context.

haberman 9y ago

I'm glad you had a good experience with Protocol Buffers. (I work on the protobuf team at Google). But I would advise a bit of caution. Writing a fully-compliant protobuf implementation is trickier than it looks. I'd recommend running any new parser through the conformance tests; I created these to exercise some of the edge cases. https://github.com/google/protobuf/tree/master/conformance https://github.com/google/protobuf/tree/master/conformance Here are some of the quirks of protobuf: - non-repeated fields can occur multiple times on the wire -- the last value "wins". - you have to be able to handle unknown fields, including unknown groups that can be nested arbitrarily. - repeated numbers have two different wire formats (packed and non-packed), you have be able to handle both. - when serializing, all signed integers need to be sign- extended to 64 bits, to support interop between different integer types. - you have to bounds-check delimited fields to make sure they don't violate the bounds of submessages you are already in. I do think protobuf is a great technology overall. But it has some complexities too; I wouldn't want to oversell its simplicity and have people be unpleasantly surprised when they come across them later. :)

zeveb 9y ago
> when serializing, all signed integers need to be sign-extended to 64 bits, to support interop between different integer types FWIW, I think it'd be awesome if the only wire integer format were a bignum, in order to support full interoperability between integer types. Maybe even do that same for floats, too …