3 ms·

Making JSON first-class is an explicit design goal of proto3, the next version of Protocol Buffers currently in alpha: https://developers.google.com/protocol-bu

Making JSON first-class is an explicit design goal of proto3, the next version of Protocol Buffers currently in alpha: https://developers.google.com/protocol-buffers/docs/proto3#json https://developers.google.com/protocol-buffers/docs/proto3#j... This will allow you to switch between JSON and protobuf binary on the wire easily, while using official protobuf client libraries. So you can choose easily whether you care more about size/speed efficiency or wire readability. Best of both worlds! I work on the protobuf team at Google and would be happy to answer any questions.

zapov 11y ago
Do you plan on improving Protobuf speed in Java? People don't expect it to be slower than JSON ;) http://hperadin.github.io/jvm-serializers-report/report.html http://hperadin.github.io/jvm-serializers-report/report.html
- haberman 11y ago
  I'm not super familiar with the Java implementation, but I believe it's pretty optimized and appears to do very well generally on that benchmark. One unavoidable issue is that, unlike JSON, protobuf serializers have to do two passes over the message tree, because in protobuf binary format all submessages are prefixed by their length. The first pass just calculates lengths, while the second performs the actual serialization. This could potentially slow down serialization compared to JSON, especially for message trees with lots of nodes/depth.
  - abecedarius 11y ago
    So encode from back to front? Then when you reach the front you know the length.
    - deleted 11y ago
      [deleted]
    - haberman 11y ago
      Interesting idea, and could be interesting to experiment with. There are a lot of practical challenges though -- to provide a useful API you'd have to reverse the string at the end, since socket APIs don't generally provide streaming "WriteReverse" functions, and for good reason, because it would force them to buffer arbitrary amounts of data. So we'd have to reverse the entire string at the end. The question is whether this would be cheaper than doing a second pass over the message tree. And also keep in mind that you would need the first pass to decode everything -- including UTF-8 data -- in reverse. But since the UTF-8 APIs for Java strings probably don't support this, you'd probably have to encode it, then reverse it to put it in the encoding buffer. That way when it gets reversed again at the end, it would be proper UTF-8. At the end of the day, this probably wouldn't end up faster than what we do now. But can't say for sure without trying!
      - abecedarius 11y ago
        Reversing the output didn't come up when I did this sort of thing in C (for a different format): you can fill a buffer from the end, then just write the tail. But I guess Java does stick you with that? And I didn't have to deal with encoding to UTF-8 (ouch), or whatever other complications you might face, like streaming. Oh well! Hope the suggestion was stimulating anyway.
caust1c 11y ago
> Message field names are mapped to lowerCamelCase Why is a mapping to camel case necessary? I imagine it creates the potential for collisions, no?
- haberman 11y ago
  I think this may be to maintain style conventions with JavaScript and previous versions of Google Cloud APIs which were all based in lowerCamelCase (not 100% sure though, don't quote me on this). A benefit of this decision is that if you create JSON manually in JavaScript (ie. without a protobuf library) your JSON objects will match JavaScript conventions.
- iagooar 11y ago
  It's called "sadCamelCase", as opposed to "HappyCamelCase" ;)
- wora 11y ago
  Co-author of proto3 here. The reasons we chose lowerCamelCase are compatibility with Google's REST APIs (like Gmail API) and readability for users who work with JSON output directly without client library. API designers should not define confusing data schemas, let alone allow collisions. Most proto messages have small number of fields, avoiding name collision is a trivial for an API designer.
sapek 11y ago
JSON [1][2] (in fact text-based protocols in general) is also first-class in Bond (Microsoft's framework similar to ProtoBuf). [1] https://microsoft.github.io/bond/manual/bond_cpp.html#simple-json https://microsoft.github.io/bond/manual/bond_cpp.html#simple... [2] https://microsoft.github.io/bond/manual/bond_cs.html#json https://microsoft.github.io/bond/manual/bond_cs.html#json
teacup50 11y ago
I'm really bummed that you got rid of required fields in pb3. Now every consumer has to write additional code to verify that their required fields are actually available, and the proto spec is barely useful as an actual interpretable spec -- you have to specify requirements purely in comments. On top of which, you've defined built-in default values for empty fields; this means that, without warning, an accidentally missing field will inject bad data into any consumer that doesn't carefully check for the existence of all required fields. These are basically killer issues for us; we're not going to adopt an "update" that requires us to write JSON-style "hey, does this field exist?" code everywhere.
borplk 11y ago
When can we expect proto3 to be stable and ready to use for general public?

zapov 11y ago

Do you plan on improving Protobuf speed in Java? People don't expect it to be slower than JSON ;) http://hperadin.github.io/jvm-serializers-report/report.html http://hperadin.github.io/jvm-serializers-report/report.html

haberman 11y ago
I'm not super familiar with the Java implementation, but I believe it's pretty optimized and appears to do very well generally on that benchmark. One unavoidable issue is that, unlike JSON, protobuf serializers have to do two passes over the message tree, because in protobuf binary format all submessages are prefixed by their length. The first pass just calculates lengths, while the second performs the actual serialization. This could potentially slow down serialization compared to JSON, especially for message trees with lots of nodes/depth.
- abecedarius 11y ago
  So encode from back to front? Then when you reach the front you know the length.
  - deleted 11y ago
    [deleted]
  - haberman 11y ago
    Interesting idea, and could be interesting to experiment with. There are a lot of practical challenges though -- to provide a useful API you'd have to reverse the string at the end, since socket APIs don't generally provide streaming "WriteReverse" functions, and for good reason, because it would force them to buffer arbitrary amounts of data. So we'd have to reverse the entire string at the end. The question is whether this would be cheaper than doing a second pass over the message tree. And also keep in mind that you would need the first pass to decode everything -- including UTF-8 data -- in reverse. But since the UTF-8 APIs for Java strings probably don't support this, you'd probably have to encode it, then reverse it to put it in the encoding buffer. That way when it gets reversed again at the end, it would be proper UTF-8. At the end of the day, this probably wouldn't end up faster than what we do now. But can't say for sure without trying!
    - abecedarius 11y ago
      Reversing the output didn't come up when I did this sort of thing in C (for a different format): you can fill a buffer from the end, then just write the tail. But I guess Java does stick you with that? And I didn't have to deal with encoding to UTF-8 (ouch), or whatever other complications you might face, like streaming. Oh well! Hope the suggestion was stimulating anyway.

caust1c 11y ago

> Message field names are mapped to lowerCamelCase Why is a mapping to camel case necessary? I imagine it creates the potential for collisions, no?

haberman 11y ago
I think this may be to maintain style conventions with JavaScript and previous versions of Google Cloud APIs which were all based in lowerCamelCase (not 100% sure though, don't quote me on this). A benefit of this decision is that if you create JSON manually in JavaScript (ie. without a protobuf library) your JSON objects will match JavaScript conventions.
iagooar 11y ago
It's called "sadCamelCase", as opposed to "HappyCamelCase" ;)
wora 11y ago
Co-author of proto3 here. The reasons we chose lowerCamelCase are compatibility with Google's REST APIs (like Gmail API) and readability for users who work with JSON output directly without client library. API designers should not define confusing data schemas, let alone allow collisions. Most proto messages have small number of fields, avoiding name collision is a trivial for an API designer.

sapek 11y ago

JSON [1][2] (in fact text-based protocols in general) is also first-class in Bond (Microsoft's framework similar to ProtoBuf). [1] https://microsoft.github.io/bond/manual/bond_cpp.html#simple-json https://microsoft.github.io/bond/manual/bond_cpp.html#simple... [2] https://microsoft.github.io/bond/manual/bond_cs.html#json https://microsoft.github.io/bond/manual/bond_cs.html#json

teacup50 11y ago

I'm really bummed that you got rid of required fields in pb3. Now every consumer has to write additional code to verify that their required fields are actually available, and the proto spec is barely useful as an actual interpretable spec -- you have to specify requirements purely in comments. On top of which, you've defined built-in default values for empty fields; this means that, without warning, an accidentally missing field will inject bad data into any consumer that doesn't carefully check for the existence of all required fields. These are basically killer issues for us; we're not going to adopt an "update" that requires us to write JSON-style "hey, does this field exist?" code everywhere.

borplk 11y ago

When can we expect proto3 to be stable and ready to use for general public?