4 ms·

Not so sure about "backwards compatibility" part. From Protocol buffer python doc: https://developers.google.com/protocol-buffers/docs/pythontutorial https://d

by znt 12y ago

Not so sure about "backwards compatibility" part. From Protocol buffer python doc: https://developers.google.com/protocol-buffers/docs/pythontutorial https://developers.google.com/protocol-buffers/docs/pythontu... "Required Is Forever You should be very careful about marking fields as required. If at some point you wish to stop writing or sending a required field, it will be problematic to change the field to an optional field – old readers will consider messages without this field to be incomplete and may reject or drop them unintentionally. You should consider writing application-specific custom validation routines for your buffers instead. Some engineers at Google have come to the conclusion that using required does more harm than good; they prefer to use only optional and repeated. However, this view is not universal." So basically I will be in trouble if I decide to get rid of some fields which are not necessary, but somehow were defined as "required" in the past. This will potentially result in bloated protobuf definitions that have a bunch of legacy fields. I will stick to the JSON, thanks.

nostrademons 12y ago
In JSON all fields are optional and you're forced to write application-specific custom validation routines anyway, so your semantics are exactly as if you had marked them all "optional" in protobufs.
Locke1689 12y ago
Just make everything optional (like suggested) if you're worried about it.
isbadawi 12y ago
If you don't currently have protobuf definitions, you can just not use required fields in any new ones you write, like the quoted passage suggests.
- clubhi 12y ago
  You absolutely don't need to use "required". It's just an added feature. I use it quite a bit though.
jamesaguilar 12y ago
This is a non-criticism. Just use optional for everything, or make sure that all your clients and servers have upgraded. If you had a JSON library that supported required semantics, you'd have exactly the same issue on your hands.
yongjik 12y ago
If your protobufs (and their producer/consumers) are short-living, just remove the "required" field from your servers and clients at the same time. The "required is forever" problem persists only if you keep files with serialized protobuf data (or legacy binaries) and cannot reasonably update them at the same time. Of course, once something hits production, you really cannot reasonably update them all at the same time. That's why the proto API has: https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.message_lite#MessageLite.ParseFromString https://developers.google.com/protocol-buffers/docs/referenc... bool ParseFromString(const string & data) Parse a protocol buffer contained in a string. bool ParsePartialFromString(const string & data) Like ParseFromString(), but accepts messages that are missing required fields. Or simply use "optional" instead as others suggested. Also, even with these "bloated" definitions with legacy unused fields, it's probably still smaller than JSON.
- sergiosgc 12y ago
  Come on! That's sidestepping the issue. If you can change all clients and servers at the same time, there's no need for protocol versioning...
gohrt 12y ago
That argument never made sense: " old readers will consider messages without this field to be incomplete and may reject or drop them unintentionally." That means the old readers -- the ones that are expecting required fields, can't accept new messages. That's good! The readers don't know how to read the new messages! The readers need to be updated to a new version before they can correctly start reading new version of the schema.
- kentonv 12y ago
  Yeah, the problem is that because "required" is checked at the protobuf parser layer, it will be enforced whether or not the application itself actually depends on that field being present. In practice this tends to escalate minor bugs into full outages. Like, there have actually been outages in Google Search, Gmail, and others that wouldn't have happened if all required fields were treated as optional instead. I've written more about this here: https://kentonv.github.io/capnproto/faq.html#how_do_i_make_a_field_required_like_in_protocol_buffers https://kentonv.github.io/capnproto/faq.html#how_do_i_make_a...

by znt 12y ago

nostrademons 12y ago

In JSON all fields are optional and you're forced to write application-specific custom validation routines anyway, so your semantics are exactly as if you had marked them all "optional" in protobufs.

Locke1689 12y ago

Just make everything optional (like suggested) if you're worried about it.

isbadawi 12y ago

If you don't currently have protobuf definitions, you can just not use required fields in any new ones you write, like the quoted passage suggests.

clubhi 12y ago
You absolutely don't need to use "required". It's just an added feature. I use it quite a bit though.

jamesaguilar 12y ago

This is a non-criticism. Just use optional for everything, or make sure that all your clients and servers have upgraded. If you had a JSON library that supported required semantics, you'd have exactly the same issue on your hands.

yongjik 12y ago

If your protobufs (and their producer/consumers) are short-living, just remove the "required" field from your servers and clients at the same time. The "required is forever" problem persists only if you keep files with serialized protobuf data (or legacy binaries) and cannot reasonably update them at the same time. Of course, once something hits production, you really cannot reasonably update them all at the same time. That's why the proto API has: https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.message_lite#MessageLite.ParseFromString https://developers.google.com/protocol-buffers/docs/referenc... bool ParseFromString(const string & data) Parse a protocol buffer contained in a string. bool ParsePartialFromString(const string & data) Like ParseFromString(), but accepts messages that are missing required fields. Or simply use "optional" instead as others suggested. Also, even with these "bloated" definitions with legacy unused fields, it's probably still smaller than JSON.

sergiosgc 12y ago
Come on! That's sidestepping the issue. If you can change all clients and servers at the same time, there's no need for protocol versioning...

gohrt 12y ago

That argument never made sense: " old readers will consider messages without this field to be incomplete and may reject or drop them unintentionally." That means the old readers -- the ones that are expecting required fields, can't accept new messages. That's good! The readers don't know how to read the new messages! The readers need to be updated to a new version before they can correctly start reading new version of the schema.

kentonv 12y ago
Yeah, the problem is that because "required" is checked at the protobuf parser layer, it will be enforced whether or not the application itself actually depends on that field being present. In practice this tends to escalate minor bugs into full outages. Like, there have actually been outages in Google Search, Gmail, and others that wouldn't have happened if all required fields were treated as optional instead. I've written more about this here: https://kentonv.github.io/capnproto/faq.html#how_do_i_make_a_field_required_like_in_protocol_buffers https://kentonv.github.io/capnproto/faq.html#how_do_i_make_a...