5 ms·

> every application has to be updated when a field is added, even if they do not use that field No, you maintain the older versions of the API. V1 of the API u

> every application has to be updated when a field is added, even if they do not use that field No, you maintain the older versions of the API. V1 of the API uses the V1 struct. V2 of the API uses the V2 struct, etc. Older applications maintain compatibility because it calls the older APIs, and you can convert between V1 to V2 and only keep one version of the API. Or, if you want, you can maintain both versions of the API, V1 and V2, at the expense of maintenance costs. But it's absolutely doable (we did it for decades).

taeric 8y ago
You are missing when you have a middle layer. Message comes in at v3 and hits a layer that only knows v1 then gets passed to a layer that is at v4. I'd wager most places don't have that many layers. But, if you are embracing microservices, you'll find yourself here fairly fast.
- docker_up 8y ago
  I doubt any api call trying to work in such a chaotic environment would actually work, and having all optional fields won't magically make things work. It will probably fail but in very mysterious ways. This sounds more like an environment where microservices are completely out of control and chaotic.
  - vkou 8y ago
    Doubt all you will, but being able to update different pieces of your service independently is a huge win. Having to update everything in lockstep is what I'd describe as 'out of control'. Which is precisely what forcing every field to be required does. There's a million and one reasons for why you may want to push, or rollback only 1 out of X services. If you follow a few simple rules when adding/removing fields, you can do this safely.
    - deleted 8y ago
      [deleted]
    - docker_up 8y ago
      You don't need to update everything lockstep. You maintain compatibility by converting the structs back and forth. However, all ambiguity is gone and everything is well understood.
      - taeric 8y ago
        This fails in several key spots. First, not every update should require a complete migration of all at rest or in flight data. Adding that as a requirement will instantly kill many efforts. Second, you can't convert "to and from" a format you don't have specified yet. So, you have to first push out all code that can convert to a new version into all spots before you can start using that version. This will similarly kill many efforts. If you are a smaller shop, you can jump both of those hurdles. But, I will gladly assert that extra work of jumping those hurdles will grow faster than your ability to advance past them.
  - jeremyjh 8y ago
    No one said that all fields should be optional, only that it should be possible to have optional fields. It may sound like chaos to those who've not worked in such environments, but in fact it is not at all unusual in large companies to have the same message pass through several layers of services in a single call-flow. This long predates the term "microservices".
    - slavik81 8y ago
      > No one said that all fields should be optional As of protobuf v3, all fields must be optional. https://github.com/protocolbuffers/protobuf/issues/2497 https://github.com/protocolbuffers/protobuf/issues/2497
      - taeric 8y ago
        Unless you are pushing validation of your data to an overly simplistic language specification, you will have to validate your data in your code. Once you are doing that, you might as well get used to the fact that you can't trust any data at rest. Even if you supposedly wrote it. To that end, I fully agree that all fields should be "optional" for the protocol layer.
- groestl 8y ago
  > You are missing when you have a middle layer. And storage. People might have petabytes of historic data stored in protobufs.

taeric 8y ago

You are missing when you have a middle layer. Message comes in at v3 and hits a layer that only knows v1 then gets passed to a layer that is at v4. I'd wager most places don't have that many layers. But, if you are embracing microservices, you'll find yourself here fairly fast.

docker_up 8y ago
I doubt any api call trying to work in such a chaotic environment would actually work, and having all optional fields won't magically make things work. It will probably fail but in very mysterious ways. This sounds more like an environment where microservices are completely out of control and chaotic.
- vkou 8y ago
  Doubt all you will, but being able to update different pieces of your service independently is a huge win. Having to update everything in lockstep is what I'd describe as 'out of control'. Which is precisely what forcing every field to be required does. There's a million and one reasons for why you may want to push, or rollback only 1 out of X services. If you follow a few simple rules when adding/removing fields, you can do this safely.
  - deleted 8y ago
    [deleted]
  - docker_up 8y ago
    You don't need to update everything lockstep. You maintain compatibility by converting the structs back and forth. However, all ambiguity is gone and everything is well understood.
    - taeric 8y ago
      This fails in several key spots. First, not every update should require a complete migration of all at rest or in flight data. Adding that as a requirement will instantly kill many efforts. Second, you can't convert "to and from" a format you don't have specified yet. So, you have to first push out all code that can convert to a new version into all spots before you can start using that version. This will similarly kill many efforts. If you are a smaller shop, you can jump both of those hurdles. But, I will gladly assert that extra work of jumping those hurdles will grow faster than your ability to advance past them.
- jeremyjh 8y ago
  No one said that all fields should be optional, only that it should be possible to have optional fields. It may sound like chaos to those who've not worked in such environments, but in fact it is not at all unusual in large companies to have the same message pass through several layers of services in a single call-flow. This long predates the term "microservices".
  - slavik81 8y ago
    > No one said that all fields should be optional As of protobuf v3, all fields must be optional. https://github.com/protocolbuffers/protobuf/issues/2497 https://github.com/protocolbuffers/protobuf/issues/2497
    - taeric 8y ago
      Unless you are pushing validation of your data to an overly simplistic language specification, you will have to validate your data in your code. Once you are doing that, you might as well get used to the fact that you can't trust any data at rest. Even if you supposedly wrote it. To that end, I fully agree that all fields should be "optional" for the protocol layer.
groestl 8y ago
> You are missing when you have a middle layer. And storage. People might have petabytes of historic data stored in protobufs.