3 ms·

How do you handle binary rollbacks and rollouts safely with an "everything is required" approach? Do you force binaries to roll out in a strict order with appro

by dweis 8y ago

How do you handle binary rollbacks and rollouts safely with an "everything is required" approach? Do you force binaries to roll out in a strict order with appropriate soak time at each layer? How does that affect developer velocity?

deleted 8y ago
[deleted]
docker_up 8y ago
No, you create conversion routines that convert between different versions of structs. This keeps things well understood with no ambiguity. This is very easy, we were doing this 20 years ago and autogenerating the conversions using ANTLR that parsed the XDR files for ONC/RPC.
- joshuamorton 8y ago
  This doesn't make sense with the concept of rollbacks. If I rollout server version 2 and client version 2 which each use a new required field, and then realize that there is some terrible error in server version 2, I can't roll it back to version 1, since it will reject all client calls from v2 clients. The only way to make it work is to add a translation layer, as you suggest, on the server, wait a while, push the new 'required' client, wait a while longer, and then push the server without the translation layer. That's the "strict ordering with appropriate soak time" GP mentions.
  - docker_up 8y ago
    Instead, you're going to get errors from the clients using version 2, because server version 2 was rolled back. You have to roll back the clients as well then. Or you could have client version 2 know how to automatically convert to server version 1, because you're know what version the server is on, and you can convert your client parameters or even behavior to fit version 1. You can't do this with protobufs because there is no such concept, you just add optional fields, and ignore them with different versions, and it's chaos.
    - joshuamorton 8y ago
      >Instead, you're going to get errors from the clients using version 2, because server version 2 was rolled back. You have to roll back the clients as well then. This depends on the update. If indeed the field was optional, you won't. A common example would be a field that is necessary for a new feature, but without which everything functions just fine, or functions with a minor degredation in experience. But more importantly, you want to decouple features from api versions and communication protocols. It should be possible to enable an feature without rolling out a new version (for example, via a flipped flag). So a degraded communication protocol shouldn't actually impact anything. The application worked fine an hour ago, adding a new field won't break it. >You have to roll back the clients as well then. And if the clients are phones? >Or you could have client version 2 know how to automatically convert to server version 1 This requires communicating with the server beforehand. Why should I have to roundtrip to negotiate which api version I should use? And what happens if you want to modify the protocol that you use to decide on api versions? Its turtles all the way down. >because you're know what version the server is on, and you can convert your client parameters or even behavior to fit version 1. So now, before I can roll out a server update, I need to roll out a client update that can make sure to negotiate back to a degraded experience until I update the server. Then I have to wait until that new client is rollback safe. Then I can update the server, then eventually I can update the client to remove the shim code. That's the "strict ordering with appropriate soak time" issue that you're still running into. This has cascading consequences, each server/client update dance has to be mostly atomic, so you can only really do one of these dances at a time, and all clients have to be in sync. If a deep dependency service wants to make an api change, it has to wait until all of the clients are prepared before updating, and if any of those clients is the server in another context, they have to wait until everyone is ready. That's chaos. And I don't want to do that when the other option is "update the server whenever, as clients upgrade, they'll see the improved service". You avoid the dance. It moves the initiative of an upgrade from the client to the server, and this is a good thing, because there are more clients than servers.
      - docker_up 8y ago
        We agree to disagree. I don't think you can convince me that all optional is better than all required and vice versa, which is okay. My point is required fields makes software age better over the long run because everything is explicit. If you don't agree, that's your prerogative. Everyone thought NOSQL without schemas was a godsend, until their code/service iterated a dozen times, developers leave, documentation gets out of date, and now all their older data can't be read because it doesn't match the code. Same thing holds true for RPC, in my opinion. Yours may differ.

by dweis 8y ago

deleted 8y ago

[deleted]

docker_up 8y ago

No, you create conversion routines that convert between different versions of structs. This keeps things well understood with no ambiguity. This is very easy, we were doing this 20 years ago and autogenerating the conversions using ANTLR that parsed the XDR files for ONC/RPC.

joshuamorton 8y ago
This doesn't make sense with the concept of rollbacks. If I rollout server version 2 and client version 2 which each use a new required field, and then realize that there is some terrible error in server version 2, I can't roll it back to version 1, since it will reject all client calls from v2 clients. The only way to make it work is to add a translation layer, as you suggest, on the server, wait a while, push the new 'required' client, wait a while longer, and then push the server without the translation layer. That's the "strict ordering with appropriate soak time" GP mentions.
- docker_up 8y ago
  Instead, you're going to get errors from the clients using version 2, because server version 2 was rolled back. You have to roll back the clients as well then. Or you could have client version 2 know how to automatically convert to server version 1, because you're know what version the server is on, and you can convert your client parameters or even behavior to fit version 1. You can't do this with protobufs because there is no such concept, you just add optional fields, and ignore them with different versions, and it's chaos.
  - joshuamorton 8y ago
    >Instead, you're going to get errors from the clients using version 2, because server version 2 was rolled back. You have to roll back the clients as well then. This depends on the update. If indeed the field was optional, you won't. A common example would be a field that is necessary for a new feature, but without which everything functions just fine, or functions with a minor degredation in experience. But more importantly, you want to decouple features from api versions and communication protocols. It should be possible to enable an feature without rolling out a new version (for example, via a flipped flag). So a degraded communication protocol shouldn't actually impact anything. The application worked fine an hour ago, adding a new field won't break it. >You have to roll back the clients as well then. And if the clients are phones? >Or you could have client version 2 know how to automatically convert to server version 1 This requires communicating with the server beforehand. Why should I have to roundtrip to negotiate which api version I should use? And what happens if you want to modify the protocol that you use to decide on api versions? Its turtles all the way down. >because you're know what version the server is on, and you can convert your client parameters or even behavior to fit version 1. So now, before I can roll out a server update, I need to roll out a client update that can make sure to negotiate back to a degraded experience until I update the server. Then I have to wait until that new client is rollback safe. Then I can update the server, then eventually I can update the client to remove the shim code. That's the "strict ordering with appropriate soak time" issue that you're still running into. This has cascading consequences, each server/client update dance has to be mostly atomic, so you can only really do one of these dances at a time, and all clients have to be in sync. If a deep dependency service wants to make an api change, it has to wait until all of the clients are prepared before updating, and if any of those clients is the server in another context, they have to wait until everyone is ready. That's chaos. And I don't want to do that when the other option is "update the server whenever, as clients upgrade, they'll see the improved service". You avoid the dance. It moves the initiative of an upgrade from the client to the server, and this is a good thing, because there are more clients than servers.
    - docker_up 8y ago
      We agree to disagree. I don't think you can convince me that all optional is better than all required and vice versa, which is okay. My point is required fields makes software age better over the long run because everything is explicit. If you don't agree, that's your prerogative. Everyone thought NOSQL without schemas was a godsend, until their code/service iterated a dozen times, developers leave, documentation gets out of date, and now all their older data can't be read because it doesn't match the code. Same thing holds true for RPC, in my opinion. Yours may differ.