3 ms·

Really interesting post. Can anyone comment on how serverless deployments help and/or exacerbate the deployment problem? To give a theoretical starting point -

Really interesting post. Can anyone comment on how serverless deployments help and/or exacerbate the deployment problem? To give a theoretical starting point - Acme corporation has a serverless API deployed at api.acme.com powering website acme.com. The staging system is deployed at staging.api.acme.com, and the corresponding web app (e.g. staging.acme.com) targets that API. Staging gets deployed to production via a pull request that triggers CI/CD, which has no ability to do an X% rollout (i.e. the underlying serverless function is updated). Developer John Smith is writing a feature and merging to his personal branch "jsmith" - he can test his branch with jsmith.staging.api.acme.com and a corresponding development web app. When he's finished the feature, he does a PR to the staging branch, which is packaged into a PR to live. You may be able to see I'm trying to get input on our own release system :) It's worked quite well so far but I'm still paranoid about not having gradual roll-out - in my experience in bigger companies, there was always some A/B testing in place so bad deploys could be reverted quickly. If the sysop gurus of HN gathering on this thread can pass any comments/critique it would be greatly appreciated.

mandelbrotwurst 6y ago
Why would the ability to roll back deploys be tied to the ability to A/B test?
- icebraining 6y ago
  I'm guessing the A/B functionality was being used as a Blue-Green deployment system.
hibbelig 6y ago
The post was talking about fleets of machines, where just restricting the number of machines that do things seems to be useful. But you only seem to have one machine in production (or one for the API plus one for the website), so it doesn't make sense to deploy to 10% of the machines. What you can do is to arrange things so that each individual upgrade can be rolled back. Let's say you want to rename a parameter in the API. You have to make a backward-compatible change to the API so that the existing website keeps working. In other words, you provide two calls, one with the old parameter name and a different call with the new parameter name. Then you deploy that. If there are problems, you can roll back to the previous API version. The next step is to change the website to invoke the new API call with the new parameter name. If that fails, you can go back to the previous website. Let's say it succeeds. Now you have a website using the new API with the new parameter name, and an API providing both the old and the new parameter name. The next step is to update the API to remove the old parameter name. If it turns out you forgot a place in the website where the old parameter is used, you can roll back to the previous API. If it goes well, you now have a website that uses the new parameter name, and also an API that provides the new parameter name, and the old parameter name is no longer used. It took a long time, but now it's mission accomplished and problem solved.
- primitivesuave 6y ago
  Thank you for the input! I agree with your strategy for changing parameters, but for something more error-prone like changing internal logic behind the same parameters, I would still feel the same anxiety doing a non-gradual release.
  - hibbelig 6y ago
    That is a very good point. And it reminds me how you can do gradual releases when you only have one installation: you do it on a per-user basis, using feature flags. If it's only the logic behind an existing interface, you can implement both the old logic and the new logic and choose between them based on a feature flag.
twic 6y ago
To do a release, create a new serverless function with the new code. Initially, no traffic routes to it, all traffic goes to the old function. Change things so 1% of traffic goes to the new function, and see if it looks okay. If it does, make that 2%, then 3%, or 4%, or 10%, etc. Once it's 100%, delete the old serverless function. The provider hosting your serverless functions might not give you that much control over routing. In that case, you would need to write another serverless function which sits in front of your serverless functions and does the routing. If you ever need to update that, well, then you have to cross your fingers.
- wging 6y ago
  > Once it's 100%, delete the old serverless function. I suggest keeping it around for a while. A function not being used should have cost equivalent to a database entry, not a running container or VM, so this should be cheap and the benefits are large. It's still possible to discover you need to roll back a change, even after you've been at 100% for a while. (And no, 'roll forward' is not a sufficient option, especially if that requires coding up a fix you may not even understand yet.)
- primitivesuave 6y ago
  This seems like a great strategy, thank you for your input! It can indeed be accomplished by AWS Global Accelerator along with application load balancers targeting Lambda functions to route a percentage of traffic. I am not sure about the overhead of having a Lambda function to route requests, I’ll have to look into the latency that adds (especially for cold starts).
  - ultrafez 6y ago
    Using a lambda to route requests to other lambdas would result in you effectively doubling your Lambda cost, as your routing lambda would sit idling while waiting for the target lambda to return a result. If you're using API Gateway, it has a canary deployment option which can route percentages of customer traffic to different Stages, which gives you the gradual rollout capability that you wanted.

mandelbrotwurst 6y ago

Why would the ability to roll back deploys be tied to the ability to A/B test?

icebraining 6y ago
I'm guessing the A/B functionality was being used as a Blue-Green deployment system.

hibbelig 6y ago

The post was talking about fleets of machines, where just restricting the number of machines that do things seems to be useful. But you only seem to have one machine in production (or one for the API plus one for the website), so it doesn't make sense to deploy to 10% of the machines. What you can do is to arrange things so that each individual upgrade can be rolled back. Let's say you want to rename a parameter in the API. You have to make a backward-compatible change to the API so that the existing website keeps working. In other words, you provide two calls, one with the old parameter name and a different call with the new parameter name. Then you deploy that. If there are problems, you can roll back to the previous API version. The next step is to change the website to invoke the new API call with the new parameter name. If that fails, you can go back to the previous website. Let's say it succeeds. Now you have a website using the new API with the new parameter name, and an API providing both the old and the new parameter name. The next step is to update the API to remove the old parameter name. If it turns out you forgot a place in the website where the old parameter is used, you can roll back to the previous API. If it goes well, you now have a website that uses the new parameter name, and also an API that provides the new parameter name, and the old parameter name is no longer used. It took a long time, but now it's mission accomplished and problem solved.

primitivesuave 6y ago
Thank you for the input! I agree with your strategy for changing parameters, but for something more error-prone like changing internal logic behind the same parameters, I would still feel the same anxiety doing a non-gradual release.
- hibbelig 6y ago
  That is a very good point. And it reminds me how you can do gradual releases when you only have one installation: you do it on a per-user basis, using feature flags. If it's only the logic behind an existing interface, you can implement both the old logic and the new logic and choose between them based on a feature flag.

twic 6y ago

To do a release, create a new serverless function with the new code. Initially, no traffic routes to it, all traffic goes to the old function. Change things so 1% of traffic goes to the new function, and see if it looks okay. If it does, make that 2%, then 3%, or 4%, or 10%, etc. Once it's 100%, delete the old serverless function. The provider hosting your serverless functions might not give you that much control over routing. In that case, you would need to write another serverless function which sits in front of your serverless functions and does the routing. If you ever need to update that, well, then you have to cross your fingers.

wging 6y ago
> Once it's 100%, delete the old serverless function. I suggest keeping it around for a while. A function not being used should have cost equivalent to a database entry, not a running container or VM, so this should be cheap and the benefits are large. It's still possible to discover you need to roll back a change, even after you've been at 100% for a while. (And no, 'roll forward' is not a sufficient option, especially if that requires coding up a fix you may not even understand yet.)
primitivesuave 6y ago
This seems like a great strategy, thank you for your input! It can indeed be accomplished by AWS Global Accelerator along with application load balancers targeting Lambda functions to route a percentage of traffic. I am not sure about the overhead of having a Lambda function to route requests, I’ll have to look into the latency that adds (especially for cold starts).
- ultrafez 6y ago
  Using a lambda to route requests to other lambdas would result in you effectively doubling your Lambda cost, as your routing lambda would sit idling while waiting for the target lambda to return a result. If you're using API Gateway, it has a canary deployment option which can route percentages of customer traffic to different Stages, which gives you the gradual rollout capability that you wanted.