12 ms·

Only partially related, but since they mention it at the bottom of the post: I really like GH's "brownout" approach to discontinuing services. They've done it a

Only partially related, but since they mention it at the bottom of the post: I really like GH's "brownout" approach to discontinuing services. They've done it a couple of times before (like this one[1]), and it's always struck me as a very sensible way to handle deprecations on such a massive service. [1]: https://github.blog/changelog/2021-08-10-brownout-notice-api-authentication-via-query-parameters-for-48-hours/ https://github.blog/changelog/2021-08-10-brownout-notice-api...

js2 4y ago
These are also known as scream tests. We use them internally at $dayjob as part of EOL'ing APIs. Are they not common?
- woodruffw 4y ago
  I have no clue! I just appreciated them as a lowly user.
- CodexArcana 4y ago
  Lol brown outs is such a nicer name, but I've always known them as a scream test.
  - adamckay 4y ago
    They're ever so slightly different. Scream test is to discontinue service entirely and see who screams, akin to pulling the plug on a server in a rack when you've got no idea what's running on it, to then plug in back in when you find out it's the finance team running payroll software. Brown outs are typically short-lived purposeful errors or aborts to either shed load to prevent a failure due to capacity exhaustion or cause issues that are investigated by a human to find the service has been sunsetted which raise awareness that something needs to be fixed before it's totally broken.
travisjungroth 4y ago
Make sure to get the rate right. Like 1/1,000 or 1/100,000. Poetry did a brownout of its install script at 5% failure. With a matrix of builds user's CI had closer to a 100% chance of failing.
jacobr1 4y ago
We started using feature flags to disable, rather than enable capabilities. Turn them off and see if anyone notices. Ramp up the rate. If you identify some cohort that needs the feature, keep it on for them, but isolate them until you figure out a migration plan to close whatever gap you may have (or at least some way to massage unhappy customers if you aren't going to support some legacy thing).

js2 4y ago

These are also known as scream tests. We use them internally at $dayjob as part of EOL'ing APIs. Are they not common?

woodruffw 4y ago
I have no clue! I just appreciated them as a lowly user.
CodexArcana 4y ago
Lol brown outs is such a nicer name, but I've always known them as a scream test.
- adamckay 4y ago
  They're ever so slightly different. Scream test is to discontinue service entirely and see who screams, akin to pulling the plug on a server in a rack when you've got no idea what's running on it, to then plug in back in when you find out it's the finance team running payroll software. Brown outs are typically short-lived purposeful errors or aborts to either shed load to prevent a failure due to capacity exhaustion or cause issues that are investigated by a human to find the service has been sunsetted which raise awareness that something needs to be fixed before it's totally broken.

travisjungroth 4y ago

Make sure to get the rate right. Like 1/1,000 or 1/100,000. Poetry did a brownout of its install script at 5% failure. With a matrix of builds user's CI had closer to a 100% chance of failing.

jacobr1 4y ago

We started using feature flags to disable, rather than enable capabilities. Turn them off and see if anyone notices. Ramp up the rate. If you identify some cohort that needs the feature, keep it on for them, but isolate them until you figure out a migration plan to close whatever gap you may have (or at least some way to massage unhappy customers if you aren't going to support some legacy thing).