3 ms·
Very considerate that they mentioned negative impact for users in the last point. I wonder though how well that is measured and controlled in real-world applica
by xg15 9y ago
Very considerate that they mentioned negative impact for users in the last point. I wonder though how well that is measured and controlled in real-world applications of this design style.
- kostarelo 9y agoI too wonder if any of these experiments actually affected real customers.
- zzzcpan 9y agoNot affecting real customers is kind of the point of controlled experiments.
- drdrey 9y agoYou can very much impact real customers, but the idea is that you should be able to only apply the experiment to a small subset of customers and abort it in case something goes wrong. For instance, you may want to check that the authorization service can function without access to the A/B testing service, so you cut the network connection between them. If authoritazion errors start rising, you stop the experiment and investigate why this happened. You may also find that clients did not retry properly on unexpected errors from the authorization service (e.g. http 500 error codes)
- falcor84 9y agoThis is what an SLA is for, no? If you have otherwise upheld your targets, you can use the remaining slack at the end of the period to run larger experiments such as these, for a long term benefit.
- extrapickles 9y agoThey have certainly impacted real customers. The idea is that ideally these outages can be easily stopped, and your engineers know exactly what the chaos monkey did rather than just seeing an uptick in errors. Basically you are creating a fault at a time of your choosing to prevent the same fault from occurring at the worest possible time.
- lallysingh 9y agoIt depends on your customer behavior. If there customer can retry, then a small failure is trivially passed.
- aaronblohowiak 9y agoI gave a talk on minimizing blast radius at velocity last year: https://m.youtube.com/watch?feature=youtu.be&v=C11LNUEaHuo https://m.youtube.com/watch?feature=youtu.be&v=C11LNUEaHuo