4 ms·
I too wonder if any of these experiments actually affected real customers.
by kostarelo 9y ago
I too wonder if any of these experiments actually affected real customers.
- zzzcpan 9y agoNot affecting real customers is kind of the point of controlled experiments.
- drdrey 9y agoYou can very much impact real customers, but the idea is that you should be able to only apply the experiment to a small subset of customers and abort it in case something goes wrong. For instance, you may want to check that the authorization service can function without access to the A/B testing service, so you cut the network connection between them. If authoritazion errors start rising, you stop the experiment and investigate why this happened. You may also find that clients did not retry properly on unexpected errors from the authorization service (e.g. http 500 error codes)
- falcor84 9y agoThis is what an SLA is for, no? If you have otherwise upheld your targets, you can use the remaining slack at the end of the period to run larger experiments such as these, for a long term benefit.
- extrapickles 9y agoThey have certainly impacted real customers. The idea is that ideally these outages can be easily stopped, and your engineers know exactly what the chaos monkey did rather than just seeing an uptick in errors. Basically you are creating a fault at a time of your choosing to prevent the same fault from occurring at the worest possible time.