4 ms·
At my job, my team owns a service that generally has great uptime. Dependent teams/services have gotten into the habit of assuming that our service will be 100%
by kevindong 6y ago
At my job, my team owns a service that generally has great uptime. Dependent teams/services have gotten into the habit of assuming that our service will be 100% available which is problematic because it's obviously not. That false assumption has caused several minor incidents unfortunately.
There have been some talk internally of doing chaos engineering to help improve the reliability of our company's products as a whole. Unfortunately, the most easily simulatable failure scenarios (e.g. entire containers go down at once instantly, etc.) tend to be the least helpful since my team designed the service to tolerate those kinds of easily modelable situations.
The more subtle/complex/interesting failure conditions are far harder to recognize and simulate (e.g. all containers hosted on one particular node experience 10s latencies on all network traffic, stale DNS entries, broken service discovery, etc.).