2 ms·
Disclosure: I work in DevOps/SRE and I am honestly a bit put off by what you are saying. I think I'm actually a little bit angry at your comment. It’s easy to
by klodolph 7y ago
Disclosure: I work in DevOps/SRE and I am honestly a bit put off by what you are saying. I think I'm actually a little bit angry at your comment.
It’s easy to say things like “should never be global” and “should not be hard to test”. These are goals, and meanwhile the business must go on, you also have other goals, and you cannot spend your entire budget preventing network outages and testing configs.
The things you are suggesting—separate control plane, non-global configs, making them easy to test—you can find these suggestions in any book on operations. So forgive me if your comment makes me a bit angry.
- grey-area 7y agoSorry about that. It wasn't intended to be a glib response, nor to minimise the work done in these areas, and I'm aware these goals are easy to state and incredibly hard to achieve. I've read the Google SRE book so probably the ideas just came from there. From the outside, it does seem like config is in need of more work, because now that other challenges have been met, it is the one area that consistently causes outages now.
- klodolph 7y agoI will say that every time I’ve seen an outage triggered by a config push, there have been several other bugs involved at the same time. Software / code is still a problem, I wouldn't consider it solved, it’s just that the bugs will turn into outages during config changes.