I worked at enough companies to see that many of them have some notion of "centralized config that's rolled to the edge and can be updated at runtime". I've do

I worked at enough companies to see that many of them have some notion of "centralized config that's rolled to the edge and can be updated at runtime". I've done this with djb's CDB (constant database). But I've seen people poll an API for JSON config files or dbm/gdbm/Berkeleydb/leveldb. This can extend to other big red buttons. It's not that elegant but I've had numerous services that checked for the presence of a file to serve health checks. So pulling a node out of load balancer rotation was as easy as creating a file. The idea is that then when there's a datab outage the system defaults to serving the last known good config.