4 ms·

DevOps does not need to carry a pager. DevOps will automate disaster recovery.

by lisivka 11y ago

toomuchtodo 11y ago
I'm a devops engineer, and I assure you, I have been on call for the last two years in my current position, attending to application and infrastructure alarms. All disaster recovery cannot be automated. A sanity check is required before your Elasticsearch or Redis clusters decide to f___ their datastore. EDIT: Or perhaps RDS is to your liking? We've had our replicas (off the same master) in AWS fail, at which point they would not further replicate. We needed to fail our app over to the master entirely, reduce its workload to not kill the master, terminate all of the replicas, relaunch all of the replicas, and then re-apply the workload back to the new replicas.
- creshal 11y ago
  > All disaster recovery cannot be automated. See also: https://github.com/blog/1261-github-availability-this-week https://github.com/blog/1261-github-availability-this-week > The automated failover of our main production database could be described as the root cause of both of these downtime events. In each situation in which that occurred, if any member of our operations team had been asked if the failover should have been performed, the answer would have been a resounding no. There are many situations in which automated failover is an excellent strategy for ensuring the availability of a service. After careful consideration, we've determined that ensuring the availability of our primary production database is not one of these situations. To this end, we've made changes to our Pacemaker configuration to ensure failover of the 'active' database role will only occur when initiated by a member of our operations team.
  - toomuchtodo 11y ago
    Pinterest as well uses manual intervention: https://engineering.pinterest.com/blog/open-sourcing-pinterest-mysql-management-tools https://engineering.pinterest.com/blog/open-sourcing-pintere...
- lisivka 11y ago
  All disaster recovery in general case cannot be automated, you are right. But it definitely can be automated to the point, when problem can wait till next Monday. To recover any disaster automagically, you need five things: your actual data, your working software, a working hardware, a working network connection, money. All that, except actual data, can be prepared ahead of time.
  - toomuchtodo 11y ago
    If your datastore can run in a degraded state through a weekend, its not a critical datastore (or you're shedding writes or making other consistency compromises).
    - lisivka 11y ago
      If you have a single point of failure, then it is your fault.

DevOps does not need to carry a pager. DevOps will automate disaster recovery.

by lisivka 11y ago

DevOps does not need to carry a pager. DevOps will automate disaster recovery.

I'm a devops engineer, and I assure you, I have been on call for the last two years in my current position, attending to application and infrastructure alarms. All disaster recovery cannot be automated. A sanity check is required before your Elasticsearch or Redis clusters decide to f___ their datastore. EDIT: Or perhaps RDS is to your liking? We've had our replicas (off the same master) in AWS fail, at which point they would not further replicate. We needed to fail our app over to the master entirely, reduce its workload to not kill the master, terminate all of the replicas, relaunch all of the replicas, and then re-apply the workload back to the new replicas.

creshal 11y ago
> All disaster recovery cannot be automated. See also: https://github.com/blog/1261-github-availability-this-week https://github.com/blog/1261-github-availability-this-week > The automated failover of our main production database could be described as the root cause of both of these downtime events. In each situation in which that occurred, if any member of our operations team had been asked if the failover should have been performed, the answer would have been a resounding no. There are many situations in which automated failover is an excellent strategy for ensuring the availability of a service. After careful consideration, we've determined that ensuring the availability of our primary production database is not one of these situations. To this end, we've made changes to our Pacemaker configuration to ensure failover of the 'active' database role will only occur when initiated by a member of our operations team.
- toomuchtodo 11y ago
  Pinterest as well uses manual intervention: https://engineering.pinterest.com/blog/open-sourcing-pinterest-mysql-management-tools https://engineering.pinterest.com/blog/open-sourcing-pintere...
lisivka 11y ago
All disaster recovery in general case cannot be automated, you are right. But it definitely can be automated to the point, when problem can wait till next Monday. To recover any disaster automagically, you need five things: your actual data, your working software, a working hardware, a working network connection, money. All that, except actual data, can be prepared ahead of time.
- toomuchtodo 11y ago
  If your datastore can run in a degraded state through a weekend, its not a critical datastore (or you're shedding writes or making other consistency compromises).
  - lisivka 11y ago
    If you have a single point of failure, then it is your fault.