3 ms·
It's interesting that there's a common theme for multi-hour outages being gated by waiting for backups to restore -- the GitLab outage was similar in cause and
by cjbprime 10y ago
It's interesting that there's a common theme for multi-hour outages being gated by waiting for backups to restore -- the GitLab outage was similar in cause and had an even longer duration.
I wonder if there's anything to be done about reducing the duration of these outages. I suppose copying TBs of data around the world is simply always going to take hours.
- miguel250 10y agoOne thing I have seen done is taking filesystem (ex. ZFS) snapshots every n minutes on a secondary server. You can then restore the secondary and promote it to master.
- danappelxx 10y agoNot an expert, but can't you just temporarily make the replica the master?
- onozz 10y agoNot if a drop database was done - the replica would have followed and dropped it too
- ulysses 10y agoIn this case, it looks like they could have, since the replica was time-delayed -- stop replication, then carefully let it continue replication up to the point you want, then switch it to master.