4 ms·

Update on the April 5th, 2017 Outage

59 points by janvdberg 10y ago

sb8244 10y ago
Does this mean that their testing servers can access production db over network? That's something we specifically put under control so that even a misconfiguration can't cause this. I'm surprised an audit didn't discover that.
lloydde 10y ago
> The root cause of this incident was a engineer-driven configuration error. A process performing automated testing was misconfigured using production credentials. As such, we will be drastically reducing access to the primary system for certain actions to ensure this does not happen again. Classic. Someone manually ran the test automation while pointing at production including its cleanup routine of DROP. Very foundational access control was not in place.
cjbprime 10y ago
It's interesting that there's a common theme for multi-hour outages being gated by waiting for backups to restore -- the GitLab outage was similar in cause and had an even longer duration. I wonder if there's anything to be done about reducing the duration of these outages. I suppose copying TBs of data around the world is simply always going to take hours.
- miguel250 10y ago
  One thing I have seen done is taking filesystem (ex. ZFS) snapshots every n minutes on a secondary server. You can then restore the secondary and promote it to master.
- danappelxx 10y ago
  Not an expert, but can't you just temporarily make the replica the master?
  - onozz 10y ago
    Not if a drop database was done - the replica would have followed and dropped it too
  - ulysses 10y ago
    In this case, it looks like they could have, since the replica was time-delayed -- stop replication, then carefully let it continue replication up to the point you want, then switch it to master.
spotman 10y ago
Imagine that! Working backups! nice job not losing data digital ocean.
ulysses 10y ago
I like DO, and appreciate them being open with the outage timeline. Two thing really jump out at me though: 1.) If they have a time-delayed replica, why not recover it to the point-in-time before the issue, and then switch master over to it? 2.) They took a backup of the replica, moved the backup to the master, and then restored from it. Do they not have direct backups of the master? A time-delayed replica is no replacement for valid backups.

59 points by janvdberg 10y ago

Does this mean that their testing servers can access production db over network? That's something we specifically put under control so that even a misconfiguration can't cause this. I'm surprised an audit didn't discover that.

lloydde 10y ago

> The root cause of this incident was a engineer-driven configuration error. A process performing automated testing was misconfigured using production credentials. As such, we will be drastically reducing access to the primary system for certain actions to ensure this does not happen again. Classic. Someone manually ran the test automation while pointing at production including its cleanup routine of DROP. Very foundational access control was not in place.

cjbprime 10y ago

It's interesting that there's a common theme for multi-hour outages being gated by waiting for backups to restore -- the GitLab outage was similar in cause and had an even longer duration. I wonder if there's anything to be done about reducing the duration of these outages. I suppose copying TBs of data around the world is simply always going to take hours.

miguel250 10y ago
One thing I have seen done is taking filesystem (ex. ZFS) snapshots every n minutes on a secondary server. You can then restore the secondary and promote it to master.
danappelxx 10y ago
Not an expert, but can't you just temporarily make the replica the master?
- onozz 10y ago
  Not if a drop database was done - the replica would have followed and dropped it too
- ulysses 10y ago
  In this case, it looks like they could have, since the replica was time-delayed -- stop replication, then carefully let it continue replication up to the point you want, then switch it to master.

spotman 10y ago

Imagine that! Working backups! nice job not losing data digital ocean.

ulysses 10y ago

I like DO, and appreciate them being open with the outage timeline. Two thing really jump out at me though: 1.) If they have a time-delayed replica, why not recover it to the point-in-time before the issue, and then switch master over to it? 2.) They took a backup of the replica, moved the backup to the master, and then restored from it. Do they not have direct backups of the master? A time-delayed replica is no replacement for valid backups.