2 ms·
Poor guys. Really interesting reading though. I initially thought it was weird they had to run several "processes" in case 1 failed. But running out of space o
by shubb 11y ago
Poor guys. Really interesting reading though.
I initially thought it was weird they had to run several "processes" in case 1 failed. But running out of space or something correctable is actually something likely to happen. Is this standard? It's quite smart.
Anyway, assuming they get the data back, I think they've done pretty good - 0 data loss and a days downtime isn't bad given this is a true disaster.
It would be nice if they'd let us know how the db got deleted, and what they suggest to mitigate in a blog after.
- ndespres 11y agoIn my experience it's good to have a few different restore attempts running in parallel, so long as they won't conflict with one another. For example, the restores may be to different hardware, from a different point-in-time, in a different datacenter, or from a unique backup system (image-based, file-based, transaction logs only, etc). One of their "restore" attempts could even be a scan of the original disk for recoverable data, though that seems unlikely. In a critical outage like the one Gliffy is experiencing, I take the same approach. Outline all your restore options, estimate time for each restore, drawbacks of each approach, etc and take every available angle. If you want to know how the database was deleted, read through some of the horror stories posted in the comments here and assume it was probably one of those!
- jsjohnst 11y agoThe mitigation is relatively simple, don't allow (by policy and by enforcement) anyone to directly use the creds with DDL access in prod. All DDL changes should be checked in, code reviewed, and tested on pre-prod environments before being ran on prod.