3 ms·
These kind of posts are awesome tabletop scenarios. (and fuel for https://twitter.com/badthingsdaily https://twitter.com/badthingsdaily ) Coming up with potenti
by viraptor 6y ago
These kind of posts are awesome tabletop scenarios. (and fuel for https://twitter.com/badthingsdaily https://twitter.com/badthingsdaily ) Coming up with potential fixes is a good exercise.
Things I would do that I haven't seen mentioned yet:
- Grab images from the relevant servers to have current backups before messing with anything.
- Try tcpdump-ing and decoding the custom KV protocol - other databases are complicated, but things like memcached/redis/... have trivial wire protocol. Maybe it's possible to recover some data by listing all entries.
- Run `strings` on the KV binary or `file` on the data to make sure it's not just some open source project with a proprietary interface. With any luck it's just something like bdb that can be accessed externally.
- Hook up a debugger to the KV and try to recover the keys (or maybe just strace to see if they keys are in any files)
- Start tcpdump on the old DNS IPs. They mentioned they have no idea which domains they own anymore - as long as NS on customer domains wasn't trashed as well, they could collect queries coming in and recover a list from there.
- See if the customer domains can be collected from billing data / some email store of notifications.
(as much as I try to design for no disasters in production, the rush of "this is so FUBAR, what insane thing I can do to fix it" for me is an amazing feeling :) )