3 ms·
I bought down one of Australasia's biggest banks for 8 hours when I requested a backup of the production database for a test cycle we were about to start. Unfo
by geoffmunn 5y ago
I bought down one of Australasia's biggest banks for 8 hours when I requested a backup of the production database for a test cycle we were about to start.
Unfortunately the database structure had changed and the backup script hadn't been updated and it locked all the tables it tried to access, while also waiting for incomplete transactions to finish... which never happened.
$8 million dollars of deposits and withdrawals vanished into thin air.
Luckily for me, there were perimeter logs for all transactions so a team of people replayed the lost transactions and fixed everything.
I had to read the incident reports (which were pretty cringing - people put money into their personal account and there was no record of it ever happening) and write a response which said what went wrong and what I would do differently next time.
No repercussions for me though... I made another request for the database and it worked the next day.