Been there, done that. I managed to loose 30 days of billing data without a working backup. At a 7 man music streaming startup 10+ years ago, there was an issu

Been there, done that. I managed to loose 30 days of billing data without a working backup. At a 7 man music streaming startup 10+ years ago, there was an issue with our production server. The application was working, but the reporting tool on another server was no longer getting the daily copy of the Firebird database from the live server. The database server had run out of disk space so was no longer able to make backups to transfer. So I stopped the apps, stopped the database, cleared out a lot of old logs and backups that was no longer needed, and brought everything back up again. And then I swore, as I am sure YP did at Gitlab when he realised what had just happened. The database had started up using as you would expect its last persisted state. In this case, its state 30 days as it was then it ran out of disk space. :( Firebird had been happy running in memory since then though not able to persist any changes. And since the backup procedure was to export the database to a local disk file then scp it to other nodes, it had been happily transferring 0 bytes files for weeks. :/ Had I cleared up some disk space then exported the database before I shut it down then there would have been no problem. As I realised the severity of this I quickly got hold of our CEO to say I totally fucked up. We then worked together to piece together the missing data from access logs, 3rd party purchase records, and other reports and sources that he had available. We managed to rebuild most of the missing data though there were some gaps in the last 7 days. Not recovering 30 days at all may have killed our tiny company. We learnt from the mistake and I worked there happily for another year before the company got bought up. Naturally the next project I did was to write a decent alerting system (I won't go into the 300 duplicate text messages I received from it during one night whilst on holiday in southern France). I have made many mistakes since, just never the same mistake. And with the years I take better and better precautions, scale horizontally, test backup restores etc. But mistakes still happen, just don't panic, and don't try to be the midnight oil hero :)