3 ms·
GitLab.com melts down after wrong directory deleted, backups fail
- pmontra 10y agoIs anybody using a procedure to automatically perform restores and check that they match production data? I guess it's not easy for complex systems and non trivial amounts of data. Do you want to share the details and give us an idea of the costs? Thanks.
- paktek123 10y agoI've setup a postgresql backup before that would use pgbasebackup and get the latest xlog segments then I would start up postgres and run select 1 against it. That simply wasn't enough as I still got corrupted indexes but at least data was there. If I were to do it again I'd probably run a whole production test suite against it. The cost is extra hardware and time as backup boxes aren't made for performance for a 1Tb+ database it would take more than 24 hours.
- vonklaus 10y agoI just really like gitlab. I am not personally affected by the outage, but as far as collosul fuckups go, I think they will do their level best to fix the issue, be transparent about it and do what they can to make it right. We'll get a really good measure of the company by how this turns out.
- jwilk 10y agohttps://news.ycombinator.com/item?id=13537052 https://news.ycombinator.com/item?id=13537052