3 ms·
The report is deliberately obscure. My interpretation is... Atlassian Cloud is hosted in shards where a shard is allocated a single AWS RDS database with (say)
by everfrustrated 5y ago
The report is deliberately obscure.
My interpretation is... Atlassian Cloud is hosted in shards where a shard is allocated a single AWS RDS database with (say) 100 customers.
Their script was run improperly and dropped tables from 400 customers across 400 different shards (across different databases)
Their backup strategy was just to restore RDS backups (recover whole database with many customers data together)
But they can't simply restore from RDS backups due to the co-mingling of customers.
As otherwise unaffected customers would now lose data.
I suspect they're manually restoring RDS backups to new DB instances and having to scrape out the data for a single customer - a process they'd not built any tooling for as it was a fault case they thought they didn't need. Possibly as running the Jira/confluence backups are very heavy and create a lot of load/disk that they can't easily accomodation in a multi tenant hosting architecture.
- 9935c101ab17a66 5y agoInteresting hypothesis, and if true, I have the utmost empathy for the engineering teams that have to figure this out a way to kludge all the right data together. The thought of extracting the lost data from the sharded backups, and then merging it into a possibly a shard with good, newer customer data makes my skin crawl.
- boboche 5y agoIf their backup strategy is as efficient as the UX of their tools, they should find what they are looking for in 2 weeks… oh wait… At least right now they are focussed on this issue and that should stop from outputting yet more unwanted bloat in my daily ops for a while. Guess its time to pay Asana another visit to see where they are at. Didn’t feel it was worth the move 2 years ago, maybe things have changed.