3 ms·
It sounds to me like they lost some sort of master reference table that maps customers to services, instances, data, and so on. Which would mean a lot of manua
by tyingq 5y ago
It sounds to me like they lost some sort of master reference table that maps customers to services, instances, data, and so on. Which would mean a lot of manual examination to recreate the map. And would give you that situation where maybe the first thirds of restores is somewhat "fast" because it's the bigger customers with more data/logs/activity, and so easier to identify. But now they've hit the long tail.
They may also be gated by limitations in-between customers and the backups. Like backups going through a transit VPC, for example. You don't see the bandwidth limitation when you're sending incrementals all day, but try and do full, parallel, multiple customer restores through it, and ... (and perhaps they are afraid of making too many changes on top of the current chaos to fix something like that)
- ollien 5y ago> It sounds to me like they lost some sort of master reference table that maps customers to services, instances, data, and so on. What makes you say this?
- tyingq 5y agoJust guessing what would drive a situation where some customers were restored quickly, others stuck for days, and matches somewhat with the limited statements they do make about the issue. Like this one: "The rebuild stage is particularly complex due to several steps that are required to validate sites and verify data." Also matches with it being a cross-product outage. But just a guess.
- wlonkly 5y agoAnd not just a cross-product outage, but a cross-product outage that includes acquisitions. I doubt there is that much in common between the JIRA/Confluence tenancy model and, say, Statuspage.