3 ms·
If you would permanently delete data for selected customers from a large multitenant system, it could actually take some time to restore it - even with proper b
by jpalomaki 5y ago
If you would permanently delete data for selected customers from a large multitenant system, it could actually take some time to restore it - even with proper backups.
You can’t just do a full recovery as that would mess those customers who were not affected (it likely takes time to notice the mistake - others have continued to use the system). You might need to write some tools to migrate the data from backups. Also you really need to test everything very carefully - otherwise you might be in even deeper trouble (looking at corrupted instead of lost data).
In large organization this kind of ”manual” recovery might require people from multiple teams as no single person knows all the areas. This adds overhead. Throwing too many people in does not help either. When you start thinking about it, few weeks is not that long.
And JIRA is definitely not simple. It’s complicated beast and likely the SaaS features combined with all the legacy makes it even more complicated.
- Gigachad 5y agoWhat a nightmare situation. Makes you wonder if some kind of 1 database per customer setup would be preferable here since you could restore only affected customers.
- wonderwonder 5y agohow would that require hundreds of engineers though? Would one not build a script and then just run it for each customer or each dept build a script? I honestly have no idea, never been in that sort of recovery situation before.
- tcangussu 5y agoI was in a similar situation many years ago (different ticket software though). What we did was spinning up a spare server with the backup data and script an extraction/injection tool to populate the production multitenant sass.