4 ms·
> Backups are mentioned almost casually and there is this: "but no process was implemented for ElasticSearch databases". From TFA, next sentence after the one
by thunky 4y ago
> Backups are mentioned almost casually and there is this: "but no process was implemented for ElasticSearch databases".
From TFA, next sentence after the one you quoted:
Also, that database was a read model and by definition, it wasn’t the source of truth for anything. In theory, read models shouldn’t have backups, they should be rebuilt fast enough that won’t cause any or minimal impact in case of a major incident. Since read models usually have information inferred from somewhere else, It is debatable if they compensate for the associated monetary cost of maintaining regular backups
The article discusses how they executed an actual recovery process from the other data sources, but it took longer than it should have (6 days), so...
We ended up with a mix of the two. We refactored the process to a point it went from 6 days to a few hours. However, due to the criticality of the component, a few hours of unavailability still had significant impacts, especially during specific timeframes (e.g. sales seasons). We had a few options to further reduce that time but it started to feel like overengineering and incurred a substantial additional infrastructure cost. So we decided to also include backups when there was a higher risk, like during sales seasons or other business-critical periods.