WE have two things that could need to be rolled back, the app/api and the dataset. The App is docker, so we have a tag called app-production, and app-productio

WE have two things that could need to be rolled back, the app/api and the dataset. The App is docker, so we have a tag called app-production, and app-production-1(up to 5) which are all the previous production versions. If anything goes wrong, we can flip over to the last known good version. We are multi-region, so we don't update all at once. The dataset is a bit harder. Because its > 100gigs, and for speed purposes it lives on EFS (its lots of 4meg files, and we might need to pull in 60 or so files at once, access time is rubbish using S3) Manually syncing it takes a couple of hours. To get round this, we have a copy on write system, with "dataset-prod" and "dataset-prod-1" up to 6. Changing the symlink of the top level directory is minimal.