4 ms·

This approach saved my bacon on a migration just yesterday. I had gotten to testing files after having done a big baseline rsync, stopping file services on the

This approach saved my bacon on a migration just yesterday. I had gotten to testing files after having done a big baseline rsync, stopping file services on the old server, and doing a catch-up rsync incremental. Oh shit - one veracrypt container is corrupted and won't mount. Turns out rsync diff updates and mounted containers being written to don't play nice together. Since the old machine was still sitting there with all the data accessible, I was able to just blow away the corrupted volume, confirm it was unmounted on the source, then copy the whole thing over. If I just had a one-time copy that I'd thrown at b2 or something, I would have been very sad. So, yeah, test restoring your backups. Even fancy checksumming filesystems and shit won't save you from bad assumptions about the integrity of your data.

lisper 5y ago
Anything you do in production should always be reversible at least one step back with a single command. You should always be able to roll back to the last-known-working version at the first sign of trouble with the new version. (Not that you should necessarily respond to all trouble with a roll back but you should always design your processes to keep this as one of your options.) The way I usually do this is that I have a production PUSH and ROLLBACK script. There is a single symlink on the production server that points to the current working version of the code. PUSH makes a complete local copy of the current working version, changes the symlink to point to that, then pushes the new version, then changes the symlink to point to that. ROLLBACK just changes the symlink back to the backup copy. This is robust and easy to tweak to allow you to roll back as many levels as you like, though I've never had to go back more than one. If a problem doesn't manifest itself immediately you probably want to fix it by going forward, not back.
- klodolph 5y ago
  In the corporate world, this is also a great way to hold vendors accountable. Story: Company Red hires company Blue to take over the company blog as part of a marketing initiative. Their point of contact was the marketing team at company Red, and I'm sure they thought, "We've nabbed a big customer, this is going to be great." However, the switch from the internal blog to the external blog goes through the operations team at company Red. The operations team tells company Blue, "Here is the plan for rolling back to the internal blog at the touch of a button." Company Blue was suddenly much more responsive to questions from company Red.

lisper 5y ago

Anything you do in production should always be reversible at least one step back with a single command. You should always be able to roll back to the last-known-working version at the first sign of trouble with the new version. (Not that you should necessarily respond to all trouble with a roll back but you should always design your processes to keep this as one of your options.) The way I usually do this is that I have a production PUSH and ROLLBACK script. There is a single symlink on the production server that points to the current working version of the code. PUSH makes a complete local copy of the current working version, changes the symlink to point to that, then pushes the new version, then changes the symlink to point to that. ROLLBACK just changes the symlink back to the backup copy. This is robust and easy to tweak to allow you to roll back as many levels as you like, though I've never had to go back more than one. If a problem doesn't manifest itself immediately you probably want to fix it by going forward, not back.

klodolph 5y ago
In the corporate world, this is also a great way to hold vendors accountable. Story: Company Red hires company Blue to take over the company blog as part of a marketing initiative. Their point of contact was the marketing team at company Red, and I'm sure they thought, "We've nabbed a big customer, this is going to be great." However, the switch from the internal blog to the external blog goes through the operations team at company Red. The operations team tells company Blue, "Here is the plan for rolling back to the internal blog at the touch of a button." Company Blue was suddenly much more responsive to questions from company Red.