4 ms·
These experiences are relatively common, we just don't see writeups that often. Kudos to the author for writing this up. I do disagree with some of the lessons
by klodolph 5y ago
These experiences are relatively common, we just don't see writeups that often. Kudos to the author for writing this up. I do disagree with some of the lessons here.
You want to switch operating systems on your server?
1. Set up monitoring. If you don't have monitoring already, hack something together with simple scripts.
2. Start up a new server.
3. Migrate services and data from your existing server to your new server. Do this at whatever pace you feel is appropriate. Point the monitoring scripts at your new server.
4. Switch DNS records.
5. Wait. You are not in a hurry to turn off the previous server. Why not wait one or two months?
6. Turn off the old server.
7. Wait some more, and then delete the old server.
The idea here is that steps which might take your site offline are easily reversible. For example, switching DNS records. It's trivial to switch the DNS records back if your migration unexpectedly failed. As much as reasonable, you want the ability to go backwards and undo the steps that you've done to get back to a known good setup.
In particular, I would say that backups are usually not the right tool for migration. This is missing from the lessons at the bottom of the article. The way you get more confidence in your backups is by doing restore tests into a sandbox environment, by adding automated monitoring to your backups, etc. Trying to address a lack of confidence by increasing the backup frequency doesn't make sense. The backup frequency is the most trivial thing to adjust and doesn't address deeper issues, like the fact that you need to dump/restore databases properly and shouldn't copy files from a live database. These issues are discovered through restore testing.
The saying goes, "Nobody wants a backup system, everyone wants a restore system." If you are making backups but not testing restores, you're gonna get bitten. Test the part of backups that you care about--the ability to restore data--and don't test it live. Test it in a sandbox.
- livueta 5y agoThis approach saved my bacon on a migration just yesterday. I had gotten to testing files after having done a big baseline rsync, stopping file services on the old server, and doing a catch-up rsync incremental. Oh shit - one veracrypt container is corrupted and won't mount. Turns out rsync diff updates and mounted containers being written to don't play nice together. Since the old machine was still sitting there with all the data accessible, I was able to just blow away the corrupted volume, confirm it was unmounted on the source, then copy the whole thing over. If I just had a one-time copy that I'd thrown at b2 or something, I would have been very sad. So, yeah, test restoring your backups. Even fancy checksumming filesystems and shit won't save you from bad assumptions about the integrity of your data.
- lisper 5y agoAnything you do in production should always be reversible at least one step back with a single command. You should always be able to roll back to the last-known-working version at the first sign of trouble with the new version. (Not that you should necessarily respond to all trouble with a roll back but you should always design your processes to keep this as one of your options.) The way I usually do this is that I have a production PUSH and ROLLBACK script. There is a single symlink on the production server that points to the current working version of the code. PUSH makes a complete local copy of the current working version, changes the symlink to point to that, then pushes the new version, then changes the symlink to point to that. ROLLBACK just changes the symlink back to the backup copy. This is robust and easy to tweak to allow you to roll back as many levels as you like, though I've never had to go back more than one. If a problem doesn't manifest itself immediately you probably want to fix it by going forward, not back.
- klodolph 5y agoIn the corporate world, this is also a great way to hold vendors accountable. Story: Company Red hires company Blue to take over the company blog as part of a marketing initiative. Their point of contact was the marketing team at company Red, and I'm sure they thought, "We've nabbed a big customer, this is going to be great." However, the switch from the internal blog to the external blog goes through the operations team at company Red. The operations team tells company Blue, "Here is the plan for rolling back to the internal blog at the touch of a button." Company Blue was suddenly much more responsive to questions from company Red.
- cfn 5y agoI do a variation of this when moving to a new developer machine. I virtualize the previous one and keep it around for a long time.
- CogitoCogito 5y agoYeah I was legitimately confused why there was a problem until I realized that author had installed the new OS to the same server as the old one. I can't understand why that was done. Previously I wouldn't even have called your method of moving services over one by one while keeping the old setup available "best practice" because I thought it was so obvious. I hope the original author realized that this should really be what is learned from the story.
- neurostimulant 5y agoI would also setup a reverse proxy on the old server so it would forward all traffics to the new server, and finally kill the old server once the reverse proxy no longer see any legitimate requests.
- klodolph 5y agoOh man, I totally forgot about that!