10 ms·

The wrong way to switch operating systems on your server

109 points by ohjeez 5y ago

simonblack 5y ago
On my server, I have three 'root' partitions. One for general day-to-day use, one for a backup if something catastrophic happens to my main system, and one for experimentation. The extra disk space taken up for my having two extra 'root' partitions is a miserable 40-50 gig. But I know that I can swap operating systems over almost instantaneously and then back again just as quickly if I did something wrong. Great peace of mind for practically no cost. The second error I see in the article is to use software that we haven't used previously for something important. My first wife had the habit of trying new recipes when we had a dinner-party. I tried to tell her repeatedly to try the recipe on us first, then she would have it down pat when she wanted to impress. The third error of course was the need to have restorable daily backups and the use of them to restore the system when need be, associated with modularity of the system. I back up my whole system daily. but the most important part of that is not backing up the distro itself (we have re-installs for that) but backing up all the config files, all the databases, all the local binaries, and a current list of all of the installed distro packages. I can replace the whole operating system from a complete wipe-out in less than two hours. I store these backups in a pseudo-exponential policy. I have more recent backups, fewer older backups. Currently I have 15 backups covering 9 years, with five of those covering just the last 3 weeks, and four covering the last five days. To augment this, I have a monthly snapshot backup also stashed away. The other stuff, personal docs etc, is deliberately kept small. Total daily backup of base system and /home is approximately 12 gigs. That is easily transported on a USB stick. I don't store music, photos, magazine .PDFs, old software, etc in my /home directory. That stuff all goes in an archive directory that's write-once, and store (practically) forever. That gets rsynced to two external USB drives daily. Most days, there's practically nothing that gets transferred out. Having several times lost much valuable data, I suppose I am really paranoid, but I still think I haven't been paranoid enough.
ghostly_s 5y ago
Spent three days fighting with this and didn't think to try the -v flag on his apparently-hanging process?
- rhn_mk1 5y ago
  The author may not have the necessary familiarity with it. Experience is gained via mistakes too.
- fak3r 5y ago
  My first thought, also always run long running jobs like this in Screen or Tmux! As it is, it's a good learning post that others should be able to build on (and don't hit control-c just bc it's taking too long!)
klodolph 5y ago
These experiences are relatively common, we just don't see writeups that often. Kudos to the author for writing this up. I do disagree with some of the lessons here. You want to switch operating systems on your server? 1. Set up monitoring. If you don't have monitoring already, hack something together with simple scripts. 2. Start up a new server. 3. Migrate services and data from your existing server to your new server. Do this at whatever pace you feel is appropriate. Point the monitoring scripts at your new server. 4. Switch DNS records. 5. Wait. You are not in a hurry to turn off the previous server. Why not wait one or two months? 6. Turn off the old server. 7. Wait some more, and then delete the old server. The idea here is that steps which might take your site offline are easily reversible. For example, switching DNS records. It's trivial to switch the DNS records back if your migration unexpectedly failed. As much as reasonable, you want the ability to go backwards and undo the steps that you've done to get back to a known good setup. In particular, I would say that backups are usually not the right tool for migration. This is missing from the lessons at the bottom of the article. The way you get more confidence in your backups is by doing restore tests into a sandbox environment, by adding automated monitoring to your backups, etc. Trying to address a lack of confidence by increasing the backup frequency doesn't make sense. The backup frequency is the most trivial thing to adjust and doesn't address deeper issues, like the fact that you need to dump/restore databases properly and shouldn't copy files from a live database. These issues are discovered through restore testing. The saying goes, "Nobody wants a backup system, everyone wants a restore system." If you are making backups but not testing restores, you're gonna get bitten. Test the part of backups that you care about--the ability to restore data--and don't test it live. Test it in a sandbox.
- livueta 5y ago
  This approach saved my bacon on a migration just yesterday. I had gotten to testing files after having done a big baseline rsync, stopping file services on the old server, and doing a catch-up rsync incremental. Oh shit - one veracrypt container is corrupted and won't mount. Turns out rsync diff updates and mounted containers being written to don't play nice together. Since the old machine was still sitting there with all the data accessible, I was able to just blow away the corrupted volume, confirm it was unmounted on the source, then copy the whole thing over. If I just had a one-time copy that I'd thrown at b2 or something, I would have been very sad. So, yeah, test restoring your backups. Even fancy checksumming filesystems and shit won't save you from bad assumptions about the integrity of your data.
dmuth 5y ago
Someone tell me if I'm missing something, but isn't the whole point of hosting things in a virtual environment so that when you want to switch/upgrade OSes, you stand up a second server and start migrating apps over one at a time? I can't understand why that doesn't done here.
- plorkyeran 5y ago
  Well the title is "The Wrong Way to Switch Operating Systems on Your Server" so it's unsurprising that it's describing completely the wrong way to do things. The concerning part is that the author seems to have learned the wrong lessons from doing things the wrong way.
  - jaywalk 5y ago
    Yep, the correct title would be "There is no Right Way to Switch Operating Systems on Your Server" Just don't do it. Fire up a new one, and migrate everything over.
justin_oaks 5y ago
Don't consider it a backup until you've successfully restored the data from it. The first thing I do after setting up a new data backup is test a restore of the data. Only after that will I feel confident that the backup procedure works right. In the article author's case, an attempt to restore would have caught the problem of the missing .env files and the large movie files. As for the Ctrl-C on both the backup and restore, you should check your I/O (network and disk) before terminating a process. Doing that would have confirmed that the process was still going, and indicate the rate at which the process was going.
- spsesk117 5y ago
  +1. `strace` can be very helpful here, to see if a process is stuck waiting on something or whether it's just zooming along with no output.
  - dspillett 5y ago
    `progress` (https://github.com/Xfennec/progress https://github.com/Xfennec/progress) and similar can be very helpful too depending on the backup utilities being used (in my case often involving rsync) even if the processes normally have everything set to quiet so no progress information is automatically forthcoming.
    - spsesk117 5y ago
      Wow this looks great, didn't know about this tool. Thanks for the recommendation!
- geoduck14 5y ago
  > Don't consider it a backup until you've successfully restored the data from it. Ouch! I back up every 2 hours - should I REALLY restore from each of those?
  - xgbi 5y ago
    No, not ALL backups, just the first backup you do: you test that you can restore the data from it. Then you can be confident that following backups will be restorable. Corrolary: If you introduce a new folder for another service in your backup, make sure you can restore it too.
lawrenceduk 5y ago
This is a bit dumb and I feel like if as much effort had been made doing it as writing the blog post, a more positive outcome would have resulted.
teekert 5y ago
What a nightmare! I swear I had something similar with rsync on a mac once, I was very certain it finished, ran again and it reports it's done. I migrate and I'm missing all these files! Although it was probably my fault, I really don't trust rsync anymore... Maybe it had to do with HFS+ and those strange aperture libraries but man it ruined my day (week). Sure migrating is 100 times more relaxed when you have the old system running, but sometimes you need to reinstall. I had only one MacBook, now I only have 1 server. What you could do in that case is just install a new disc, unplug the old one until the new systems is running. It's worth the money and effort. I'm looking to install NixOS to my Home Server next week. All my personal infra is in Docker compose, on Ubuntu 20.04 atm. I only have one m.2 slot in the server and I don't want to buy a second drive just for this... So I'm sweating already. Maybe I should first migrate to my nuc, then back to the new server... hmmmm...
pinkythepig 5y ago
I feel like a lot of this article really should be about how bad tarsnap is. Defaults to no progress updates, has no builtin multithreading, has failures around large files, can't backup sym links properly, no builtin way to detect an in progress restore so you have to manually tell it to resume, etc. If tarsnap didn't have a bad UX, this entire article would have instead been 'the time I forgot to backup my .env file', none of the other issues would have occurred.
- Negitivefrags 5y ago
  I agree. Used to use tarsnap but the speed is garbage. It’s so bad that actually restoring anything of any size is just not viable.
juped 5y ago
if you get nervous and ctrl-c things with no output, try using -v mode
- fake-name 5y ago
  This is fine, except for the annoying tools that think they need to then print their software version, and exit.
wildmanx 5y ago
> I woke and the backup was finished! I wiped the VPS Just this single line made me scream in horror. Let me get this straight. He launched some backup command, it didn't output anything for hours, he suspected it hadn't done anything, aborted with ctrl-c, and then learned that he aborted it at 90%. Wipes the partial backup, starts again. After _that_ experience, he blindly trusts the result of _the same tool_, blindly wipes everything? Wtf.
- wildmanx 5y ago
  Spoiler: That's not even part of his "lessons learned". I know whom I won't hire for my company IT or devops or whatnot.
  - abraae 5y ago
    Brutal honesty can be a wonderful trait all of its own in a team however. We can assume the author is not a BS artist from his very candid self-evaluation. This is a good thing he should hang onto. "My haphazard strategy resulted in three days of stress and frustration as I clambered to restore a self-hosting empire that I myself had reduced to ash."
    - argomo 5y ago
      Plus he's able to reflect on his own mistakes and articulate what he learned from them. Would hire.
  - bluehatbrit 5y ago
    One of the articles they link off to about choosing a different provider explains that the author is a student. I think it's reasonable to assume they don't have much experience with this type of work. Hopefully they'll learn a lot from this process and maybe from the comments here as well. Maybe lets not write someone off before we understand more about their experience and background, especially after they shared an honest account of something they screwed up and learned from.
  - gperciva 5y ago
Shank 5y ago
Far and above, the best strategy is to spin up the new server, scp/rsync the data to the new server directly from the old server, and then boot services, and only decommission after you’ve moved all DNS over and confirmed the new site is working. Using Tarsnap for this is not only time consuming but needless unless you already have it setup and working.
- roywashere 5y ago
  Using tarsnap has one big advantage: it proves that you can recover from your backups. Using this method caused the OP to realize the backups were there but the secrets were missing! I agree with keeping the old server in place until the new one is working obv
- neurostimulant 5y ago
  My strategy is to setup the new server, and once everything is ready, stop the service on the old server and replace it with a reverse proxy to the new server, and THEN finally update the dns record. When the old server no longer see any meaningful traffic, I'll decommission the old server. This way I don't have to deal with both server out of sync.
deleted 5y ago
[deleted]
deleted 5y ago
[deleted]
fak3r 5y ago
For me: rsync > tarsnap I have a backup rsync script that parses a file I have that lists every path I want backed up. Yes, this considers dotfiles, so the poster's .env file would have been backed up. My script runs locally, backs up to my main (home) server, and then does another rsync to a 'cloud' server. Want to backup a new file or path? Add it to the manifest file. Adding another server or device? Build another manifest script, have rsync write to the same dir on the server, it'll automatically get sync'd to the cloud server too.
isatty 5y ago
Why tarsnap when wasabi or backblaze would be significantly cheaper? You can just encrypt by yourselves anyway. Also I run my own personal infra and here’s what I do: * treat servers as cattle, not pets. This is really important. Have mandatory reboots, never be afraid of reboots. * preferably do things with an automation method, I use ansible for n=5 but pick whatever you like * have SOME monitoring. It’s not too hard to throw up prom+grafana so get on it early. * VPN instead of securing internal services. Attack surface is way too high if you’ve too many services. Just throw them all behind a vpn and expose selectively. I use WireGuard. * personally: don’t self host critical infrastructure. I can’t afford downtime on email etc so I rather just pay someone to host that. Personal infra is for fun, not a second job (and I’m an SRE).
- MiscIdeaMaker99 5y ago
  > Have mandatory reboots, never be afraid of reboots. Yes! It is not a badge of honor to have a server that's been online for 365+ days -- it just means you haven't applied any kernel security updates for a long, long time. :-p
  - toast0 5y ago
    > Yes! It is not a badge of honor to have a server that's been online for 365+ days The badges come at 1000 days and every 500 after that. If you have a minimal kernel, and a limited software set, many kernel security bugs found won't apply to your system and you don't need to reboot for them. Not that you need to be scared of reboots, and you should probably schedule some reboots so you can be confident your systems can reboot, but you also don't need to be scared of uptime or stability, either.
    - addingnumbers 5y ago
      1000 days of uptime is 1000 days of configuration drift since the last time you confirmed your services still boot properly. I don't want to wait for a power failure to find out whether my services still recover from a cold boot.
ajnin 5y ago
My current backup strategy is to backup the whole filesystem. I run services in actual VMs, not docker containers, with disks mounted from LVM volumes which allows me to take snapshots and back them up live without needing to shut them down. I'm using bup to do the actual backups to a server I'm keeping in my home. I wrote a few custom scripts to backup and restore servers, and keep a history of the last x days, y weeks and z months. That way it gives me more time to figure out if something's wrong, as it's hobby stuff that I'm not checking every day. My advice for OP would be to 1/ ditch tarsnap. A backups tool that runs for hours without any feedback ? A restore tool that fails if the files are too big ? Everything extremely slow ? Just forget it. 2/ keep more than 3 days of backups. It's too short if you make a mistake, it took 3 days to recover from this one already. 3/ backup everything. Don't try to pick and choose files, you're likely to forget something, and if not now then some time later when you create a new file but forget to add it to the list of things to back up.
gperciva 5y ago
Wow, that's much more technically advanced than I was as a teenager! Way to go! To print progress with tarsnap 1.0.39, send it a SIGUSR1 or SIGINFO. On FreeBSD, you can do this by pressing ctrl-t. On Linux, you have to use the unfortunately-named `kill` or `killall` command, such as killall -SIGUSR1 tarsnap https://www.tarsnap.com/tips.html#check-current https://www.tarsnap.com/tips.html#check-current (Note that Tarsnap is not responsible for naming the unix `kill` or `killall` commands.) In the unreleased git version of tarsnap, there's a `--progress-bytes SIZE` command, which prints a progress message after every SIZE bytes are processed. As a general note: the tarsnap-users mailing list is a great place to ask for tips. As you mentioned in your lessons learned, some of the options could have helped a lot (such as `--recover`) https://www.tarsnap.com/lists.html https://www.tarsnap.com/lists.html (Disclaimer: I'm employed by Tarsnap Backup Inc.)
- bigiain 5y ago
  "employed by" <grin>
  - gjm11 5y ago
    Note that that's gperciva not cperciva. I believe the person's name is Graham Percival, and I am fairly sure Colin Percival is still called Colin Percival. They may well be related, but I'm pretty sure that's a bona fide "employed by" :-).
    - cperciva 5y ago
      Yes.
    - gperciva 5y ago
      Yes. For the curious, https://github.com/Tarsnap/tarsnap/graphs/contributors https://github.com/Tarsnap/tarsnap/graphs/contributors
geofft 5y ago
> My terminal sat empty for hours. There were no changes – the process was running, but there was no feedback. I was nervous. > What if it failed silently? How can I check? What should I do? On Linux, find the process ID and run e.g. ls -l /proc/12345/fd which will show you all the files currently open by the process. For something like a backup of a whole directory, or something generating a lot of output, run it again a few seconds later. If it's opened different files, then you know it's making progress and it's not stuck. If it's something that operates on a single file, find the number corresponding to that file in the list (the file descriptor), and run e.g. cat /proc/12345/fd/3 which will output a "pos" field showing the position in the file, in bytes. Compare it with the actual size of the file, and also run it again a few seconds later to see how fast it's making progress. (You can also use strace, but that slows down your program and potentially changes how it behaves in extremely unusual cases, so it isn't the first thing I'd reach for unless I really think the program is misbehaving and I want to see what it's doing in more detail. And there are tools like iostat too, but they're systemwide.)
- Dylan16807 5y ago
  Also there's the simple method of using iotop or a network equivalent.
ay 5y ago
The moment I see: - shutdown - backup + restore anew - bring up This triggers shivers. Nononono. - backup (while your current gif is running) - restore and test—verify (via a separate vhost) - delta-backup, delta-restore and test-verify - stop, delta-backup, delta restore, start at new place This way you minimize the amount of unexpected. Of course can be infeasible in some contexts…
simlevesque 5y ago
He forgot one mistake: never take down your existing infrastructure before the new one is up.
rekabis 5y ago
One big WTF in my mind is the default lack of feedback with tarsnap. The -v should be implicit and automatic, and should need a flag to turn OFF, not on. I mean, that’s just UI/UX 101. _Always_ provide feedback on progress, unless explicitly and intentionally silenced.
- wolrah 5y ago
  Tools that are intended to be used in scripts often have fairly quiet default modes. tar, cp, and many other *nix tools in that general realm tend to only speak up when something goes wrong, at least by default. That said, I'm on your side and I think quiet mode should be the option with a reasonable human-friendly verbosity being default. Or even better, some form of detection of if it's being run interactively versus in a script.
  - bombcar 5y ago
    Exactly - it’s very possible to detect of being run interactively and at least provide a hint - or even when Ctrl-C is hit (I know some programs will print the status on ctrl-c and tell you to hit it twice to kill it). At least when exiting and saying 90% give a hint how you could have seen thy without killing it.
  - deleted 5y ago
    [deleted]
ddtaylor 5y ago
I've not used tarsnap before but it doesn't seem to be worth the frustration being caused here. I think rsync works pretty well and it's end to end encrypted over SSH with multiple ways to perform compression.
siraben 5y ago
Without a sweat, I recently changed from Ubuntu to NixOS on a server I have access to only via SSH (so mounting an ISO wasn't an option) using[0], so the switch happens live(!) and when you reboot you end up in NixOS. I've also done it to CentOS 7.5, but manually via[1]. It was the case here that backups were not necessary, but it's pretty incredible what's possible when the system configuration is declarative and only /nix and /boot are needed by NixOS to boot. I highly recommend people new to running servers to try NixOS. [0] https://github.com/elitak/nixos-infect https://github.com/elitak/nixos-infect [1] https://nixos.org/manual/nixos/unstable/index.html#sec-installing-from-other-distro https://nixos.org/manual/nixos/unstable/index.html#sec-insta...
elondaits 5y ago
The problem with the conclusion is that he implies his error was to not be careful / smart or knowledgable enough… but his error was to not use a fool/error-proof method.
stavros 5y ago
I had exactly the same problems, and I decided to change not my backup strategy, but my deployment strategy. I wrote a small tool to deploy everything in a single directory, using Docker Compose: https://gitlab.com/stavros/harbormaster https://gitlab.com/stavros/harbormaster It allows you to separate important state (data) from non-important state (caches). This way, all you need to do is back up the data directory, and then you can restore the Harbormaster config file (along with the data) on the new server and you're done.
FpUser 5y ago
Every server of mine has "history" in a form of script that can rebuild it from the scratch on brand new clean machine. Every time anything is added / modified the script is ran on clean machine to test if all still works as supposed. My databases are all automatically backed to a different backup servers and are restored when needed by the same script. So for me setting up new server involves following steps: 1) Modify config portion of the script (basically change default values if needed) and run said script on a new server 2) Test 3) Put old server into "read only mode" 4) Run script again with the option to restore the database only. 5) Switch DNS There are/could be slight variations of course. Practically, I did recent migration of the new server for new product that was running on my location in Toronto to Hetzner. It took a "whopping" 15 minutes of which 13 been taken by restoring database from backup.
zrav 5y ago
Moving containers between hosts is not a use case that is well supported by docker. The tenor being that you should rather delete and recreate. While not a replacement for docker, with LXD moving containers between hosts is a single command. In many aspects I find LXD's UX more polished and its capabilities more complete, so I often do recommend it to people. Even if just to containerize docker itself to gain some of the missing functionality...
z3t4 5y ago
Tip: have your "data" on a separate file system mounted outside the operating system (OS) root. Then you can just remount it on the new OS

109 points by ohjeez 5y ago

On my server, I have three 'root' partitions. One for general day-to-day use, one for a backup if something catastrophic happens to my main system, and one for experimentation. The extra disk space taken up for my having two extra 'root' partitions is a miserable 40-50 gig. But I know that I can swap operating systems over almost instantaneously and then back again just as quickly if I did something wrong. Great peace of mind for practically no cost. The second error I see in the article is to use software that we haven't used previously for something important. My first wife had the habit of trying new recipes when we had a dinner-party. I tried to tell her repeatedly to try the recipe on us first, then she would have it down pat when she wanted to impress. The third error of course was the need to have restorable daily backups and the use of them to restore the system when need be, associated with modularity of the system. I back up my whole system daily. but the most important part of that is not backing up the distro itself (we have re-installs for that) but backing up all the config files, all the databases, all the local binaries, and a current list of all of the installed distro packages. I can replace the whole operating system from a complete wipe-out in less than two hours. I store these backups in a pseudo-exponential policy. I have more recent backups, fewer older backups. Currently I have 15 backups covering 9 years, with five of those covering just the last 3 weeks, and four covering the last five days. To augment this, I have a monthly snapshot backup also stashed away. The other stuff, personal docs etc, is deliberately kept small. Total daily backup of base system and /home is approximately 12 gigs. That is easily transported on a USB stick. I don't store music, photos, magazine .PDFs, old software, etc in my /home directory. That stuff all goes in an archive directory that's write-once, and store (practically) forever. That gets rsynced to two external USB drives daily. Most days, there's practically nothing that gets transferred out. Having several times lost much valuable data, I suppose I am really paranoid, but I still think I haven't been paranoid enough.

ghostly_s 5y ago

Spent three days fighting with this and didn't think to try the -v flag on his apparently-hanging process?

rhn_mk1 5y ago
The author may not have the necessary familiarity with it. Experience is gained via mistakes too.
fak3r 5y ago
My first thought, also always run long running jobs like this in Screen or Tmux! As it is, it's a good learning post that others should be able to build on (and don't hit control-c just bc it's taking too long!)

klodolph 5y ago

These experiences are relatively common, we just don't see writeups that often. Kudos to the author for writing this up. I do disagree with some of the lessons here. You want to switch operating systems on your server? 1. Set up monitoring. If you don't have monitoring already, hack something together with simple scripts. 2. Start up a new server. 3. Migrate services and data from your existing server to your new server. Do this at whatever pace you feel is appropriate. Point the monitoring scripts at your new server. 4. Switch DNS records. 5. Wait. You are not in a hurry to turn off the previous server. Why not wait one or two months? 6. Turn off the old server. 7. Wait some more, and then delete the old server. The idea here is that steps which might take your site offline are easily reversible. For example, switching DNS records. It's trivial to switch the DNS records back if your migration unexpectedly failed. As much as reasonable, you want the ability to go backwards and undo the steps that you've done to get back to a known good setup. In particular, I would say that backups are usually not the right tool for migration. This is missing from the lessons at the bottom of the article. The way you get more confidence in your backups is by doing restore tests into a sandbox environment, by adding automated monitoring to your backups, etc. Trying to address a lack of confidence by increasing the backup frequency doesn't make sense. The backup frequency is the most trivial thing to adjust and doesn't address deeper issues, like the fact that you need to dump/restore databases properly and shouldn't copy files from a live database. These issues are discovered through restore testing. The saying goes, "Nobody wants a backup system, everyone wants a restore system." If you are making backups but not testing restores, you're gonna get bitten. Test the part of backups that you care about--the ability to restore data--and don't test it live. Test it in a sandbox.

livueta 5y ago
This approach saved my bacon on a migration just yesterday. I had gotten to testing files after having done a big baseline rsync, stopping file services on the old server, and doing a catch-up rsync incremental. Oh shit - one veracrypt container is corrupted and won't mount. Turns out rsync diff updates and mounted containers being written to don't play nice together. Since the old machine was still sitting there with all the data accessible, I was able to just blow away the corrupted volume, confirm it was unmounted on the source, then copy the whole thing over. If I just had a one-time copy that I'd thrown at b2 or something, I would have been very sad. So, yeah, test restoring your backups. Even fancy checksumming filesystems and shit won't save you from bad assumptions about the integrity of your data.

dmuth 5y ago

Someone tell me if I'm missing something, but isn't the whole point of hosting things in a virtual environment so that when you want to switch/upgrade OSes, you stand up a second server and start migrating apps over one at a time? I can't understand why that doesn't done here.

plorkyeran 5y ago
Well the title is "The Wrong Way to Switch Operating Systems on Your Server" so it's unsurprising that it's describing completely the wrong way to do things. The concerning part is that the author seems to have learned the wrong lessons from doing things the wrong way.
- jaywalk 5y ago
  Yep, the correct title would be "There is no Right Way to Switch Operating Systems on Your Server" Just don't do it. Fire up a new one, and migrate everything over.

justin_oaks 5y ago

Don't consider it a backup until you've successfully restored the data from it. The first thing I do after setting up a new data backup is test a restore of the data. Only after that will I feel confident that the backup procedure works right. In the article author's case, an attempt to restore would have caught the problem of the missing .env files and the large movie files. As for the Ctrl-C on both the backup and restore, you should check your I/O (network and disk) before terminating a process. Doing that would have confirmed that the process was still going, and indicate the rate at which the process was going.

spsesk117 5y ago
+1. `strace` can be very helpful here, to see if a process is stuck waiting on something or whether it's just zooming along with no output.
- dspillett 5y ago
  `progress` (https://github.com/Xfennec/progress https://github.com/Xfennec/progress) and similar can be very helpful too depending on the backup utilities being used (in my case often involving rsync) even if the processes normally have everything set to quiet so no progress information is automatically forthcoming.
  - spsesk117 5y ago
    Wow this looks great, didn't know about this tool. Thanks for the recommendation!
geoduck14 5y ago
> Don't consider it a backup until you've successfully restored the data from it. Ouch! I back up every 2 hours - should I REALLY restore from each of those?
- xgbi 5y ago
  No, not ALL backups, just the first backup you do: you test that you can restore the data from it. Then you can be confident that following backups will be restorable. Corrolary: If you introduce a new folder for another service in your backup, make sure you can restore it too.

lawrenceduk 5y ago

This is a bit dumb and I feel like if as much effort had been made doing it as writing the blog post, a more positive outcome would have resulted.

teekert 5y ago

What a nightmare! I swear I had something similar with rsync on a mac once, I was very certain it finished, ran again and it reports it's done. I migrate and I'm missing all these files! Although it was probably my fault, I really don't trust rsync anymore... Maybe it had to do with HFS+ and those strange aperture libraries but man it ruined my day (week). Sure migrating is 100 times more relaxed when you have the old system running, but sometimes you need to reinstall. I had only one MacBook, now I only have 1 server. What you could do in that case is just install a new disc, unplug the old one until the new systems is running. It's worth the money and effort. I'm looking to install NixOS to my Home Server next week. All my personal infra is in Docker compose, on Ubuntu 20.04 atm. I only have one m.2 slot in the server and I don't want to buy a second drive just for this... So I'm sweating already. Maybe I should first migrate to my nuc, then back to the new server... hmmmm...

pinkythepig 5y ago

I feel like a lot of this article really should be about how bad tarsnap is. Defaults to no progress updates, has no builtin multithreading, has failures around large files, can't backup sym links properly, no builtin way to detect an in progress restore so you have to manually tell it to resume, etc. If tarsnap didn't have a bad UX, this entire article would have instead been 'the time I forgot to backup my .env file', none of the other issues would have occurred.

Negitivefrags 5y ago
I agree. Used to use tarsnap but the speed is garbage. It’s so bad that actually restoring anything of any size is just not viable.

juped 5y ago

if you get nervous and ctrl-c things with no output, try using -v mode

fake-name 5y ago
This is fine, except for the annoying tools that think they need to then print their software version, and exit.

wildmanx 5y ago

> I woke and the backup was finished! I wiped the VPS Just this single line made me scream in horror. Let me get this straight. He launched some backup command, it didn't output anything for hours, he suspected it hadn't done anything, aborted with ctrl-c, and then learned that he aborted it at 90%. Wipes the partial backup, starts again. After _that_ experience, he blindly trusts the result of _the same tool_, blindly wipes everything? Wtf.

wildmanx 5y ago
Spoiler: That's not even part of his "lessons learned". I know whom I won't hire for my company IT or devops or whatnot.
- abraae 5y ago
  Brutal honesty can be a wonderful trait all of its own in a team however. We can assume the author is not a BS artist from his very candid self-evaluation. This is a good thing he should hang onto. "My haphazard strategy resulted in three days of stress and frustration as I clambered to restore a self-hosting empire that I myself had reduced to ash."
  - argomo 5y ago
    Plus he's able to reflect on his own mistakes and articulate what he learned from them. Would hire.
- bluehatbrit 5y ago
  One of the articles they link off to about choosing a different provider explains that the author is a student. I think it's reasonable to assume they don't have much experience with this type of work. Hopefully they'll learn a lot from this process and maybe from the comments here as well. Maybe lets not write someone off before we understand more about their experience and background, especially after they shared an honest account of something they screwed up and learned from.
- gperciva 5y ago

Shank 5y ago

Far and above, the best strategy is to spin up the new server, scp/rsync the data to the new server directly from the old server, and then boot services, and only decommission after you’ve moved all DNS over and confirmed the new site is working. Using Tarsnap for this is not only time consuming but needless unless you already have it setup and working.

roywashere 5y ago
Using tarsnap has one big advantage: it proves that you can recover from your backups. Using this method caused the OP to realize the backups were there but the secrets were missing! I agree with keeping the old server in place until the new one is working obv
neurostimulant 5y ago
My strategy is to setup the new server, and once everything is ready, stop the service on the old server and replace it with a reverse proxy to the new server, and THEN finally update the dns record. When the old server no longer see any meaningful traffic, I'll decommission the old server. This way I don't have to deal with both server out of sync.

deleted 5y ago

[deleted]

deleted 5y ago

[deleted]

fak3r 5y ago

For me: rsync > tarsnap I have a backup rsync script that parses a file I have that lists every path I want backed up. Yes, this considers dotfiles, so the poster's .env file would have been backed up. My script runs locally, backs up to my main (home) server, and then does another rsync to a 'cloud' server. Want to backup a new file or path? Add it to the manifest file. Adding another server or device? Build another manifest script, have rsync write to the same dir on the server, it'll automatically get sync'd to the cloud server too.

isatty 5y ago

Why tarsnap when wasabi or backblaze would be significantly cheaper? You can just encrypt by yourselves anyway. Also I run my own personal infra and here’s what I do: * treat servers as cattle, not pets. This is really important. Have mandatory reboots, never be afraid of reboots. * preferably do things with an automation method, I use ansible for n=5 but pick whatever you like * have SOME monitoring. It’s not too hard to throw up prom+grafana so get on it early. * VPN instead of securing internal services. Attack surface is way too high if you’ve too many services. Just throw them all behind a vpn and expose selectively. I use WireGuard. * personally: don’t self host critical infrastructure. I can’t afford downtime on email etc so I rather just pay someone to host that. Personal infra is for fun, not a second job (and I’m an SRE).

MiscIdeaMaker99 5y ago
> Have mandatory reboots, never be afraid of reboots. Yes! It is not a badge of honor to have a server that's been online for 365+ days -- it just means you haven't applied any kernel security updates for a long, long time. :-p
- toast0 5y ago
  > Yes! It is not a badge of honor to have a server that's been online for 365+ days The badges come at 1000 days and every 500 after that. If you have a minimal kernel, and a limited software set, many kernel security bugs found won't apply to your system and you don't need to reboot for them. Not that you need to be scared of reboots, and you should probably schedule some reboots so you can be confident your systems can reboot, but you also don't need to be scared of uptime or stability, either.
  - addingnumbers 5y ago
    1000 days of uptime is 1000 days of configuration drift since the last time you confirmed your services still boot properly. I don't want to wait for a power failure to find out whether my services still recover from a cold boot.

ajnin 5y ago

My current backup strategy is to backup the whole filesystem. I run services in actual VMs, not docker containers, with disks mounted from LVM volumes which allows me to take snapshots and back them up live without needing to shut them down. I'm using bup to do the actual backups to a server I'm keeping in my home. I wrote a few custom scripts to backup and restore servers, and keep a history of the last x days, y weeks and z months. That way it gives me more time to figure out if something's wrong, as it's hobby stuff that I'm not checking every day. My advice for OP would be to 1/ ditch tarsnap. A backups tool that runs for hours without any feedback ? A restore tool that fails if the files are too big ? Everything extremely slow ? Just forget it. 2/ keep more than 3 days of backups. It's too short if you make a mistake, it took 3 days to recover from this one already. 3/ backup everything. Don't try to pick and choose files, you're likely to forget something, and if not now then some time later when you create a new file but forget to add it to the list of things to back up.

gperciva 5y ago

Wow, that's much more technically advanced than I was as a teenager! Way to go! To print progress with tarsnap 1.0.39, send it a SIGUSR1 or SIGINFO. On FreeBSD, you can do this by pressing ctrl-t. On Linux, you have to use the unfortunately-named `kill` or `killall` command, such as killall -SIGUSR1 tarsnap https://www.tarsnap.com/tips.html#check-current https://www.tarsnap.com/tips.html#check-current (Note that Tarsnap is not responsible for naming the unix `kill` or `killall` commands.) In the unreleased git version of tarsnap, there's a `--progress-bytes SIZE` command, which prints a progress message after every SIZE bytes are processed. As a general note: the tarsnap-users mailing list is a great place to ask for tips. As you mentioned in your lessons learned, some of the options could have helped a lot (such as `--recover`) https://www.tarsnap.com/lists.html https://www.tarsnap.com/lists.html (Disclaimer: I'm employed by Tarsnap Backup Inc.)

bigiain 5y ago
"employed by" <grin>
- gjm11 5y ago
  Note that that's gperciva not cperciva. I believe the person's name is Graham Percival, and I am fairly sure Colin Percival is still called Colin Percival. They may well be related, but I'm pretty sure that's a bona fide "employed by" :-).
  - cperciva 5y ago
    Yes.
  - gperciva 5y ago
    Yes. For the curious, https://github.com/Tarsnap/tarsnap/graphs/contributors https://github.com/Tarsnap/tarsnap/graphs/contributors

geofft 5y ago

> My terminal sat empty for hours. There were no changes – the process was running, but there was no feedback. I was nervous. > What if it failed silently? How can I check? What should I do? On Linux, find the process ID and run e.g. ls -l /proc/12345/fd which will show you all the files currently open by the process. For something like a backup of a whole directory, or something generating a lot of output, run it again a few seconds later. If it's opened different files, then you know it's making progress and it's not stuck. If it's something that operates on a single file, find the number corresponding to that file in the list (the file descriptor), and run e.g. cat /proc/12345/fd/3 which will output a "pos" field showing the position in the file, in bytes. Compare it with the actual size of the file, and also run it again a few seconds later to see how fast it's making progress. (You can also use strace, but that slows down your program and potentially changes how it behaves in extremely unusual cases, so it isn't the first thing I'd reach for unless I really think the program is misbehaving and I want to see what it's doing in more detail. And there are tools like iostat too, but they're systemwide.)

Dylan16807 5y ago
Also there's the simple method of using iotop or a network equivalent.

ay 5y ago

The moment I see: - shutdown - backup + restore anew - bring up This triggers shivers. Nononono. - backup (while your current gif is running) - restore and test—verify (via a separate vhost) - delta-backup, delta-restore and test-verify - stop, delta-backup, delta restore, start at new place This way you minimize the amount of unexpected. Of course can be infeasible in some contexts…

simlevesque 5y ago

He forgot one mistake: never take down your existing infrastructure before the new one is up.

rekabis 5y ago

One big WTF in my mind is the default lack of feedback with tarsnap. The -v should be implicit and automatic, and should need a flag to turn OFF, not on. I mean, that’s just UI/UX 101. _Always_ provide feedback on progress, unless explicitly and intentionally silenced.

wolrah 5y ago
Tools that are intended to be used in scripts often have fairly quiet default modes. tar, cp, and many other *nix tools in that general realm tend to only speak up when something goes wrong, at least by default. That said, I'm on your side and I think quiet mode should be the option with a reasonable human-friendly verbosity being default. Or even better, some form of detection of if it's being run interactively versus in a script.
- bombcar 5y ago
  Exactly - it’s very possible to detect of being run interactively and at least provide a hint - or even when Ctrl-C is hit (I know some programs will print the status on ctrl-c and tell you to hit it twice to kill it). At least when exiting and saying 90% give a hint how you could have seen thy without killing it.
- deleted 5y ago
  [deleted]

ddtaylor 5y ago

I've not used tarsnap before but it doesn't seem to be worth the frustration being caused here. I think rsync works pretty well and it's end to end encrypted over SSH with multiple ways to perform compression.

siraben 5y ago

Without a sweat, I recently changed from Ubuntu to NixOS on a server I have access to only via SSH (so mounting an ISO wasn't an option) using[0], so the switch happens live(!) and when you reboot you end up in NixOS. I've also done it to CentOS 7.5, but manually via[1]. It was the case here that backups were not necessary, but it's pretty incredible what's possible when the system configuration is declarative and only /nix and /boot are needed by NixOS to boot. I highly recommend people new to running servers to try NixOS. [0] https://github.com/elitak/nixos-infect https://github.com/elitak/nixos-infect [1] https://nixos.org/manual/nixos/unstable/index.html#sec-installing-from-other-distro https://nixos.org/manual/nixos/unstable/index.html#sec-insta...

elondaits 5y ago

The problem with the conclusion is that he implies his error was to not be careful / smart or knowledgable enough… but his error was to not use a fool/error-proof method.

stavros 5y ago

I had exactly the same problems, and I decided to change not my backup strategy, but my deployment strategy. I wrote a small tool to deploy everything in a single directory, using Docker Compose: https://gitlab.com/stavros/harbormaster https://gitlab.com/stavros/harbormaster It allows you to separate important state (data) from non-important state (caches). This way, all you need to do is back up the data directory, and then you can restore the Harbormaster config file (along with the data) on the new server and you're done.

FpUser 5y ago

Every server of mine has "history" in a form of script that can rebuild it from the scratch on brand new clean machine. Every time anything is added / modified the script is ran on clean machine to test if all still works as supposed. My databases are all automatically backed to a different backup servers and are restored when needed by the same script. So for me setting up new server involves following steps: 1) Modify config portion of the script (basically change default values if needed) and run said script on a new server 2) Test 3) Put old server into "read only mode" 4) Run script again with the option to restore the database only. 5) Switch DNS There are/could be slight variations of course. Practically, I did recent migration of the new server for new product that was running on my location in Toronto to Hetzner. It took a "whopping" 15 minutes of which 13 been taken by restoring database from backup.

zrav 5y ago

Moving containers between hosts is not a use case that is well supported by docker. The tenor being that you should rather delete and recreate. While not a replacement for docker, with LXD moving containers between hosts is a single command. In many aspects I find LXD's UX more polished and its capabilities more complete, so I often do recommend it to people. Even if just to containerize docker itself to gain some of the missing functionality...

z3t4 5y ago

Tip: have your "data" on a separate file system mounted outside the operating system (OS) root. Then you can just remount it on the new OS