4 ms·

Another great question is "When did you last try to restore from a backup?" which usually is answered with "It's the built-in tooling, why would we assume it's

• KingOfCoders 2y ago
  Agreed, if you haven't tested your backups recently (daily, automatic best), you don't have backups. Several of my clients (CTO Coaching) had problems in the past because they restored backups and where finding they were not complete (for various reasons).
  • w8whut 2y ago
    daily test restore is infeasible for anything but toy projects. You should periodically test your restore procedures, but its incredibly costly and time consuming for sizeable platforms. Its just not that easy to restore a 10+TB backup for example, and thats a _tiny_ backup size for a b2c product. they can easily go into the hundreds of TB, depending on your platform. and i might add: i vividly remember gitlabs article how they have had automated backups and test restores for years, but when they actually needed them... it turned out some data wasn't part of it after all. just because youre testing your restore procedure doesnt mean you've actually accomplished anything.
    • rodgerd 2y ago
      > but its incredibly costly and time consuming for sizeable platforms. If your restores are too time consuming to test regularly, they sure as shit aren't going to be useful in a disaster.
      • usrnm 2y ago
        Why is that? Hours or even days of downtime are still better than just losing all data. It's a simple cost-benefit analysis, and it's ok to pick different trade-offs depending on your use case
        • rodgerd 2y ago
          I have spent most of my career in newspaper publishing and banking. A newspaper that doesn't publish for a few days might recover. A bank that drops off the Swift network for days isn't a bank any more.
          • cyberax 2y ago
            Russia and Iran would like to have a few words with you. Banks regularly close for multiple days for bank holidays. Unscheduled downtime is a somewhat different story, though. Luckily, the traditional SWIFT/banking infrastructure is so negligible these days, my phone can host a classic banking infrastructure for an entire small country.
            • deleted 2y ago
              [deleted]
            • com 2y ago
              This is a nonsense response. A bank exists not as an isolated entitity, but as a node in a local, regional and global network of transactions. Your phone as a “classic banking infrastructure” (nonsense phrase) can’t do credit card acquiring or realtime transactions because it’s not connected to the payment rails, transaction switches and so on (like SWIFT but all of the other less global ones run by central banks and private entities). In developed societies, instant settlement for bank-bank transfers is the norm, and cash flow is dependent on that. Russia and Iran pay about 2-5% for above-board (non-sanctioned) cross border transactions due to their extra costs of not being in SWIFT and USD sanctions, and between 20-50% where physical middle-people are needed to move pallets of USD.
            • rodgerd 2y ago
              This is absolutely peak "confidently incorrect"; it's hilarious, but completely expected on this site.
      • prmoustache 2y ago
        It depends on the accepted downtime.
    • KingOfCoders 2y ago
      Have two backups, the most recent data and everything else. Archive data to different databases, e.g. only have the most recent 6 months in a production OLTP database. "daily test restore is infeasible for anything but toy projects. " It probably depends on what you call "toy" project. If you work for Google, yes I think everything is a toy project, and you're right. I only worked for ~$200M ARR/1M DAU businesses and restoring was no problem. From your point working for a FAANG business it's a toy project I can see that. But there are many more "toy projects" of this kind than FAANG companies. "10+TB backup for example, and thats a _tiny_ backup size for a b2c product." Sure.
• amyames 2y ago
  aws-cli will sync your s3 buckets to a local system. I’m doing that to linux, and then the Linux box is furthermore backed up with nakivo. Not my favorite but the price was okay and I can run the whole director on Linux, unlike all their other competitors. [veeam’s next major release 13 or 14 should do this in the next year or so too.] While nakivo backs up s3 buckets, NFS shares, and local file servers… to your point, I don’t trust it (or any other backup software I can’t extract and unpack the resulting backup by hand) as far as I can throw it. So I rsync or mirror it to a local Linux box with aws-cli and then back THAT up. I think you can do all this with windows stuff too but I don’t know it that well Additionally you can take servers that are linux vps’es and do the reverse: mirror THEIR content to an s3 bucket. You can also run minio open source/free on your fileserver and set up s3 to s3 sync. Cloudflare for example will ingest and replicate your minio server automatically and you can firewall it all off to their address ranges. It’s not free but it actually prices out favorably compared to veeam and nakivo if that’s all you need backed up.
• ambicapter 2y ago
  One thing I've never figured out is, what is the difference between backups and replication? And, does restoring from backups always mean losing more _recent_ data than replication?
  • rolph 2y ago
    replication is a snapshot of everything, at the time of file access. backup is a replacement of specified files required by a system recovery procedure. it may be a total image, or a collection of config, and dat files, that are daily bootup settings,
  • justusthane 2y ago
    Replication isn't a backup, because if you accidentally delete a file, and that deletion is immediately replicated, then you can't get that file back. Backups are a specific point in time.
    • mavhc 2y ago
      Backup is a useless word, it's too overloaded Are snapshots backups? snapshots on raid? snapshots on replicated disks?
      • ianburrell 2y ago
        Backup describes what they are for. There are many ways to do backups. The main thing is that backups are archive stored away from danger. There are different kinds of danger and different need for protection. Snapshots can be backups depending on where they are stored usually not if stored locally. For example, RDS snapshot is backup for database going down but not account being deleted or region destroyed. Generally, snapshots are way to make backups to more durable medium.
      • andrewaylett 2y ago
        A backup is something that will functionally replace the original should the original fail, regardless of how the original failed. For data, this means that the restore process is part of the backup. Snapshots are not backups. Snapshots on RAID are not backups. Snapshots on replicated disks are probably backups, so long as the disks being replicated to are not inside the same case/building/city/continent (pick your risk suitably) and you're not able to delete the snapshots from the machine hosting the originals. The second SIM in my phone provides a backup for my primary service provider, so long as I keep it activated. The torch in my pocket is a backup for the lighting in my house, so long as I keep it charged. My data in tarsnap is a backup, so long as I'm able to restore it. Which means data in tarsnap isn't a complete backup on its own: unless I'm able to recover the encryption key, I don't actually have a backup.
• sidewndr46 2y ago
  my dad told me about this customer that had a server that made automatic backups each Sunday night. The backup script would backup all the data then eject the tape so the manager could put it in the vault and rotate in the other one from the vault. When the hard drive failed, they restored the customer to the latest backup. Which was the tape still sitting in the tape drive in the server. It was from the first Sunday night after the system was installed years ago
  • Terr_ 2y ago
    I'm confused, it sounds like you're saying the same tape was being ejected every week and then reinserted without any rotation. But in that case, shouldn't the weekly backup process have failed because the tape was full? Was nobody getting those alerts? Or do you mean the backup process was fine, but they restored from the wrong media, a very old tape that was about to be overwritten, instead of retrieving the one with last-week's copy?
    • ender341341 2y ago
      I read it as they were saying the manual part of the process never happened, so the backup from the first week was just sitting ejected forever and they had no alarming to notify them that the new backup failed to write to tape.
    • sidewndr46 2y ago
      no, the first Sunday night after the backup process completed. It ejected the tape. It sat there for years until someone realized they needed to restore from the most recent backup. Since a new tape was never inserted, the backup was from years ago.
      • deleted 2y ago
        [deleted]
      • cutemonster 2y ago
        What happened afterwards? Was that manager still at the company, at that time?
• remram 2y ago
  This famously happened at GitLab: https://about.gitlab.com/blog/2017/02/01/gitlab-dot-com-database-incident/ https://about.gitlab.com/blog/2017/02/01/gitlab-dot-com-data... > Regular backups seem to also only be taken once per 24 hours, though team-member-1 has not yet been able to figure out where they are stored. According to team-member-2 these don’t appear to be working, producing files only a few bytes in size.
• dawnerd 2y ago
  In addition to making sure it works, you should make sure you know how to deal with restoring. Sure running a command is easy but what about spinning up new infra? What if it’s corrupted? What if the one person that knows the setup is gone or asleep? Mainly a problem for smaller teams that don’t have the redundancy or resources - they really need to make sure there’s at least docs on how stuff was setup. Reminds me I need to do my yearly checkup too.
• bazzargh 2y ago
  A fun one like that, a few years back we had some code using dynamodb that used the automatic point-in-time backups. I asked if it had been tested, need you guess the reply? Of course it turns out that the restore can only happen to a _new database name_ not the original, and the code had in multiple places hardcoded the assumption of what the db was called. So restoring also involved patching the code and rolling that out; you can't "roll back" because to roll back the db the code must roll forward.
  • banku_brougham 2y ago
    'Roll-middle-out'
  • jamesfinlayson 2y ago
    Ugh, very annoying limitation. I've written scripts to create dummy tables, restore backups and sync across to existing tables twice now.
• kstrauser 2y ago
  We've avoided that in various shops by making backups/restores part of regular maintenance processes. How do we upgrade the database? By stopping it, backing it up, restoring that to the new server, pointing all code at the new DB, then turning off the old server. As with code deployment, it's not so scary when it's something you do so frequently that it's just a little script you run.
  • diggan 2y ago
    > it's not so scary when it's something you do so frequently Yeah, I've found this to be the trick for ongoing hassle-free maintenance too. Make tearing stuff down and up frequent enough and you'll feel confident and safe when you're required to do so to recover from something. Scariest are applications/services/servers that has been running for years but never restarted nor ever restored. Those scare me.
    • kstrauser 2y ago
      Cattle, not kittens. My favorite thing about deploying containerized apps? They’re completely fungible and I never have to care about an individual instance. Oh, it hung due to some weird network interaction? Spawn a new one, then come back to see what went wrong with this one before you kill it.
• belter 2y ago
  "Amateurs backup. Professionals restore."
• auspex 2y ago
  You have to test restoration as part of SOC 2, so most companies with real customers do it at least once a year.