15 ms·

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I fee

The classic "lp0 on fire" error message comes to mind: https://en.wikipedia.org/wiki/Lp0_on_fire https://en.wikipedia.org/wiki/Lp0_on_fire Really though, I feel truly awful for anyone affected by this. The post recommends implementing a disaster recovery plan. The truth is that most people don't have one. So, let's use this post to talk about Disaster Recovery Plans! Mine: I have 5 servers at OVH (not at SBG) and they all back up to Amazon S3 or Backblaze B2, and I also have a dedicated server (also OVH/Kimsufi) that gets the backups. I can redeploy in less than a day on fresh hardware, and that's good enough for my purposes. What's YOUR Disaster Recovery Plan?

tialaramex 6y ago
If you are a corporate entity of some kind, the final layer of your plan should always be "Go bankrupt". You can't successfully recover from every possible disaster and you shouldn't try to. In the event of a sufficiently unlikely event, your business fails and every penny spent attempting the impossible will be wasted, move on and let professional administrators salvage what they can for your creditors. Lots of people plan for specific elements they can imagine and forget other equally or even more important things they are going to need in a disaster. Check out how many organisations that doubtless have 24/7 IT support in case a web server goes down somehow had no plan for what happens if it's unsafe for their 500 call centre employees to sit in tiny cubicles answering phones all day even though pandemic respiratory viruses are so famously likely that Gates listed them consistently as the #1 threat.
- jfrunyon 6y ago
  IMHO, the part they had no plan for was being unable to just require their employees to come in anyway...
  - osmano807 6y ago
    Just lobby the government to put call centers in "essential services". In my state they are open even with a partial lockdown.
  - tialaramex 6y ago
    The more insecure your workers, the easier it is to get them to come in, regardless of what the supposed rules may or may not be. Fast Fashion for example often employs workers in more or less sweatshop conditions close to the customers (this makes commercial sense, if you make the hot new items in Bangladesh you either need to expensively air freight them to customers or they're going to take weeks to arrive after they're first ordered - there's a reason it isn't called "Slow fashion"). These jobs are poorly paid, many workers have dubious right-to-work status, weak local language skills, may even be paid in cash - and so if you tell them they must come in, none of them are going to say "No". In fact the slackening off in R for the area where my sister lives (today the towering chimneys and cavernous brick factories are just for tourists, your new dress was made in an anonymous single story building on an industrial estate) might be driven more by people not needing to own new frocks every week when they've been no further than their kitchen in a month than because it would actually be illegal to staff their business - if nobody's buying what you make then suddenly it makes sense to take a handout from the government and actually shut rather than pretend making mauve turtleneck sweaters or whatever is "essential".
    - Mauricebranagh 6y ago
      For non uk residents "Frock" is a regional term for dress - quite common in the Midlands.
    - namibj 6y ago
      Just to clarify: trans-atlantic shipments take a week port-to-port, e.g. Newark, NJ, USA to Antwerp, Belgium. (Bangladesh to Italy via Suez-channel looks like a 2-week voyage, or 3 weeks to the US west coast. Especially the latter would probably have quite a few stops on the way along the Asian coast.) You get better economics than shipping via air-freight from one full pallet and up. Overland truck transport to and from the port is still cheaper than air freight, at least in the US and central Europe. For these major routes, there are typically at least bi-weekly voyages scheduled, so for this kind of distance, you can expect about 11 days pretty uniformly distributed +-2 days, if you pay to get on the next ship. This may lead to (committing to) paying for the spot on the ship when your pallet is ready for pickup at the factory, not when it arrives at the port) and use low-delay overland trucking services. Which operate e.g. in lockstep with the port processing to get your pallet on the move within half a day of the container being unloaded from the ship, ideally having containers pre-sorted at the origin to match truck routes at the destination. So they can go on a trailer directly from the ship and rotate drivers on the delivery tour, spending only a few minutes at each drop-off. Because those can't rely on customers to be there and get you unloaded in less than 5 minutes, they need locations they can unload at with on-board equipment. They'd notify the customer with a GPS-based ETA display, so the customer can be ready and immediately move the delivery inside. Rely on 360-degree "dashcam" coverage and encourage the customer to have the drop-off point under video surveillance, just to easily handle potential disputes. Have the delivery person use some suitable high-res camera with a built-in light to get some full-surface-coverage photographic evidence of the condition it was delivered in. I'd guess with a hydraulic lift on the trailer's back and some kind of folding manual pallet jack stuck on that (fold-up) lift, so they drive up to the location, unlock the pallet jack, un-fold the lift, lower the lift almost to the ground, detach the pallet jack to drop it the last inch/few cm to the ground, pull the jack out, lower the lift the rest of the way, drive it on to the lift, open the container, get up with the pallet jack, drive the pallets (one-by-one) for this drop-off out of the container and leave them on the ground, close and lock the container, re-arm the jack's hooks, shove it jack back under the slightly-lowered folding lift, make it hook back in, fold it up, lock the hooking mechanism (against theft at a rest stop (short meal and toilet breaks exist, but showering can be delayed for the up to 2 nights)), fold it all the way up, and go on to drive to their next drop-off point.
- jacquesm 6y ago
  The final layer is call the insurance company.
  - corty 6y ago
    Not really, the insurance won't make things right in an instant. They will usually compensate you financially, but often only after painstaking evaluation of all circumstances, weighing their chances in court to get out of paying you and maybe a lengthy court battle and a race against your bankruptcy. So yes, getting insurance can be a good idea to offset some losses you may have, as long as they are somewhat limited compared to your companies overall assets and income. But as soon as the insurance payout matches a significant part of your net worth, the insurance might not save you.
    - jacquesm 6y ago
      Fair enough.
  - tomatocracy 6y ago
    There are always uninsurable events and for large enough companies/risks there are also liquidity limits to the size of coverage you can get from the market even for insurable events. As such, it makes sense to make the level of risk you plan to accept (by not being insured against it and not mitigating) a conscious economic decision rather than pretending you've covered everything.
    - jacquesm 6y ago
      As long as you have outside shareholders you can decide that. If you do you'd be surprised about how they will respond to an attitude like that. After all: you can decide the levels of risk that you personally are comfortable with leading to extinguishing of the business, but a typical shareholder is looking at you to protect their investment and not insuring against a known risk which at some point in time materializes is an excellent way to find yourself in the crosshairs of a minority shareholder lawsuit against a (former) company executive.
- brmgb 6y ago
  "Go bankrupt" is not a plan. Becoming insolvent might be the end result of a situation but it's not going to help you deal with it. Let's take an example which might lead to bankruptcy. A typical answer to a major disaster (let's say your main and sole building burning as a typical case) for an SME would be to cease activity, furlough employes and stop or defer every payments you can while you claim insurance and assess your options. Well, none of these things are obvious to do especially if all your archive and documents just burnt. If you think about it (which you should), you will quickly realise that you at least need a way to contact all your employes, your bank and your counsel (which would most likely be the accountant certifying your results rather than a lawyer if you are an SME in my country) offsite. That's the heart of disaster planning: having solutions at the ready for what was easy to foresee so you can better focus on what wasn't.
  - Twisell 6y ago
    Hence the "final layer" statement. Bankruptcy when dealt with correctly is a process not an end. If everything else fail it's better to fill for bankruptcy when there is still something to recover with help of others than to burn everything to ashes because of your vanity. At least that's how I understood parent's comment.
    - brmgb 6y ago
      > Bankruptcy when dealt with correctly is a process not an end. Yes, that's why "Go bankrupt" is not a plan which was the entire point of my reply. That's like saying that your disaster recovery plan is "solve the disaster".
    - Sanzig 6y ago
      As a quick interlude, since this may be confusing to non-US readers: bankruptcy in the United States in the context of business usually refers to two concepts, whereas in many other countries it refers to just one. There are two types of bankruptcies in the US used most often by insolvent businesses: Chapter 7, and Chapter 11. A Chapter 7 bankruptcy is what most people in other countries think of when they hear "bankruptcy" - it's the total dissolution of a business and liquidiation of its assets to satisfy its creditors. A business does not survive a Chapter 7. This is often referred to as a "bankruptcy" or "liquidation" in other countries. A Chapter 11 bankruptcy, on the other hand, is a process by which a business is given court protection from its creditors and allowed to restructure. If the creditors are satisfied with the reorganisation plan (which may include agreeing to change the terms of outstanding debts), the business emerges from Chapter 11 protection and is allowed to continue operating. Otherwise, if an agreement can't be reached, the business may end up in Chapter 7 and get liquidated. Most countries have an equivalent to a Chapter 11, but the name for it varies widely. For example, Canada calls it a "Division 1 Proposal," Australia and the UK call it "administation," and Ireland calls it "examinership." Since there's a lot of international visitors to HN I just thought I'd jump in and provide a bit of clarity so we can all ensure we're using the same definition of "bankruptcy." A US Chapter 7 bankruptcy is not a plan, it's the game over state. A US Chapter 11 bankruptcy, on the other hand, can definitely be a strategic maneuver when you're in serious trouble, so it can be part of the plan (hopefully far down the list).
- physicsguy 6y ago
  It's not quite that simple, the data you might have may be needed for compliance or regulatory reasons. Having no backup strategy might make you personally liable depending on the country!
ahmedalsudani 6y ago
My disaster recovery plan: we shall rebuild.
- edoceo 6y ago
  F
- julianwachholz 6y ago
  Finally we get to rewrite everything from scratch!
  - philpem 6y ago
    The spice will flow, and the tech debt will go!
kijin 6y ago
One of my backup servers used to be in the same datacenter as the primary server. I only recently moved it to a different host. It's still in the same city, though, so I'm considering other options. I'm not a big fan of just-make-a-tarball-of-everything-and-upload-it-to-the-cloud backup methodology, I prefer something a bit more incremental. But with Backblaze B2 being so cheap, I might as well just upload tarballs to B2. As long as I have the data, the servers can be redeployed in a couple of hours at most. The SBG fire illustrates the importance of geographical redundancy. Just because the datacenters have different numbers at the end doesn't mean that they won't fail at the same time. Apart from a large fire or power outage, there are lots of things that can take out several datacenters in close vicinity at the same time, such as hurricanes and earthquakes.
- sebmellen 6y ago
  Duplicity is your best bet for incremental backups using B2. I use this for my personal server and it works brilliantly.
  - gingerlime 6y ago
    I thought so too for a long while. Until I was trying to restore something (just to test things), and wasn’t able to... it might have been specific to our GPG or an older version or something... but I decided to switch to restic and am much happier now. Restic has a single binary that takes care of everything. It feels more modern and seems to work really well. Never had any issue restoring from it. Just one data point. Stick to whatever works for you. But important to test not only your backups, but also restores!
    - sebmellen 6y ago
      Interesting, I will check Restic out, I’ve heard other good things about it. Duplicity is a bit of a pain to set up and Restic’s single binary model is more straightforward (Go is a miracle). Thanks for the recommendation! GPG is a bit quirky but I do regularly check my backups and restores (if once every few months counts as regular).
    - 6y ago
tothrowaway 6y ago
I'm at OVH as well (in the BHS datacenter, fortunately). I run my entire production system on one beefy machine. The apps and database are replicated to a backup machine hosted with Hetzner (in their Germany datacenter). I also run a tiny VM at OVH which proxies all traffic to Hetzner. I use a failover IP to point at the big rig at OVH. If the main machine fails, I move the failover IP to the VM, which sends all traffic to Hetzner. If OVH is totally down, and the fail over IP doesn't work, I have a fairly low TTL on the DNS. I backup the database state to S3 every day. Since I'm truly paranoid, I have an Intel NUC at my house that also replicates the DB. I like knowing that I have a complete backup of my entire business within arm's reach.
- warrenm 6y ago
  Yet another cheer for Hetzner!
- AmericanChopper 6y ago
  > I like knowing that I have a complete backup of my entire business within arm's reach. It could also provide a burglar a fantastic opportunity to pivot into career in data breaches.
  - baggy_trough 6y ago
    It all depends on your paranoia level of data hacking burglars vs. vaporized data centers.
  - tgsovlerkhgsel 6y ago
    This problem is usually solved through encryption.
    - AmericanChopper 6y ago
      If I were to ask my CISO if I was allowed to bring the production database home, I’m pretty sure his answer wouldn’t be “as long as you encrypt it”.
ethanpil 6y ago
Does anyone have experience with lvarchive or FSArchiver (or similar) to backup images of live systems instead of file based backup solutions?
jbeales 6y ago
My recovery plan: tarball & upload to Object Store. I'm going to check out exactly how much replication the OVH object store offers, and see about adding a second geographic location, and maybe even a second provider, tomorrow. (My servers aren't in SBG either - phew!)
- nucleardog 6y ago
  If your primary data is on OVH, I'd look at using another company's object store if feasible (S3, B2, etc). If possible, on another payment method. (If you want to be really paranoid, something issued under another legal entity.) There's a whole class of (mostly non-technical) risks that you solve for when you do this. If anything happens with your payment method (fails and you don't notice in time; all accounts frozen for investigation), OVH account (hacked, suspended), OVH itself (sudden bankruptcy?), etc, then at least you have _one_ other copy. It's not stuff that's likely to happen, but the cost of planning for it at least as far as "haven't completely lost all my data even if it's going to be a pain to restore" here is relatively minimal.
juangacovas 6y ago
Less than a day for disaster recovery on fresh hardware? Same as my case. As you say, good enough for most purposes, but I'm also looking for improvement. I have offsite realtime replicas for data and mariaDBs, and offsite nightly backups (combo of rsnapshot, lsyncd, mariaDB multi-source replication, and a post-new-install script that setups almost everything in case you have to recover on bare-metal, i.e. no available VM snapshots). Currently trying to reduce that "less than a day" though. Recently discovered "ReaR" (Relax and Recover) from RedHat and sounds really nice for bare-metal servers. Not everybody runs on virtualized/cloud (being able to recover from VM snapshots is really a plus). Let's share experiencies :)
sparrish 6y ago
Got burned once (no pun intended), learned my lesson. Hot spare on a different continent with replicated data along with a third box just for backups. The backup box gets offsite backups held in a safe with another redundant copy in another site in another safe. Restores are tested quarterly. Keep backups of backups. Once bitten, twice shy.
- gameshot911 6y ago
  Have you been bitten, personally? If so, story time?
- Twirrim 6y ago
  > Hot spare on a different continent Just be cautious about data locality laws (not likely to affect you as joe average, more for businesses)
  - bongoman37 6y ago
    yeah, there's lots of countries with regulations that certain data can't leave the geographical boundary of the country. Often, it is the most sensitive data.
    - mike_d 6y ago
      These laws generally don't work how people think they do. For example, the Russian data residency law states that a copy of the data must be stored domestically, not that it can't be replicated outside the country. The UAE has poorly written laws that have different regulations for different types of data - including fun stuff like only being subject to specific requirements if the data enters a 270 acre business park in Dubai. Don't even get me started on storing encrypted data in one country and the keys in another...
  - Mauricebranagh 6y ago
    A few years ago I worked on the British Telecom Worldwide intranet team and we had a matrix mapping various countries encryption laws. This was so we remained legal in all of the countries BT worked in which required a lot of behind the scenes work to make sure we didn't serve "illegaly encypted" Data.
fy20 6y ago
I have three servers (1 OVH - different location, 2 DO). The only thing I backup is the DB, which is synced daily to S3. There's a rule to automatically delete files after 30 days to handle GDPR and stop the bucket and costs spiralling out of control. Everything is managed with Ansible and Terraform (on DO side), so I could probably get everything back up and running in less than an hour if needed.
- sverhagen 6y ago
  > probably That makes it sound like you didn't try/practice. I imagine that in a real-life scenario things will be a little more painful than in one's imagination.
  - dylan604 6y ago
    Exactly. Having a plan is only part of it. Good disaster plans do dry runs a couple of times a year (when time changes is always convenient reminder). If you rehearse the recovery when you're not panicked, you have a better chance of not skipping a step when the timing is much more crucial. Also, some sort of guide with steps given procedurally is a great idea.
    - slaymaker1907 6y ago
      I don't think this is necessarily true for all parts of a disaster plan. Some mechanisms may be untestable because it is unknown how to actually trigger it (think certain runtime assertions, but on a larger scale). Even if it possible to trigger and test, actually using the recovery mechanism may have some high cost either monetarily or maybe losing some small amount of data. These mechanisms should almost always be an additional layer of defense and only be invoked in case of true catastrophe. In both cases, the mechanisms should be tested as thoroughly as possibly, either through artificial environments that can simulate improbable scenarios or in the latter case on a small test environment to minimize cost.
  - fy20 6y ago
    I haven't ever deleted everything and timed how long I could get it up and running again, but I have tested it works by spinning up new machines and moving everything over to there (it was easier than running "sudo apt-get dist-upgrade").
dfsegoat 6y ago
We test rolling over the entire stack to another AWS DR region (just one we dont normally use) from S3 backups, etc. We do this annually and try to introduce some variations to the scenarios. It takes us about 18 hours realistically. Documentation / SOPs that have been tested thoroughly by various team members are really important. It helps work out any kinks in interpretation, syntax errors etc. It does feel a little ridiculous at the time for all the effort involved, but incidents like this show why it's so important.
jfrunyon 6y ago
Servers are at a mix of "cloud" providers, and on-site. Most data (including system configs!) is backed up on-site nightly, and to B2 nightly with historical copies - and critical data is also live-replicated to our international branches. (Some "meh" data is backed up only to B2, like our phone logs; we can get most of the info from our carrier anyway). Our goal and the reason we have a lot of stuff backed up on-prem is to have our most time-critical operations back up within a couple of hours - unless the building is destroyed, in which case that's a moot point and we'll take what we can get. A dev wiped our almost-monolithic sales/manufacturing/billing/etc MySQL database a month or two ago. (I have been repeatedly overruled on the topic of taking access to prod away from devs) We were down for around an hour. Most of that time was spent pulling gigs of data out of the binlog without also wiping it all again. Because our nightly backups had failed a couple weeks prior - after our most recent monthly "glance at it".
el-salvador 6y ago
Pictures from the fire: https://www.dna.fr/amp/faits-divers-justice/2021/03/10/strasbourg-important-incendie-dans-une-entreprise-situee-sur-un-site-seveso-au-port-du-rhin https://www.dna.fr/amp/faits-divers-justice/2021/03/10/stras...
- pulse7 6y ago
  Uff... it looks like half of the containers on this picture were on fire...
  - deleted 6y ago
    [deleted]
  - jannes 6y ago
    Are you making a joke about docker containers or am I missing something?
    - dgellow 6y ago
      Part of OVH datacenters are literal, physical containers with racks, power supply, vents, etc. You can see more details here: https://baxtel.com/data-center/ovh-strasbourg-campus https://baxtel.com/data-center/ovh-strasbourg-campus
      - Mauricebranagh 6y ago
        Do the containers have fire suppressant systems installed?
- simonke 6y ago
  Without AMP: https://www.dna.fr/faits-divers-justice/2021/03/10/strasbourg-important-incendie-dans-une-entreprise-situee-sur-un-site-seveso-au-port-du-rhin https://www.dna.fr/faits-divers-justice/2021/03/10/strasbour...
- jacquesm 6y ago
  Nobody hurt. That's a bit of good news.
jacquesm 6y ago
It's true: most companies do not have a disaster recovery plan, and many of them confuse a breach protocol with a disaster recovery plan ('we have backups'). Fires in DCs aren't rare at all, I know of at least three, one of those in a building where I had servers. This one seems to be worse than the other two. Datacenters tend to concentrate a lot of flammable stuff, throws a ton of current through them and does so 24x7. The risk of a fire is definitely not imaginary, which is why most DCs have fire suppression mechanisms. Whether those work as advertised depends on the nature of the fire. An exploding on prem transformer took out a good chunk of EV1's datacenter in the early 2000's, and it wasn't so much the fire that caused problems for their customers, but the fact that someone got injured (or even died, I don't recall exactly), and before the investigation was completed and the DC released to the owners again took a long time. Being paranoid and having off-site backups is what allowed us to be back online before the fire was out. If not for that I don't know if our company would have survived.
lgeorget 6y ago
We have two servers at OVH (RBX and GRA, not SBG). I make backups of all containers and VMs every day and keep the last three, plus one one each month. Backups are stored in a separate OVH storage disk and also downloaded to a NAS on-premise. In case of a disaster, we'd have to rent a new server, reprovision the VMs and containers and restore the backups. About two days of work to make sure everything works fine and we could lose about 24 hours of data. It's not the best in terms of Disaster Recovery Plan but we accept that level of risk.
Cthulhu_ 6y ago
Personal: I run a webserver for some website (wordpress + xenforo), I've set up a cronjob that creates a backup of /var/www, /etc and a mysql database dump, then uploads it to an S3 bucket (with automatic Glacier archiving after X period set up). It should be fairly straightforward to rent a new server and set things back up. I still dislike having to set up a webserver + php manually though, I don't get why that hasn't been streamlined yet. My employer has a single rack of servers at HQ. It's positioned at a very specific angle with an AC unit facing it, their exact positions are marked out on the floor in tape. The servers contain VMs that most employees work on, our git repository, issue trackers, and probably customer admin as well. They say they do off-site backups, but honestly, when (not if) that thing goes it'll be a pretty serious impact on the business. They don't like people keeping their code on their take-home laptop either (I can't fathom how my colleagues work and how they can stand working in a large codebase using barebones vim over ssh), but I've employed some professional disobedience there.
- catbuttes 6y ago
  Have you considered writing an ansible playbook to set all that up? You could even have it pull down the backup and do a full restore for you...
walrus01 6y ago
Video of the aftermath: https://twitter.com/PompiersFR/status/1369544801817944071 https://twitter.com/PompiersFR/status/1369544801817944071 https://twitter.com/abonin_DNA https://twitter.com/abonin_DNA
shog_hn 6y ago
Self-hosted Kuberenetes and a FreeNAS storage system at home, and a couple of VMs in the cloud. I've got a mixed strategy, but it covers everything to remote locations. I use S3 API compatible object storage platforms for remote backup. E.g. BackBlaze B2. I wrote about my backup scripts for FreeNAS (jail that runs s3cmd to copy files to B2) here: https://www.shogan.co.uk/cloud-2/cheap-s3-cloud-backup-with-backblaze-b2/ https://www.shogan.co.uk/cloud-2/cheap-s3-cloud-backup-with-... For Kuberenetes I use velero which can be configured with an S3 storage backend target: https://www.shogan.co.uk/kubernetes/kubernetes-backup-on-raspberry-pi/ https://www.shogan.co.uk/kubernetes/kubernetes-backup-on-ras...
KronisLV 6y ago
Here's what i do for my homelab setup that has a few machines running locally and some VPSes "in the cloud": I personally have almost all of the software running in containers with an orchestrator on top (Docker Swarm in my case, others may also use Nomad, Kubernetes or something else). That way, rescheduling services on different nodes becomes less of a hassle in case of any one of them failing, since i know what should be running and what configuration i expect it to have, as well as what data needs to be persisted. At the moment i'm using Time4VPS ( affiliate link: https://www.time4vps.com/?affid=5294 https://www.time4vps.com/?affid=5294 ) for the stuff that needs decent availability and because they're cheaper than almost all of the alternatives i've looked at (DigitalOcean, Vultr, Scaleway, AWS, Azure) and that matters to me. Now, in case the entire data centre disappears, all of my data would still be available on a few HDDs under my desk (which are then replicated to other HDDs with rsync locally), given that i use BackupPC for incremental scheduled backups with rsync: https://backuppc.github.io/backuppc/ https://backuppc.github.io/backuppc/ For simplicity, the containers also use bind mounts, so all of the data is readable directly from the file system, for example, under /docker (not really following some of the *nix file system layout practices, but this works for me because it's really easy to tell where the data that i want is). I actually had to migrate over to a new node a while back, took around 30 minutes in total (updating DNS records included). Ansible can also really help with configuring new nodes. I'm not saying that my setup would work for most people or even anything past startups, but it seems sufficient for my homelab/VPS needs. My conclusions: - containers are pretty useful for reproducing software across servers - knowing exactly which data you want to preserve (such as /var/lib/postgresql/data/pgdata) is also pretty useful, even though a lot of software doesn't really play nicely with the idea - backups and incremental backups are pretty doable even without relying on a particular platform's offerings, BackupPC is more than competent and buying HDDs is far more cost effective than renting that space - automatic failover (both DNS and moving the data to a new node) seems complicated, as does using distributed file systems; those are probably useful but far beyond what i actually want to spend time on in my homelab - you should still check your backups
vbsteven 6y ago
In my case: * All my services are dockerized and have gitlab pipelines to deploy on a kubernetes cluster (RKE/K3s/baremetal-k8s) * git repo's containing the build scripts/pipelines are replicated on my gitlab instance and multiple work computers (laptop & desktop) * Data and databases are regularly dumped and stored in S3 and my home server * Most of the infrastructure setup (AWS/DO/Azure, installing kubernetes) is in Terraform git repositories. And a bit of Ansible for some older projects. Because of the above, if anything happens all I need to restore a service is a fresh blank VM/dedicated machine or a cloud account with a hosted Kubernetes offering. From there it's just configuring terraform/ansible variables with the new hosts and executing the scripts.
- jrib 6y ago
  How often do you test starting from a clean slate?
  - kuschku 6y ago
    I have a similar setup: I recreate everything for every major kubernetes update.
neurostimulant 6y ago
Nothing too crazy, just a simple daily cron to see sync user data and database dumps on our OVH boxes to backblaze and rsync.net. This simple setup is already saved our asses a few times already.
cm2187 6y ago
Also stupid things not to forget: make sure your dns provider is independent otherwise you won’t be able to point to your new server (or have a secondary DNS provider). Make sure any email required for 2FA or communicating with your hosting service managing your infrastructure isn’t running on that same infrastructure.
pmlnr 6y ago
I only have a personal server running in Hetzner but it's mirrored onto a tiny local computer at home. They both run postfix + dovecot, so mail is synced via dovecot replication. Data is rsync-ed daily, and everything has ZFS snapshots. MySQL is not set into replication - my home internet breaks often enough to have serious issues, so instead I drop everything every day import a full dump from the main server, and do a local dump as backup on both sides. I don't have automatic failover set up.
- clan 6y ago
  Not saying that you should never do a full mysql dump. Nor that you should not ensure that you can import a full dump. But when you already use ZFS you can do a very speedy full backup with: mysql << EOF FLUSH TABLES WITH READ LOCK; system zfs snapshot data/db@snapname UNLOCK TABLES; EOF Transfer the snapshot off-site (and test!). Either as a simple filecopy (the snapshot ensured a consistent database) or a little more advanced with zfs send/receive. This is much quicker and more painless than mysql dump. Especially with sizeable databases.
  - mwcampbell 6y ago
    Do you even need to flush the tables and grab a read lock while taking the ZFS snapshot? My understanding was that since ZFS snapshots are point-in-time consistent, taking a snapshot without flushing tables or grabbing a read lock would be safe; restoring from that snapshot would be like rebooting after losing power.
    - clan 6y ago
      I think you are correct. But then you risk data as you would with the unclean shutdown. I much prefer to have a known clean state which all things considered should be a safer bet. Just like some are OK running without fsync.
  - pmlnr 6y ago
    Good point, but my DB is tiny, so for now, I can afford the mysqldump. But I'll keep this in mind.
dave_sullivan 6y ago
I am literally in SBG2 so that has been fun. Turns out, our disaster recovery plan is pretty good. Datacenter burned down and I still was up 4 hours later in another data center with zero data loss. Good times.
- sebastianconcpt 6y ago
  And how's SBG1 doing?
  - BadBadJellyBean 6y ago
    About 1/3 destroyed
tomxor 6y ago
Basically the same (offsite backups), but the details are in the what and how which is subjective... For my purposes I decided that offsite backups should only comprise user data and that all server configuration be 100% scripted with some interactive parts to speed up any customization including recovering backups. I also have my own backup servers rather than using a service, and implement immutable incremental backups with rotated ZFS snapshots (this is way simpler than it sounds) - I can highly recommend ZFS as an extremely reliable incremental backup solution but you must enable block level deduplication and expect it to gobble up all the server RAM to be effective (but that's why I dedicate a server to it and don't need masses of cheap slow storage)... also the backup server is restorable by script and only relies on having at least one of the mirrored block devices in tact which I make a local copy of occasionally. I'm not sure how normal this strategy is outside of container land but I like just using scripts, they are simple and transparent - if you take time and care to write them well.
- mwcampbell 6y ago
  This sounds like what I want to do for the new infrastructure I'm setting up in one of OVH's US-based data centers. Are you running on virtual machines or bare metal? What kind of scripting or config management are you using?
  - tomxor 6y ago
    VPS although there is no dependency on VPS manager stuff so I don't see any issue with running on bare metal. No config managers, just bash scripts. They basically install and configure packages using sed or heredocs with a few user prompts here and there for setting up domains etc. If you are constantly tweaking stuff this might not suit you, but if you know what you need and only occasionally do light changes (which you must ensure the scripts reflect) then this could be an option for you. It does take some care to write reliable clear bash scripts, and there are some critical choices like `set -e` so that you can walk away and have it hit the end and know that it didn't just error in the middle without you noticing.
Ceezy 6y ago
A lot of prayers....
NicoJuicy 6y ago
Snapshots, db backups and data backups. Rolling backups with a month retention to box using rsync. It creates a network drive to box by default when I boot my desktop. I have some scripts for putting production db's in test and when I went them locally.
tetha 6y ago
At work, there are several layers. As an immediate plan, the 2-3 business critical systems are replicating their primary storages to systems in a different datacenter. This allows us to kick off the configuration management in a disaster, and we need something in between 1-4 hours to setup the necessary application servers and middlewares to get critical production running again. Regarding backups, backups are archived daily to 2 different borg repo hosts on different cloud providers. We could lose an entire hoster to shenanigans and the damage would be limited to ~2 days of data loss at worst. Later this year, we're also considering to export some of these archives to our sister team, so they can place a monthly or weekly backup on tape in a safe in order to have a proper offline backup. Regarding restores - there are daily automated restore tests for our prod databases, which are then used for a bunch of other tests after anonymization. On top, we've built most database handling on top of the backup/restore infra in order to force us to test these restores during normal business processes. As I keep saying, installing a database is not hard. Making backups also isn't hard. Ensuring you can restore backups, and ensuring you are not losing backups almost regardless of what happens... that's hard and expensive.
temp8964 6y ago
Do you have access to your control panel when the servers are down?
SergeAx 6y ago
I don't have to "backup servers" for a long time now. I have an Ansible playbook to deploy and orchestrate services, which, in turn, are mostly dockerized. So my recovery plan is to turn on "sorry, maintenance" banner via CDN, spin up a bunch of new VPSes, run Ansible scenario for deployment and restore database from hidden replica or latest dump.
- dboreham 6y ago
  > restore database from hidden replica or latest dump You do have backup servers.
  - deleted 6y ago
    [deleted]
  - jaywalk 6y ago
    He said he doesn't have to backup servers, not that he doesn't have backup servers.
traveler01 6y ago
Most people/companies don't have money to setup those disaster plans. They need you to have a similar server ready to go and also a backuo solution like Amazon S3. I was affected, my personal VPS is safe but down and other VPS I was managing I don't know anything about. I have the backups and right now I'd love for them to just set me up a new VPS so I can restore the backups and restore the services.
- rainmaking 6y ago
  Spin it up now and refuse to pay for the old one later.
lesquivemeau 6y ago
Personally my email hosting is down but thanksfully my web hosting and nextcloud instance were both at GRA2 (Gravelines). But i have a friend who potentially lost important uni work hosted on his nextcloud instance... On SBG2. A rough reminder that backups are really important, even if you are just an individual
cbozeman 6y ago
> What's YOUR Disaster Recovery Plan? Prayer and hope, usually.
Cub3 6y ago
Never thought i'd read the header > Printer flammability

tialaramex 6y ago

If you are a corporate entity of some kind, the final layer of your plan should always be "Go bankrupt". You can't successfully recover from every possible disaster and you shouldn't try to. In the event of a sufficiently unlikely event, your business fails and every penny spent attempting the impossible will be wasted, move on and let professional administrators salvage what they can for your creditors. Lots of people plan for specific elements they can imagine and forget other equally or even more important things they are going to need in a disaster. Check out how many organisations that doubtless have 24/7 IT support in case a web server goes down somehow had no plan for what happens if it's unsafe for their 500 call centre employees to sit in tiny cubicles answering phones all day even though pandemic respiratory viruses are so famously likely that Gates listed them consistently as the #1 threat.

jfrunyon 6y ago
IMHO, the part they had no plan for was being unable to just require their employees to come in anyway...
- osmano807 6y ago
  Just lobby the government to put call centers in "essential services". In my state they are open even with a partial lockdown.
- tialaramex 6y ago
  The more insecure your workers, the easier it is to get them to come in, regardless of what the supposed rules may or may not be. Fast Fashion for example often employs workers in more or less sweatshop conditions close to the customers (this makes commercial sense, if you make the hot new items in Bangladesh you either need to expensively air freight them to customers or they're going to take weeks to arrive after they're first ordered - there's a reason it isn't called "Slow fashion"). These jobs are poorly paid, many workers have dubious right-to-work status, weak local language skills, may even be paid in cash - and so if you tell them they must come in, none of them are going to say "No". In fact the slackening off in R for the area where my sister lives (today the towering chimneys and cavernous brick factories are just for tourists, your new dress was made in an anonymous single story building on an industrial estate) might be driven more by people not needing to own new frocks every week when they've been no further than their kitchen in a month than because it would actually be illegal to staff their business - if nobody's buying what you make then suddenly it makes sense to take a handout from the government and actually shut rather than pretend making mauve turtleneck sweaters or whatever is "essential".
  - Mauricebranagh 6y ago
    For non uk residents "Frock" is a regional term for dress - quite common in the Midlands.
  - namibj 6y ago
    Just to clarify: trans-atlantic shipments take a week port-to-port, e.g. Newark, NJ, USA to Antwerp, Belgium. (Bangladesh to Italy via Suez-channel looks like a 2-week voyage, or 3 weeks to the US west coast. Especially the latter would probably have quite a few stops on the way along the Asian coast.) You get better economics than shipping via air-freight from one full pallet and up. Overland truck transport to and from the port is still cheaper than air freight, at least in the US and central Europe. For these major routes, there are typically at least bi-weekly voyages scheduled, so for this kind of distance, you can expect about 11 days pretty uniformly distributed +-2 days, if you pay to get on the next ship. This may lead to (committing to) paying for the spot on the ship when your pallet is ready for pickup at the factory, not when it arrives at the port) and use low-delay overland trucking services. Which operate e.g. in lockstep with the port processing to get your pallet on the move within half a day of the container being unloaded from the ship, ideally having containers pre-sorted at the origin to match truck routes at the destination. So they can go on a trailer directly from the ship and rotate drivers on the delivery tour, spending only a few minutes at each drop-off. Because those can't rely on customers to be there and get you unloaded in less than 5 minutes, they need locations they can unload at with on-board equipment. They'd notify the customer with a GPS-based ETA display, so the customer can be ready and immediately move the delivery inside. Rely on 360-degree "dashcam" coverage and encourage the customer to have the drop-off point under video surveillance, just to easily handle potential disputes. Have the delivery person use some suitable high-res camera with a built-in light to get some full-surface-coverage photographic evidence of the condition it was delivered in. I'd guess with a hydraulic lift on the trailer's back and some kind of folding manual pallet jack stuck on that (fold-up) lift, so they drive up to the location, unlock the pallet jack, un-fold the lift, lower the lift almost to the ground, detach the pallet jack to drop it the last inch/few cm to the ground, pull the jack out, lower the lift the rest of the way, drive it on to the lift, open the container, get up with the pallet jack, drive the pallets (one-by-one) for this drop-off out of the container and leave them on the ground, close and lock the container, re-arm the jack's hooks, shove it jack back under the slightly-lowered folding lift, make it hook back in, fold it up, lock the hooking mechanism (against theft at a rest stop (short meal and toilet breaks exist, but showering can be delayed for the up to 2 nights)), fold it all the way up, and go on to drive to their next drop-off point.
jacquesm 6y ago
The final layer is call the insurance company.
- corty 6y ago
  Not really, the insurance won't make things right in an instant. They will usually compensate you financially, but often only after painstaking evaluation of all circumstances, weighing their chances in court to get out of paying you and maybe a lengthy court battle and a race against your bankruptcy. So yes, getting insurance can be a good idea to offset some losses you may have, as long as they are somewhat limited compared to your companies overall assets and income. But as soon as the insurance payout matches a significant part of your net worth, the insurance might not save you.
  - jacquesm 6y ago
    Fair enough.
- tomatocracy 6y ago
  There are always uninsurable events and for large enough companies/risks there are also liquidity limits to the size of coverage you can get from the market even for insurable events. As such, it makes sense to make the level of risk you plan to accept (by not being insured against it and not mitigating) a conscious economic decision rather than pretending you've covered everything.
  - jacquesm 6y ago
    As long as you have outside shareholders you can decide that. If you do you'd be surprised about how they will respond to an attitude like that. After all: you can decide the levels of risk that you personally are comfortable with leading to extinguishing of the business, but a typical shareholder is looking at you to protect their investment and not insuring against a known risk which at some point in time materializes is an excellent way to find yourself in the crosshairs of a minority shareholder lawsuit against a (former) company executive.
brmgb 6y ago
"Go bankrupt" is not a plan. Becoming insolvent might be the end result of a situation but it's not going to help you deal with it. Let's take an example which might lead to bankruptcy. A typical answer to a major disaster (let's say your main and sole building burning as a typical case) for an SME would be to cease activity, furlough employes and stop or defer every payments you can while you claim insurance and assess your options. Well, none of these things are obvious to do especially if all your archive and documents just burnt. If you think about it (which you should), you will quickly realise that you at least need a way to contact all your employes, your bank and your counsel (which would most likely be the accountant certifying your results rather than a lawyer if you are an SME in my country) offsite. That's the heart of disaster planning: having solutions at the ready for what was easy to foresee so you can better focus on what wasn't.
- Twisell 6y ago
  Hence the "final layer" statement. Bankruptcy when dealt with correctly is a process not an end. If everything else fail it's better to fill for bankruptcy when there is still something to recover with help of others than to burn everything to ashes because of your vanity. At least that's how I understood parent's comment.
  - brmgb 6y ago
    > Bankruptcy when dealt with correctly is a process not an end. Yes, that's why "Go bankrupt" is not a plan which was the entire point of my reply. That's like saying that your disaster recovery plan is "solve the disaster".
  - Sanzig 6y ago
    As a quick interlude, since this may be confusing to non-US readers: bankruptcy in the United States in the context of business usually refers to two concepts, whereas in many other countries it refers to just one. There are two types of bankruptcies in the US used most often by insolvent businesses: Chapter 7, and Chapter 11. A Chapter 7 bankruptcy is what most people in other countries think of when they hear "bankruptcy" - it's the total dissolution of a business and liquidiation of its assets to satisfy its creditors. A business does not survive a Chapter 7. This is often referred to as a "bankruptcy" or "liquidation" in other countries. A Chapter 11 bankruptcy, on the other hand, is a process by which a business is given court protection from its creditors and allowed to restructure. If the creditors are satisfied with the reorganisation plan (which may include agreeing to change the terms of outstanding debts), the business emerges from Chapter 11 protection and is allowed to continue operating. Otherwise, if an agreement can't be reached, the business may end up in Chapter 7 and get liquidated. Most countries have an equivalent to a Chapter 11, but the name for it varies widely. For example, Canada calls it a "Division 1 Proposal," Australia and the UK call it "administation," and Ireland calls it "examinership." Since there's a lot of international visitors to HN I just thought I'd jump in and provide a bit of clarity so we can all ensure we're using the same definition of "bankruptcy." A US Chapter 7 bankruptcy is not a plan, it's the game over state. A US Chapter 11 bankruptcy, on the other hand, can definitely be a strategic maneuver when you're in serious trouble, so it can be part of the plan (hopefully far down the list).
physicsguy 6y ago
It's not quite that simple, the data you might have may be needed for compliance or regulatory reasons. Having no backup strategy might make you personally liable depending on the country!

ahmedalsudani 6y ago

My disaster recovery plan: we shall rebuild.

edoceo 6y ago
F
julianwachholz 6y ago
Finally we get to rewrite everything from scratch!
- philpem 6y ago
  The spice will flow, and the tech debt will go!

kijin 6y ago

One of my backup servers used to be in the same datacenter as the primary server. I only recently moved it to a different host. It's still in the same city, though, so I'm considering other options. I'm not a big fan of just-make-a-tarball-of-everything-and-upload-it-to-the-cloud backup methodology, I prefer something a bit more incremental. But with Backblaze B2 being so cheap, I might as well just upload tarballs to B2. As long as I have the data, the servers can be redeployed in a couple of hours at most. The SBG fire illustrates the importance of geographical redundancy. Just because the datacenters have different numbers at the end doesn't mean that they won't fail at the same time. Apart from a large fire or power outage, there are lots of things that can take out several datacenters in close vicinity at the same time, such as hurricanes and earthquakes.

sebmellen 6y ago
Duplicity is your best bet for incremental backups using B2. I use this for my personal server and it works brilliantly.
- gingerlime 6y ago
  I thought so too for a long while. Until I was trying to restore something (just to test things), and wasn’t able to... it might have been specific to our GPG or an older version or something... but I decided to switch to restic and am much happier now. Restic has a single binary that takes care of everything. It feels more modern and seems to work really well. Never had any issue restoring from it. Just one data point. Stick to whatever works for you. But important to test not only your backups, but also restores!
  - sebmellen 6y ago
    Interesting, I will check Restic out, I’ve heard other good things about it. Duplicity is a bit of a pain to set up and Restic’s single binary model is more straightforward (Go is a miracle). Thanks for the recommendation! GPG is a bit quirky but I do regularly check my backups and restores (if once every few months counts as regular).
  - 6y ago

tothrowaway 6y ago

I'm at OVH as well (in the BHS datacenter, fortunately). I run my entire production system on one beefy machine. The apps and database are replicated to a backup machine hosted with Hetzner (in their Germany datacenter). I also run a tiny VM at OVH which proxies all traffic to Hetzner. I use a failover IP to point at the big rig at OVH. If the main machine fails, I move the failover IP to the VM, which sends all traffic to Hetzner. If OVH is totally down, and the fail over IP doesn't work, I have a fairly low TTL on the DNS. I backup the database state to S3 every day. Since I'm truly paranoid, I have an Intel NUC at my house that also replicates the DB. I like knowing that I have a complete backup of my entire business within arm's reach.

warrenm 6y ago
Yet another cheer for Hetzner!
AmericanChopper 6y ago
> I like knowing that I have a complete backup of my entire business within arm's reach. It could also provide a burglar a fantastic opportunity to pivot into career in data breaches.
- baggy_trough 6y ago
  It all depends on your paranoia level of data hacking burglars vs. vaporized data centers.
- tgsovlerkhgsel 6y ago
  This problem is usually solved through encryption.
  - AmericanChopper 6y ago
    If I were to ask my CISO if I was allowed to bring the production database home, I’m pretty sure his answer wouldn’t be “as long as you encrypt it”.

ethanpil 6y ago

Does anyone have experience with lvarchive or FSArchiver (or similar) to backup images of live systems instead of file based backup solutions?

jbeales 6y ago

My recovery plan: tarball & upload to Object Store. I'm going to check out exactly how much replication the OVH object store offers, and see about adding a second geographic location, and maybe even a second provider, tomorrow. (My servers aren't in SBG either - phew!)

nucleardog 6y ago
If your primary data is on OVH, I'd look at using another company's object store if feasible (S3, B2, etc). If possible, on another payment method. (If you want to be really paranoid, something issued under another legal entity.) There's a whole class of (mostly non-technical) risks that you solve for when you do this. If anything happens with your payment method (fails and you don't notice in time; all accounts frozen for investigation), OVH account (hacked, suspended), OVH itself (sudden bankruptcy?), etc, then at least you have _one_ other copy. It's not stuff that's likely to happen, but the cost of planning for it at least as far as "haven't completely lost all my data even if it's going to be a pain to restore" here is relatively minimal.

juangacovas 6y ago

Less than a day for disaster recovery on fresh hardware? Same as my case. As you say, good enough for most purposes, but I'm also looking for improvement. I have offsite realtime replicas for data and mariaDBs, and offsite nightly backups (combo of rsnapshot, lsyncd, mariaDB multi-source replication, and a post-new-install script that setups almost everything in case you have to recover on bare-metal, i.e. no available VM snapshots). Currently trying to reduce that "less than a day" though. Recently discovered "ReaR" (Relax and Recover) from RedHat and sounds really nice for bare-metal servers. Not everybody runs on virtualized/cloud (being able to recover from VM snapshots is really a plus). Let's share experiencies :)

sparrish 6y ago

Got burned once (no pun intended), learned my lesson. Hot spare on a different continent with replicated data along with a third box just for backups. The backup box gets offsite backups held in a safe with another redundant copy in another site in another safe. Restores are tested quarterly. Keep backups of backups. Once bitten, twice shy.

gameshot911 6y ago
Have you been bitten, personally? If so, story time?
Twirrim 6y ago
> Hot spare on a different continent Just be cautious about data locality laws (not likely to affect you as joe average, more for businesses)
- bongoman37 6y ago
  yeah, there's lots of countries with regulations that certain data can't leave the geographical boundary of the country. Often, it is the most sensitive data.
  - mike_d 6y ago
    These laws generally don't work how people think they do. For example, the Russian data residency law states that a copy of the data must be stored domestically, not that it can't be replicated outside the country. The UAE has poorly written laws that have different regulations for different types of data - including fun stuff like only being subject to specific requirements if the data enters a 270 acre business park in Dubai. Don't even get me started on storing encrypted data in one country and the keys in another...
- Mauricebranagh 6y ago
  A few years ago I worked on the British Telecom Worldwide intranet team and we had a matrix mapping various countries encryption laws. This was so we remained legal in all of the countries BT worked in which required a lot of behind the scenes work to make sure we didn't serve "illegaly encypted" Data.

fy20 6y ago

I have three servers (1 OVH - different location, 2 DO). The only thing I backup is the DB, which is synced daily to S3. There's a rule to automatically delete files after 30 days to handle GDPR and stop the bucket and costs spiralling out of control. Everything is managed with Ansible and Terraform (on DO side), so I could probably get everything back up and running in less than an hour if needed.

sverhagen 6y ago
> probably That makes it sound like you didn't try/practice. I imagine that in a real-life scenario things will be a little more painful than in one's imagination.
- dylan604 6y ago
  Exactly. Having a plan is only part of it. Good disaster plans do dry runs a couple of times a year (when time changes is always convenient reminder). If you rehearse the recovery when you're not panicked, you have a better chance of not skipping a step when the timing is much more crucial. Also, some sort of guide with steps given procedurally is a great idea.
  - slaymaker1907 6y ago
    I don't think this is necessarily true for all parts of a disaster plan. Some mechanisms may be untestable because it is unknown how to actually trigger it (think certain runtime assertions, but on a larger scale). Even if it possible to trigger and test, actually using the recovery mechanism may have some high cost either monetarily or maybe losing some small amount of data. These mechanisms should almost always be an additional layer of defense and only be invoked in case of true catastrophe. In both cases, the mechanisms should be tested as thoroughly as possibly, either through artificial environments that can simulate improbable scenarios or in the latter case on a small test environment to minimize cost.
- fy20 6y ago
  I haven't ever deleted everything and timed how long I could get it up and running again, but I have tested it works by spinning up new machines and moving everything over to there (it was easier than running "sudo apt-get dist-upgrade").

dfsegoat 6y ago

We test rolling over the entire stack to another AWS DR region (just one we dont normally use) from S3 backups, etc. We do this annually and try to introduce some variations to the scenarios. It takes us about 18 hours realistically. Documentation / SOPs that have been tested thoroughly by various team members are really important. It helps work out any kinks in interpretation, syntax errors etc. It does feel a little ridiculous at the time for all the effort involved, but incidents like this show why it's so important.

jfrunyon 6y ago

Servers are at a mix of "cloud" providers, and on-site. Most data (including system configs!) is backed up on-site nightly, and to B2 nightly with historical copies - and critical data is also live-replicated to our international branches. (Some "meh" data is backed up only to B2, like our phone logs; we can get most of the info from our carrier anyway). Our goal and the reason we have a lot of stuff backed up on-prem is to have our most time-critical operations back up within a couple of hours - unless the building is destroyed, in which case that's a moot point and we'll take what we can get. A dev wiped our almost-monolithic sales/manufacturing/billing/etc MySQL database a month or two ago. (I have been repeatedly overruled on the topic of taking access to prod away from devs) We were down for around an hour. Most of that time was spent pulling gigs of data out of the binlog without also wiping it all again. Because our nightly backups had failed a couple weeks prior - after our most recent monthly "glance at it".

el-salvador 6y ago

Pictures from the fire: https://www.dna.fr/amp/faits-divers-justice/2021/03/10/strasbourg-important-incendie-dans-une-entreprise-situee-sur-un-site-seveso-au-port-du-rhin https://www.dna.fr/amp/faits-divers-justice/2021/03/10/stras...

pulse7 6y ago
Uff... it looks like half of the containers on this picture were on fire...
- deleted 6y ago
  [deleted]
- jannes 6y ago
  Are you making a joke about docker containers or am I missing something?
  - dgellow 6y ago
    Part of OVH datacenters are literal, physical containers with racks, power supply, vents, etc. You can see more details here: https://baxtel.com/data-center/ovh-strasbourg-campus https://baxtel.com/data-center/ovh-strasbourg-campus
    - Mauricebranagh 6y ago
      Do the containers have fire suppressant systems installed?
simonke 6y ago
Without AMP: https://www.dna.fr/faits-divers-justice/2021/03/10/strasbourg-important-incendie-dans-une-entreprise-situee-sur-un-site-seveso-au-port-du-rhin https://www.dna.fr/faits-divers-justice/2021/03/10/strasbour...
jacquesm 6y ago
Nobody hurt. That's a bit of good news.

jacquesm 6y ago

It's true: most companies do not have a disaster recovery plan, and many of them confuse a breach protocol with a disaster recovery plan ('we have backups'). Fires in DCs aren't rare at all, I know of at least three, one of those in a building where I had servers. This one seems to be worse than the other two. Datacenters tend to concentrate a lot of flammable stuff, throws a ton of current through them and does so 24x7. The risk of a fire is definitely not imaginary, which is why most DCs have fire suppression mechanisms. Whether those work as advertised depends on the nature of the fire. An exploding on prem transformer took out a good chunk of EV1's datacenter in the early 2000's, and it wasn't so much the fire that caused problems for their customers, but the fact that someone got injured (or even died, I don't recall exactly), and before the investigation was completed and the DC released to the owners again took a long time. Being paranoid and having off-site backups is what allowed us to be back online before the fire was out. If not for that I don't know if our company would have survived.

lgeorget 6y ago

We have two servers at OVH (RBX and GRA, not SBG). I make backups of all containers and VMs every day and keep the last three, plus one one each month. Backups are stored in a separate OVH storage disk and also downloaded to a NAS on-premise. In case of a disaster, we'd have to rent a new server, reprovision the VMs and containers and restore the backups. About two days of work to make sure everything works fine and we could lose about 24 hours of data. It's not the best in terms of Disaster Recovery Plan but we accept that level of risk.

Cthulhu_ 6y ago

Personal: I run a webserver for some website (wordpress + xenforo), I've set up a cronjob that creates a backup of /var/www, /etc and a mysql database dump, then uploads it to an S3 bucket (with automatic Glacier archiving after X period set up). It should be fairly straightforward to rent a new server and set things back up. I still dislike having to set up a webserver + php manually though, I don't get why that hasn't been streamlined yet. My employer has a single rack of servers at HQ. It's positioned at a very specific angle with an AC unit facing it, their exact positions are marked out on the floor in tape. The servers contain VMs that most employees work on, our git repository, issue trackers, and probably customer admin as well. They say they do off-site backups, but honestly, when (not if) that thing goes it'll be a pretty serious impact on the business. They don't like people keeping their code on their take-home laptop either (I can't fathom how my colleagues work and how they can stand working in a large codebase using barebones vim over ssh), but I've employed some professional disobedience there.

catbuttes 6y ago
Have you considered writing an ansible playbook to set all that up? You could even have it pull down the backup and do a full restore for you...

walrus01 6y ago

Video of the aftermath: https://twitter.com/PompiersFR/status/1369544801817944071 https://twitter.com/PompiersFR/status/1369544801817944071 https://twitter.com/abonin_DNA https://twitter.com/abonin_DNA

shog_hn 6y ago

Self-hosted Kuberenetes and a FreeNAS storage system at home, and a couple of VMs in the cloud. I've got a mixed strategy, but it covers everything to remote locations. I use S3 API compatible object storage platforms for remote backup. E.g. BackBlaze B2. I wrote about my backup scripts for FreeNAS (jail that runs s3cmd to copy files to B2) here: https://www.shogan.co.uk/cloud-2/cheap-s3-cloud-backup-with-backblaze-b2/ https://www.shogan.co.uk/cloud-2/cheap-s3-cloud-backup-with-... For Kuberenetes I use velero which can be configured with an S3 storage backend target: https://www.shogan.co.uk/kubernetes/kubernetes-backup-on-raspberry-pi/ https://www.shogan.co.uk/kubernetes/kubernetes-backup-on-ras...

KronisLV 6y ago

Here's what i do for my homelab setup that has a few machines running locally and some VPSes "in the cloud": I personally have almost all of the software running in containers with an orchestrator on top (Docker Swarm in my case, others may also use Nomad, Kubernetes or something else). That way, rescheduling services on different nodes becomes less of a hassle in case of any one of them failing, since i know what should be running and what configuration i expect it to have, as well as what data needs to be persisted. At the moment i'm using Time4VPS ( affiliate link: https://www.time4vps.com/?affid=5294 https://www.time4vps.com/?affid=5294 ) for the stuff that needs decent availability and because they're cheaper than almost all of the alternatives i've looked at (DigitalOcean, Vultr, Scaleway, AWS, Azure) and that matters to me. Now, in case the entire data centre disappears, all of my data would still be available on a few HDDs under my desk (which are then replicated to other HDDs with rsync locally), given that i use BackupPC for incremental scheduled backups with rsync: https://backuppc.github.io/backuppc/ https://backuppc.github.io/backuppc/ For simplicity, the containers also use bind mounts, so all of the data is readable directly from the file system, for example, under /docker (not really following some of the *nix file system layout practices, but this works for me because it's really easy to tell where the data that i want is). I actually had to migrate over to a new node a while back, took around 30 minutes in total (updating DNS records included). Ansible can also really help with configuring new nodes. I'm not saying that my setup would work for most people or even anything past startups, but it seems sufficient for my homelab/VPS needs. My conclusions: - containers are pretty useful for reproducing software across servers - knowing exactly which data you want to preserve (such as /var/lib/postgresql/data/pgdata) is also pretty useful, even though a lot of software doesn't really play nicely with the idea - backups and incremental backups are pretty doable even without relying on a particular platform's offerings, BackupPC is more than competent and buying HDDs is far more cost effective than renting that space - automatic failover (both DNS and moving the data to a new node) seems complicated, as does using distributed file systems; those are probably useful but far beyond what i actually want to spend time on in my homelab - you should still check your backups

vbsteven 6y ago

In my case: * All my services are dockerized and have gitlab pipelines to deploy on a kubernetes cluster (RKE/K3s/baremetal-k8s) * git repo's containing the build scripts/pipelines are replicated on my gitlab instance and multiple work computers (laptop & desktop) * Data and databases are regularly dumped and stored in S3 and my home server * Most of the infrastructure setup (AWS/DO/Azure, installing kubernetes) is in Terraform git repositories. And a bit of Ansible for some older projects. Because of the above, if anything happens all I need to restore a service is a fresh blank VM/dedicated machine or a cloud account with a hosted Kubernetes offering. From there it's just configuring terraform/ansible variables with the new hosts and executing the scripts.

jrib 6y ago
How often do you test starting from a clean slate?
- kuschku 6y ago
  I have a similar setup: I recreate everything for every major kubernetes update.

neurostimulant 6y ago

Nothing too crazy, just a simple daily cron to see sync user data and database dumps on our OVH boxes to backblaze and rsync.net. This simple setup is already saved our asses a few times already.

cm2187 6y ago

Also stupid things not to forget: make sure your dns provider is independent otherwise you won’t be able to point to your new server (or have a secondary DNS provider). Make sure any email required for 2FA or communicating with your hosting service managing your infrastructure isn’t running on that same infrastructure.

pmlnr 6y ago

I only have a personal server running in Hetzner but it's mirrored onto a tiny local computer at home. They both run postfix + dovecot, so mail is synced via dovecot replication. Data is rsync-ed daily, and everything has ZFS snapshots. MySQL is not set into replication - my home internet breaks often enough to have serious issues, so instead I drop everything every day import a full dump from the main server, and do a local dump as backup on both sides. I don't have automatic failover set up.

clan 6y ago
Not saying that you should never do a full mysql dump. Nor that you should not ensure that you can import a full dump. But when you already use ZFS you can do a very speedy full backup with: mysql << EOF FLUSH TABLES WITH READ LOCK; system zfs snapshot data/db@snapname UNLOCK TABLES; EOF Transfer the snapshot off-site (and test!). Either as a simple filecopy (the snapshot ensured a consistent database) or a little more advanced with zfs send/receive. This is much quicker and more painless than mysql dump. Especially with sizeable databases.
- mwcampbell 6y ago
  Do you even need to flush the tables and grab a read lock while taking the ZFS snapshot? My understanding was that since ZFS snapshots are point-in-time consistent, taking a snapshot without flushing tables or grabbing a read lock would be safe; restoring from that snapshot would be like rebooting after losing power.
  - clan 6y ago
    I think you are correct. But then you risk data as you would with the unclean shutdown. I much prefer to have a known clean state which all things considered should be a safer bet. Just like some are OK running without fsync.
- pmlnr 6y ago
  Good point, but my DB is tiny, so for now, I can afford the mysqldump. But I'll keep this in mind.

dave_sullivan 6y ago

I am literally in SBG2 so that has been fun. Turns out, our disaster recovery plan is pretty good. Datacenter burned down and I still was up 4 hours later in another data center with zero data loss. Good times.

sebastianconcpt 6y ago
And how's SBG1 doing?
- BadBadJellyBean 6y ago
  About 1/3 destroyed

tomxor 6y ago

Basically the same (offsite backups), but the details are in the what and how which is subjective... For my purposes I decided that offsite backups should only comprise user data and that all server configuration be 100% scripted with some interactive parts to speed up any customization including recovering backups. I also have my own backup servers rather than using a service, and implement immutable incremental backups with rotated ZFS snapshots (this is way simpler than it sounds) - I can highly recommend ZFS as an extremely reliable incremental backup solution but you must enable block level deduplication and expect it to gobble up all the server RAM to be effective (but that's why I dedicate a server to it and don't need masses of cheap slow storage)... also the backup server is restorable by script and only relies on having at least one of the mirrored block devices in tact which I make a local copy of occasionally. I'm not sure how normal this strategy is outside of container land but I like just using scripts, they are simple and transparent - if you take time and care to write them well.

mwcampbell 6y ago
This sounds like what I want to do for the new infrastructure I'm setting up in one of OVH's US-based data centers. Are you running on virtual machines or bare metal? What kind of scripting or config management are you using?
- tomxor 6y ago
  VPS although there is no dependency on VPS manager stuff so I don't see any issue with running on bare metal. No config managers, just bash scripts. They basically install and configure packages using sed or heredocs with a few user prompts here and there for setting up domains etc. If you are constantly tweaking stuff this might not suit you, but if you know what you need and only occasionally do light changes (which you must ensure the scripts reflect) then this could be an option for you. It does take some care to write reliable clear bash scripts, and there are some critical choices like `set -e` so that you can walk away and have it hit the end and know that it didn't just error in the middle without you noticing.

Ceezy 6y ago

A lot of prayers....

NicoJuicy 6y ago

Snapshots, db backups and data backups. Rolling backups with a month retention to box using rsync. It creates a network drive to box by default when I boot my desktop. I have some scripts for putting production db's in test and when I went them locally.

tetha 6y ago

At work, there are several layers. As an immediate plan, the 2-3 business critical systems are replicating their primary storages to systems in a different datacenter. This allows us to kick off the configuration management in a disaster, and we need something in between 1-4 hours to setup the necessary application servers and middlewares to get critical production running again. Regarding backups, backups are archived daily to 2 different borg repo hosts on different cloud providers. We could lose an entire hoster to shenanigans and the damage would be limited to ~2 days of data loss at worst. Later this year, we're also considering to export some of these archives to our sister team, so they can place a monthly or weekly backup on tape in a safe in order to have a proper offline backup. Regarding restores - there are daily automated restore tests for our prod databases, which are then used for a bunch of other tests after anonymization. On top, we've built most database handling on top of the backup/restore infra in order to force us to test these restores during normal business processes. As I keep saying, installing a database is not hard. Making backups also isn't hard. Ensuring you can restore backups, and ensuring you are not losing backups almost regardless of what happens... that's hard and expensive.

temp8964 6y ago

Do you have access to your control panel when the servers are down?

SergeAx 6y ago

I don't have to "backup servers" for a long time now. I have an Ansible playbook to deploy and orchestrate services, which, in turn, are mostly dockerized. So my recovery plan is to turn on "sorry, maintenance" banner via CDN, spin up a bunch of new VPSes, run Ansible scenario for deployment and restore database from hidden replica or latest dump.

dboreham 6y ago
> restore database from hidden replica or latest dump You do have backup servers.
- deleted 6y ago
  [deleted]
- jaywalk 6y ago
  He said he doesn't have to backup servers, not that he doesn't have backup servers.

traveler01 6y ago

Most people/companies don't have money to setup those disaster plans. They need you to have a similar server ready to go and also a backuo solution like Amazon S3. I was affected, my personal VPS is safe but down and other VPS I was managing I don't know anything about. I have the backups and right now I'd love for them to just set me up a new VPS so I can restore the backups and restore the services.

rainmaking 6y ago
Spin it up now and refuse to pay for the old one later.

lesquivemeau 6y ago

Personally my email hosting is down but thanksfully my web hosting and nextcloud instance were both at GRA2 (Gravelines). But i have a friend who potentially lost important uni work hosted on his nextcloud instance... On SBG2. A rough reminder that backups are really important, even if you are just an individual

cbozeman 6y ago

> What's YOUR Disaster Recovery Plan? Prayer and hope, usually.

Cub3 6y ago

Never thought i'd read the header > Printer flammability