16 ms·

"Test your backups" is so easy to say, but quite difficult for many to do. There are a lot of shops that probably don't know how to recreate a machine from scra

"Test your backups" is so easy to say, but quite difficult for many to do. There are a lot of shops that probably don't know how to recreate a machine from scratch. How many systems are developed as balls of clay. Little bits added and smeared in over time until the ball just gets bigger, but each piece lost in the process. How many folks can go through their local config files and explain all of entries, how many can even tell which ones they have changed, or why? Especially when they were changed by Frank, but he left 2 years ago. You'd like to think you can just restore the system from backup and it'll just light back up. But how do you test this without cratering your existing system? Like a boat in a basement, many system are built in-situ and can be very rigid. Modern environments like cloud computing and creation scripts can mitigate this a bit organically, but how many of these systems are just a tower running Windows w/SQL Server and who knows what else? Plus whatever client software is on the client machines. How do you test that in isolation? At least read the media to see if it can be read (who doesn't love watching a backup tape fail halfway through the restore). Simply, it takes a lot of engineering to make a system that can be reliably restored, much less on a frequent basis. And this is all engineering that doesn't impact the actual project -- getting features to users and empowering the business. Which can make the task even more difficult.

xwdv 5y ago
This is like a sales pitch for just paying the ransom.
- MattGaiser 5y ago
  It probably works more often than not.
  - shyn3 5y ago
    It works until it doesn't, such as in a hardware failure. [1] https://www.anandtech.com/show/15673/dell-hpe-updates-for-40k-hour-essd-failures https://www.anandtech.com/show/15673/dell-hpe-updates-for-40...
  - quickthrower2 5y ago
    It's a prisoners dilemma
  - xwdv 5y ago
    Seriously, I started running through all those points he made and started thinking about other things he didn't mention and then halfway I was just like fuck it.
tw04 5y ago
This is all solved, it just takes money and typically bringing in outside experts. Occasionally it will require changes to apps but most of the time it can be retrofit. No it isn’t easy, but it’s also not an impossible task.
- foxpurple 5y ago
  I think easy or hard are not even appropriate terms. As a business owner it is not easy or hard, it just costs an amount of money. If you pay enough money, you can get the result. You just have to decide how much a disaster costs and how much preventing a disaster costs to work out if it’s worth it.
- sneak 5y ago
  It doesn't even require outside experts, just one or two competent sysadmins on staff.
briffle 5y ago
Honestly, this is one of the problems the cloud is great at solving. We keep things in seprate projects, and restore a backup from one production project (Kubernetes, Database servers, etc) to a special DR project we have set aside. The only step we do not do is update the front end DNS, and then run our tests after our Infrastrucutre as Code deployment is complete.
- rhizome 5y ago
  So a complete hot standby?
  - Godel_unicode 5y ago
    Sounds like a DR test; spin up an entire new thing to make sure the backups worked, then turn it off. Costs a few hours of extra resources.
    - dspillett 5y ago
      I do something similar with my home mail server (not sure I'd go with self-hosting if I started fresh today, but it has run without issue other than maintenance for upgrades & such for well over a decade). I have a second copy running in a low spec VM. Every day it wipes itself down and restores the latest backup, I check regularly (a couple of times per week) to see that it is running and has copies of recent mail (that part could be more automated but I've never go round to it...). It wasn't setup for DR specifically, but it is on a separate network so if push came to shove I could ramp up its resources, change a few DNS settings, open up relevant bits of its firewalling, and have it take over, if the main server failed in a way it hadn't copied yet.
    - rhizome 5y ago
      Could be; it's ambiguous. But yeah, not updating DNS could be just declining to test operation in the wilds of outside traffic before tearing it all down. That said, in this way, maybe a periodic "DR" that actually replaces the current operations would be a helpful...well, not test, but..."resilience practice" maybe? It could be a new twist on continuous deployment: continuous recovery.
nn3 5y ago
> How do you test that in isolation? Is it easier to test after the ransomware attack?
- bsder 5y ago
  Yes, actually. Before the attack, all machines are active and being used. If you screw one of them up that happens to be running something obscure but vital, the screwup is your fault. After the attack, all machines are dead so the screwup is blamed on somebody else. If you have the email/memo (you did print it out and file it, did you not?) showing that you informed the CTO/CIO, blame for the screwup will get buried.
- whartung 5y ago
  Actually it is easier because you get to "test" it by reloading on the actual hardware, not separate hardware. To do a solid test, you need to restore the system. It's difficult to restore a running system because, well, it's running. That typically means a parallel environment. A single box represents a bunch of "magic values" that are stuffed in some config somewhere. Imagine several of those. "We need to restore SQL Server, the AD Server, the Application Server..." Reloading on a new environment is an easy way to find out those magic numbers, typically the hard way. Restoring on your existing hardware, with existing networks, existing IPs, etc. you're laying your software and configs over a known, working environment. How do you test a recovery of licensed software thats bound to the machine that you have running in production, for example? "Oh, sorry, you have a duplicate license detected" and it shuts down your production system for a license violation. "Sorry, we detect the wrong number of cores", or whatever other horrors modern licensing systems make you jump through. You DO have another dongle, right? (Do they still use dongles?) It can be far easier to do an image restore to your already working system than trying to load it up on something else. Since your production box is horked anyway, an image restore should "just work". But testing it, that's another story completely.
  - Godel_unicode 5y ago
    Who is configuring physical servers by hand. Are you a part of some kind of museum exhibit? In all seriousness, you're talking about a business which is carrying a truly staggering amount of risk. If the margins are so tight they're running hand-rolled physical servers then restoration is a moot point. They'll go out of business if attacked. If you're not testing your backups, then they're broken with close to 100% certainty.
tomc1985 5y ago
God these "we're so ignorant!" responses are sooooo tiring. If people don't want to hire the hardcore nerds that know this shit, if they don't want to pay for the expertise to work the technology they build their business on, then maybe they deserve it. This modern attitude of compartmentalization, extremely localized expertise, and outsourcing everything to the cloud is going to be our downfall. Stop hiring muggles!
- jiggawatts 5y ago
  "No." Says a million finance people that prefer armies of compliant muggles over a few self-important wizards.
- flyinglizard 5y ago
  You can’t just man every brick and mortar out there with “hardcore nerds”. There simply isn’t enough of them.
- g_p 5y ago
  It strikes me that companies need to look back at covid and consider the value IT provided to their business during that period, and the extent to which it has grown their business since the 90s. Then take that and allocate a reasonable chunk of that growth (say 15%) to ensuring that can continue, through ongoing investment in IT. Their alternative is to abandon the internet, go back to paper, and dump their IT costs (and boost to business). Unfortunately though, as long as there are bean counters trying to cut every outgoing and outsource every cost, and play accounting bingo to turn everything into monthly opex, it's unlikely to change. Most companies need skilled technical people because the sheer aggregation of risk in a few outsourced providers (see Kaseya recently) shows that they won't be top priority when something hits the fan. If they want to be top priority, they need the people on payroll and on site. Not everyone needs a group of rockstar 10x-types, but we do definitely need better fundamental IT knowledge and ability to solve problems. And business needs to make clear this is what's needed in terms of job adverts and compensation - supply tends to rapidly learn to deliver what's valued... If you can convince bean counters to pay anything for it, that is...
  - 5y ago
kenniskrag 5y ago
I provide a test system in a vm. I have an installation script for the db. Dump the db and replace paths, import, start application, done. This can then be used to test the new release before deployment.
- kenniskrag 5y ago
  bonus: the vm can be used for debugging or used as playground
  - ashtonkem 5y ago
    Be mindful of what customer data you’re handling though.
    - handrous 5y ago
      Underrated problem of testing backups, generally. Making it easy/common can also mean greater exposure to risk of data leaks, if you don't take great care.
MeinBlutIstBlau 5y ago
You don't even need system images. Just the data. So SQL backups and just file directories. However for a company of our size, it's not really possible. I was talking to my lead about this last year and we have about 50TB of our entire system I believe stored in our databases. All he said was "we have one, but hopefully I'm retired before needing to find out how to restore from it."
emodendroket 5y ago
Well, that's why you pay more for PaaS than hosting it yourself, right? You can make it someone else's problem, to an extent.
paxys 5y ago
Every business in the world hires accountants to balance their books, lawyers when they need to file paperwork or respond to a problem, mechanics, electricians, plumbers and other professionals to fix physical stuff. Yet when it comes to software the most they will consider is an online monthly fee for something cheap and out of the box, and when things get complicated or go wrong "this is out of our area of expertise" is always the excuse. It's really time to treat software professionals, especially when related to security, as a core requirement for EVERY organization, starting from a 2 person mom-and-pop shop.
- 908B64B197 5y ago
  > It's really time to treat software professionals, especially when related to security, as a core requirement for EVERY organization Some organizations do, some don't. Long time valuation seems to agree with the former, short term the latter!
- htrp 5y ago
  Those other groups have professional organizations with licensing. Plumbers will almost always refuse to do maintenance work on any work done by an unlicensed person. .... software/IT just hasn't had people willing to do that.
  - nitwit005 5y ago
    If all you need is "person who is able to properly administer Windows systems", there are things like Microsoft's certifications.
jms703 5y ago
A failure to invest in business continuity and maintaining your systems will cost you one way or the other. Hard drive failure, fire, flood, lightning strike, or ransomeware. These aren’t new problems facing businesses.
deleted 5y ago
[deleted]
z3t4 5y ago
Testing the backups is harder then you think... It's not like you are going to double up your entire server fleet just to see if you can restore everything from backup. You maybe test restoration of one or two servers and then assume the rest will also work. And you probably have some redundancy so that the data is saved on different machines plus a backup plus an additional backup. Then if things goes down, which is very, very unlikely, like a coordinated nuclear attack on several data-centers, but of course can happen, you assume you will figure things out... And then someone will ask: so what happens if your team also die in the nuclear attack, then you add one zero to the backup price/cost estimation.
- dougpa 5y ago
  Finding something reasonable between "we're at risk for a nuclear attack" (a black swan event) and "we're at risk for having our root credentials exfiltrated" (a daily occurrence) is not hard. I feel like these kinds of dramatizing hyperboles are nowhere near the nuts & bolts of the situation. And honestly, why not double up your entire server fleet for a temporary build-from-scratch rehearsal? Many shops could quintuple their infrastructural costs and still sit far above being in red. Most software enterprises in the modern day don't reap economic value as a function of how well they can convert hardware resources, but human resources. Optimizing on infra costs is not a main priority for any shop I've seen. I imagine not even for an IaaS provider these days.
newsyyswen 5y ago
Many places have a separate "test" or "beta" environment, with less resources and maybe a small database with spoofed entries. Maybe we should get into the habit of going all Shiva/Brahma on that environment every week or month. Burn it to the ground, and recreate it with an automated process. Sort of like a meta CI test.
- silly-silly 5y ago
  Beautiful resiliance.
mycall 5y ago
> There are a lot of shops that probably don't know how to recreate a machine from scratch. You can't fix already broken processes. VMware solved this 20 years ago. It is pretty simple to restore VMs on different systems, you don't need to worry about the ball of mud when you can duplicate it.
- dougpa 5y ago
  You can't really serialize something like an ami. So how are you going to make an offsite backup? Things need to be relatively simple & reproducible otherwise you will get bitten in many different ways due to strategies like this.
  - nuker 5y ago
    > You can't really serialize something like an ami. Copy AMI to separate AWS account, not in your Org, and keep keys to that account offline.
    - dougpa 5y ago
      oooo true, that does invalidate my point. AMI's are very easy to copy-over across accounts, i.e. to a potentially firewalled account.
  - Godel_unicode 5y ago
    If you want to backup an individual AMI you're probably doing it wrong. That probability goes to near certainty when you're talking about serializing it for off-site backup. Backup the deployment automation and the data, sure.
    - dougpa 5y ago
      I agree, that was kind of my point. He was talking about taking VM's but in a cloud environment this is a bit more awkward. Copying and storing VM's securely is not hard, but transferring AMI's is the only rough equivalent I know of in the cloud world. Ideally, you don't have to do this. But for one part of my current stack, the configuration that this specially configured windows box has been lost for a while. Rebuilding from scratch has not worked each time it has been tried.
dylan604 5y ago
That's the point of running these tests though. Take a system down (by turning it off, and putting it safely in the corner). Then bring up a new machine to replace it. Go through the steps you think would be the proper procedure, then document the shit out of what didn't work and what needed to be done. Write that up so that it becomes the new procedure. The next time you run this quarterly test, you start with the latest procedures. Updating accordingly, rinse, repeat.
- galuano1 5y ago
  When you work with the old systems, you are always afraid to shutdown the machine, because more often than not, it will not boot up. And if you petition to replace it with a new machine, you may very likely hear the phrase "We are planning to move the app to cloud, so no need to build a backup in the meanwhile".
  - 908B64B197 5y ago
    "What if it shuts down tomorrow?"
  - whartung 5y ago
    Back in the day, we had purchased a new, bigger machine, and transferred over, and everything was just peachy. Months later, we had a power outage (scheduled I think, I don't recall). Anyway, at some point during the transition and such, I managed to have the machines hard mount NFS across each other. As long as one of the machines is up, everything is rosy. But cold start? They were both hanging while restoring the mount (which, they couldn't because neither was "up"). Took us about 1/2 hr to suss out what was happening and get single user in to tweak it, but...yea...exciting! "Smart" people do this kind of innocent stuff all the time.
joe_the_user 5y ago
"Test your backups" is so easy to say, but quite difficult for many to do. It's difficult to do, yes. But it's difficult in a "dig a tunnel through that mountain" way, not in a "solve this mysterious math problem" way. It certainly could be done. It would just take time and money (money including hiring people). People constantly point to the difficulty of backups and the difficulty of hardware level separations between systems. But these are merely difficult and costly. "No being hacked" and "writing always secure code" are impossible and so they won't protect from backup failure. And yes, companies would rather spend money on other things. That's what it come down to.
- irishsultan 5y ago
  It's interesting that you make that comparison, because I would choose the mysterious math problem every time. Probability of success is about equal, but it's far less dangerous to do math than to cut through a mountain.
  - pnt12 5y ago
    Nitpicking an analogy goes against the point - it simplifies some ideas at the cost of modifying the details. I think OP meant to compare a known hard task vs an unknown hard task. Both are hard, but first is known to be possible to finish, but the other not so.
    - irishsultan 5y ago
      I agree that's what he meant, but it's just interesting to me that for the known hard task he took something that also comes with danger. Exactly the feeling that I get when I think about restoring backups.
      - mannykannot 5y ago
        Well, one of the things about testing backups is that (if you are doing it remotely right) it reduces the danger when you need to actually restore. A widely accepted method fur judging at least subjective risk when there are insufficient facts for truly objective measure is to ask how much you would bet on various outcomes - such as whether P=NP is solved before New York's 2nd. Ave. subway is completed.
Spooky23 5y ago
No, it requires baseline professional competence. I’ve worked with 9-figure turnover entities broken by this sort of thing and the first recommendation is always fire or manage out the CIO, risk/audit officer and/or CFO. Everyone cries about having no money. What is lacking is an ability to identify and manage risk that puts the existence of the company as a going concern at risk.
snicker7 5y ago
I think what you are describing is a perfect advert for declarative/immutable infrastructure. Yes, it may require work and talent. But that's the price to pay for resiliency. Invest and modernize your tech stack or go bankrupt. Cloud tech can help. As can Guix/Nix.
Robotbeat 5y ago
You should definitely try. Bare metal backups are a good place to start if you have a complicated system. A lot of environments aren't as complicated, and you COULD just pull your hard drive out, put in a spare, and test how your backup works. At least, test it while airgapped to see if it comes up at all.
dheera 5y ago
Also, ransom gangs can take over your backups. Keep offline, airgapped backups.
EVa5I7bHFq9mnYK 5y ago
Ransomware gangs go after large organizations, those that can afford testing backups, not after pop and mom coffee shops.
- amelius 5y ago
  Is this true? I think big organizations just end up in the news more often.
  - yodelshady 5y ago
    Third-hand evidence, mom n' pop shops don't have the expertise to pay in bitcoin. Or perhaps, more charitably, they've been inoculated by previous low-effort scams and smartly assume bitcoin payments are another. Honestly I'm not sure many large IT depts have it either, but for $1M+ the attackers can afford good customer service.
- pbhjpbhj 5y ago
  There was a school group featured in a report on BBC Radio 4 recently, they did handle quite large budgets, but they were also an educational charity. One of the governor's refrained (paraphrasing) "they must be totally immoral to target us"; I very much doubt they were targeted beyond 'can we exploit this box'.
londons_explore 5y ago
I think a big part of the issue is that backup processes are designed for a small number of machines failing. The the IT department restores data from backup, and manually adjusts the config and software till stuff is working. That process works well till ransomware comes in and destroys every server and client machine at once, and suddenly you've just given the IT department multiple years worth of work to all do at an emergency pace.
scruffyherder 5y ago
If only virtual machines and networks were a thing.
KronisLV 5y ago
> There are a lot of shops that probably don't know how to recreate a machine from scratch. How many systems are developed as balls of clay. Little bits added and smeared in over time until the ball just gets bigger, but each piece lost in the process. How many folks can go through their local config files and explain all of entries, how many can even tell which ones they have changed, or why? Especially when they were changed by Frank, but he left 2 years ago. This is a non-issue with Git + Ansible, even without getting into tools like Terraform. At my dayjob, i set it up so that there's an Ansible playbook that does around 200 administrative tasks for each of the servers for a particular environment - all of the configuration is within Git repositories. Changes are applied by CI, all of the process also being thoroughly documented for local development, if necessary. Everything from installing packages, creating or removing user accounts, setting up firewall rules, setting up directories and sending the necessary configuration, systemd services for legacy stuff, container clusters, container deployments, monitoring, container registries, analytics, APM and so on are handled this way. Noone has write access on the server itself (unless explicitly given in the playbook, or the admins) and even the servers themselves can be wiped and reinstalled with a newer OS version (mostly thanks to Docker, in which the apps reside in), plus all of the changes are auditable, since they coincide with the Git repo history. It took me maybe 2 weeks to get that up and running, another 2 to handle all of the containerization and utility software aspects. I'm not even that knowledgeable or well paid, there's very few excuses for running setups that don't let you do things "the right way" in a somewhat easy manner, like Windows Server. That's like picking a hammer when you need to screw in a screw - it'll do the job but chances are that there will be serious drawbacks.
mannykannot 5y ago
All this "but it's so hard" whining should be balanced against one simple fact: If you get hit by a ransomware attack, you will be trusting the attackers to restore your system.
- dspillett 5y ago
  Yep. And if it is so hard to do, how come the attackers can restore your systems when you pay them?
stonemetal12 5y ago
A first step would be restore a backup, then check md5s of your executables\data, after all you are testing the backups not system functionality. A second step would be to run automated component level tests, or data integrity tests on the restore to verify that hashes aren't lying to you. The primary problem you are protecting against is that backups are broken, or corrupted. A broken, as in failure to write, backup won't restore. A corrupted backup probably wouldn't restore either, even if it did it would fail the MD5 checks. No one is expecting you to destroy and repair Prod on a weekly basis.
bluGill 5y ago
So you do it anyway! The less sure you are that you can do this, the more important it is to do it now: you are more likely to remember the change that was made (and that person is more like to be around) 30 years ago mainframe companies started realizing that their mainframes couldn't restart anymore - after many years of uptime all the on the fly configuration changes wouldn't be reapplied and so the whole couldn't restart. (all hardware had redundancies and backup power supplies, so any individual component could be replaced and most had been over time) So they started scheduling twice a year restarts to test that the whole could come back up. The mainframe itself is fully able to run for years without the restart, but the configuration wasn't.
tinus_hn 5y ago
If your systems are in this state better pray you never experience hardware failure.

xwdv 5y ago

This is like a sales pitch for just paying the ransom.

MattGaiser 5y ago
It probably works more often than not.
- shyn3 5y ago
  It works until it doesn't, such as in a hardware failure. [1] https://www.anandtech.com/show/15673/dell-hpe-updates-for-40k-hour-essd-failures https://www.anandtech.com/show/15673/dell-hpe-updates-for-40...
- quickthrower2 5y ago
  It's a prisoners dilemma
- xwdv 5y ago
  Seriously, I started running through all those points he made and started thinking about other things he didn't mention and then halfway I was just like fuck it.

tw04 5y ago

This is all solved, it just takes money and typically bringing in outside experts. Occasionally it will require changes to apps but most of the time it can be retrofit. No it isn’t easy, but it’s also not an impossible task.

foxpurple 5y ago
I think easy or hard are not even appropriate terms. As a business owner it is not easy or hard, it just costs an amount of money. If you pay enough money, you can get the result. You just have to decide how much a disaster costs and how much preventing a disaster costs to work out if it’s worth it.
sneak 5y ago
It doesn't even require outside experts, just one or two competent sysadmins on staff.

briffle 5y ago

Honestly, this is one of the problems the cloud is great at solving. We keep things in seprate projects, and restore a backup from one production project (Kubernetes, Database servers, etc) to a special DR project we have set aside. The only step we do not do is update the front end DNS, and then run our tests after our Infrastrucutre as Code deployment is complete.

rhizome 5y ago
So a complete hot standby?
- Godel_unicode 5y ago
  Sounds like a DR test; spin up an entire new thing to make sure the backups worked, then turn it off. Costs a few hours of extra resources.
  - dspillett 5y ago
    I do something similar with my home mail server (not sure I'd go with self-hosting if I started fresh today, but it has run without issue other than maintenance for upgrades & such for well over a decade). I have a second copy running in a low spec VM. Every day it wipes itself down and restores the latest backup, I check regularly (a couple of times per week) to see that it is running and has copies of recent mail (that part could be more automated but I've never go round to it...). It wasn't setup for DR specifically, but it is on a separate network so if push came to shove I could ramp up its resources, change a few DNS settings, open up relevant bits of its firewalling, and have it take over, if the main server failed in a way it hadn't copied yet.
  - rhizome 5y ago
    Could be; it's ambiguous. But yeah, not updating DNS could be just declining to test operation in the wilds of outside traffic before tearing it all down. That said, in this way, maybe a periodic "DR" that actually replaces the current operations would be a helpful...well, not test, but..."resilience practice" maybe? It could be a new twist on continuous deployment: continuous recovery.

nn3 5y ago

> How do you test that in isolation? Is it easier to test after the ransomware attack?

bsder 5y ago
Yes, actually. Before the attack, all machines are active and being used. If you screw one of them up that happens to be running something obscure but vital, the screwup is your fault. After the attack, all machines are dead so the screwup is blamed on somebody else. If you have the email/memo (you did print it out and file it, did you not?) showing that you informed the CTO/CIO, blame for the screwup will get buried.
whartung 5y ago
Actually it is easier because you get to "test" it by reloading on the actual hardware, not separate hardware. To do a solid test, you need to restore the system. It's difficult to restore a running system because, well, it's running. That typically means a parallel environment. A single box represents a bunch of "magic values" that are stuffed in some config somewhere. Imagine several of those. "We need to restore SQL Server, the AD Server, the Application Server..." Reloading on a new environment is an easy way to find out those magic numbers, typically the hard way. Restoring on your existing hardware, with existing networks, existing IPs, etc. you're laying your software and configs over a known, working environment. How do you test a recovery of licensed software thats bound to the machine that you have running in production, for example? "Oh, sorry, you have a duplicate license detected" and it shuts down your production system for a license violation. "Sorry, we detect the wrong number of cores", or whatever other horrors modern licensing systems make you jump through. You DO have another dongle, right? (Do they still use dongles?) It can be far easier to do an image restore to your already working system than trying to load it up on something else. Since your production box is horked anyway, an image restore should "just work". But testing it, that's another story completely.
- Godel_unicode 5y ago
  Who is configuring physical servers by hand. Are you a part of some kind of museum exhibit? In all seriousness, you're talking about a business which is carrying a truly staggering amount of risk. If the margins are so tight they're running hand-rolled physical servers then restoration is a moot point. They'll go out of business if attacked. If you're not testing your backups, then they're broken with close to 100% certainty.

tomc1985 5y ago

God these "we're so ignorant!" responses are sooooo tiring. If people don't want to hire the hardcore nerds that know this shit, if they don't want to pay for the expertise to work the technology they build their business on, then maybe they deserve it. This modern attitude of compartmentalization, extremely localized expertise, and outsourcing everything to the cloud is going to be our downfall. Stop hiring muggles!

jiggawatts 5y ago
"No." Says a million finance people that prefer armies of compliant muggles over a few self-important wizards.
flyinglizard 5y ago
You can’t just man every brick and mortar out there with “hardcore nerds”. There simply isn’t enough of them.
g_p 5y ago
It strikes me that companies need to look back at covid and consider the value IT provided to their business during that period, and the extent to which it has grown their business since the 90s. Then take that and allocate a reasonable chunk of that growth (say 15%) to ensuring that can continue, through ongoing investment in IT. Their alternative is to abandon the internet, go back to paper, and dump their IT costs (and boost to business). Unfortunately though, as long as there are bean counters trying to cut every outgoing and outsource every cost, and play accounting bingo to turn everything into monthly opex, it's unlikely to change. Most companies need skilled technical people because the sheer aggregation of risk in a few outsourced providers (see Kaseya recently) shows that they won't be top priority when something hits the fan. If they want to be top priority, they need the people on payroll and on site. Not everyone needs a group of rockstar 10x-types, but we do definitely need better fundamental IT knowledge and ability to solve problems. And business needs to make clear this is what's needed in terms of job adverts and compensation - supply tends to rapidly learn to deliver what's valued... If you can convince bean counters to pay anything for it, that is...
- 5y ago

kenniskrag 5y ago

I provide a test system in a vm. I have an installation script for the db. Dump the db and replace paths, import, start application, done. This can then be used to test the new release before deployment.

kenniskrag 5y ago
bonus: the vm can be used for debugging or used as playground
- ashtonkem 5y ago
  Be mindful of what customer data you’re handling though.
  - handrous 5y ago
    Underrated problem of testing backups, generally. Making it easy/common can also mean greater exposure to risk of data leaks, if you don't take great care.

MeinBlutIstBlau 5y ago

You don't even need system images. Just the data. So SQL backups and just file directories. However for a company of our size, it's not really possible. I was talking to my lead about this last year and we have about 50TB of our entire system I believe stored in our databases. All he said was "we have one, but hopefully I'm retired before needing to find out how to restore from it."

emodendroket 5y ago

Well, that's why you pay more for PaaS than hosting it yourself, right? You can make it someone else's problem, to an extent.

paxys 5y ago

Every business in the world hires accountants to balance their books, lawyers when they need to file paperwork or respond to a problem, mechanics, electricians, plumbers and other professionals to fix physical stuff. Yet when it comes to software the most they will consider is an online monthly fee for something cheap and out of the box, and when things get complicated or go wrong "this is out of our area of expertise" is always the excuse. It's really time to treat software professionals, especially when related to security, as a core requirement for EVERY organization, starting from a 2 person mom-and-pop shop.

908B64B197 5y ago
> It's really time to treat software professionals, especially when related to security, as a core requirement for EVERY organization Some organizations do, some don't. Long time valuation seems to agree with the former, short term the latter!
htrp 5y ago
Those other groups have professional organizations with licensing. Plumbers will almost always refuse to do maintenance work on any work done by an unlicensed person. .... software/IT just hasn't had people willing to do that.
- nitwit005 5y ago
  If all you need is "person who is able to properly administer Windows systems", there are things like Microsoft's certifications.

jms703 5y ago

A failure to invest in business continuity and maintaining your systems will cost you one way or the other. Hard drive failure, fire, flood, lightning strike, or ransomeware. These aren’t new problems facing businesses.

deleted 5y ago

[deleted]

z3t4 5y ago

Testing the backups is harder then you think... It's not like you are going to double up your entire server fleet just to see if you can restore everything from backup. You maybe test restoration of one or two servers and then assume the rest will also work. And you probably have some redundancy so that the data is saved on different machines plus a backup plus an additional backup. Then if things goes down, which is very, very unlikely, like a coordinated nuclear attack on several data-centers, but of course can happen, you assume you will figure things out... And then someone will ask: so what happens if your team also die in the nuclear attack, then you add one zero to the backup price/cost estimation.

dougpa 5y ago
Finding something reasonable between "we're at risk for a nuclear attack" (a black swan event) and "we're at risk for having our root credentials exfiltrated" (a daily occurrence) is not hard. I feel like these kinds of dramatizing hyperboles are nowhere near the nuts & bolts of the situation. And honestly, why not double up your entire server fleet for a temporary build-from-scratch rehearsal? Many shops could quintuple their infrastructural costs and still sit far above being in red. Most software enterprises in the modern day don't reap economic value as a function of how well they can convert hardware resources, but human resources. Optimizing on infra costs is not a main priority for any shop I've seen. I imagine not even for an IaaS provider these days.

newsyyswen 5y ago

Many places have a separate "test" or "beta" environment, with less resources and maybe a small database with spoofed entries. Maybe we should get into the habit of going all Shiva/Brahma on that environment every week or month. Burn it to the ground, and recreate it with an automated process. Sort of like a meta CI test.

silly-silly 5y ago
Beautiful resiliance.

mycall 5y ago

> There are a lot of shops that probably don't know how to recreate a machine from scratch. You can't fix already broken processes. VMware solved this 20 years ago. It is pretty simple to restore VMs on different systems, you don't need to worry about the ball of mud when you can duplicate it.

dougpa 5y ago
You can't really serialize something like an ami. So how are you going to make an offsite backup? Things need to be relatively simple & reproducible otherwise you will get bitten in many different ways due to strategies like this.
- nuker 5y ago
  > You can't really serialize something like an ami. Copy AMI to separate AWS account, not in your Org, and keep keys to that account offline.
  - dougpa 5y ago
    oooo true, that does invalidate my point. AMI's are very easy to copy-over across accounts, i.e. to a potentially firewalled account.
- Godel_unicode 5y ago
  If you want to backup an individual AMI you're probably doing it wrong. That probability goes to near certainty when you're talking about serializing it for off-site backup. Backup the deployment automation and the data, sure.
  - dougpa 5y ago
    I agree, that was kind of my point. He was talking about taking VM's but in a cloud environment this is a bit more awkward. Copying and storing VM's securely is not hard, but transferring AMI's is the only rough equivalent I know of in the cloud world. Ideally, you don't have to do this. But for one part of my current stack, the configuration that this specially configured windows box has been lost for a while. Rebuilding from scratch has not worked each time it has been tried.

dylan604 5y ago

That's the point of running these tests though. Take a system down (by turning it off, and putting it safely in the corner). Then bring up a new machine to replace it. Go through the steps you think would be the proper procedure, then document the shit out of what didn't work and what needed to be done. Write that up so that it becomes the new procedure. The next time you run this quarterly test, you start with the latest procedures. Updating accordingly, rinse, repeat.

galuano1 5y ago
When you work with the old systems, you are always afraid to shutdown the machine, because more often than not, it will not boot up. And if you petition to replace it with a new machine, you may very likely hear the phrase "We are planning to move the app to cloud, so no need to build a backup in the meanwhile".
- 908B64B197 5y ago
  "What if it shuts down tomorrow?"
- whartung 5y ago
  Back in the day, we had purchased a new, bigger machine, and transferred over, and everything was just peachy. Months later, we had a power outage (scheduled I think, I don't recall). Anyway, at some point during the transition and such, I managed to have the machines hard mount NFS across each other. As long as one of the machines is up, everything is rosy. But cold start? They were both hanging while restoring the mount (which, they couldn't because neither was "up"). Took us about 1/2 hr to suss out what was happening and get single user in to tweak it, but...yea...exciting! "Smart" people do this kind of innocent stuff all the time.

joe_the_user 5y ago

"Test your backups" is so easy to say, but quite difficult for many to do. It's difficult to do, yes. But it's difficult in a "dig a tunnel through that mountain" way, not in a "solve this mysterious math problem" way. It certainly could be done. It would just take time and money (money including hiring people). People constantly point to the difficulty of backups and the difficulty of hardware level separations between systems. But these are merely difficult and costly. "No being hacked" and "writing always secure code" are impossible and so they won't protect from backup failure. And yes, companies would rather spend money on other things. That's what it come down to.

irishsultan 5y ago
It's interesting that you make that comparison, because I would choose the mysterious math problem every time. Probability of success is about equal, but it's far less dangerous to do math than to cut through a mountain.
- pnt12 5y ago
  Nitpicking an analogy goes against the point - it simplifies some ideas at the cost of modifying the details. I think OP meant to compare a known hard task vs an unknown hard task. Both are hard, but first is known to be possible to finish, but the other not so.
  - irishsultan 5y ago
    I agree that's what he meant, but it's just interesting to me that for the known hard task he took something that also comes with danger. Exactly the feeling that I get when I think about restoring backups.
    - mannykannot 5y ago
      Well, one of the things about testing backups is that (if you are doing it remotely right) it reduces the danger when you need to actually restore. A widely accepted method fur judging at least subjective risk when there are insufficient facts for truly objective measure is to ask how much you would bet on various outcomes - such as whether P=NP is solved before New York's 2nd. Ave. subway is completed.

Spooky23 5y ago

No, it requires baseline professional competence. I’ve worked with 9-figure turnover entities broken by this sort of thing and the first recommendation is always fire or manage out the CIO, risk/audit officer and/or CFO. Everyone cries about having no money. What is lacking is an ability to identify and manage risk that puts the existence of the company as a going concern at risk.

snicker7 5y ago

I think what you are describing is a perfect advert for declarative/immutable infrastructure. Yes, it may require work and talent. But that's the price to pay for resiliency. Invest and modernize your tech stack or go bankrupt. Cloud tech can help. As can Guix/Nix.

Robotbeat 5y ago

You should definitely try. Bare metal backups are a good place to start if you have a complicated system. A lot of environments aren't as complicated, and you COULD just pull your hard drive out, put in a spare, and test how your backup works. At least, test it while airgapped to see if it comes up at all.

dheera 5y ago

Also, ransom gangs can take over your backups. Keep offline, airgapped backups.

EVa5I7bHFq9mnYK 5y ago

Ransomware gangs go after large organizations, those that can afford testing backups, not after pop and mom coffee shops.

amelius 5y ago
Is this true? I think big organizations just end up in the news more often.
- yodelshady 5y ago
  Third-hand evidence, mom n' pop shops don't have the expertise to pay in bitcoin. Or perhaps, more charitably, they've been inoculated by previous low-effort scams and smartly assume bitcoin payments are another. Honestly I'm not sure many large IT depts have it either, but for $1M+ the attackers can afford good customer service.
pbhjpbhj 5y ago
There was a school group featured in a report on BBC Radio 4 recently, they did handle quite large budgets, but they were also an educational charity. One of the governor's refrained (paraphrasing) "they must be totally immoral to target us"; I very much doubt they were targeted beyond 'can we exploit this box'.

londons_explore 5y ago

I think a big part of the issue is that backup processes are designed for a small number of machines failing. The the IT department restores data from backup, and manually adjusts the config and software till stuff is working. That process works well till ransomware comes in and destroys every server and client machine at once, and suddenly you've just given the IT department multiple years worth of work to all do at an emergency pace.

scruffyherder 5y ago

If only virtual machines and networks were a thing.

KronisLV 5y ago

> There are a lot of shops that probably don't know how to recreate a machine from scratch. How many systems are developed as balls of clay. Little bits added and smeared in over time until the ball just gets bigger, but each piece lost in the process. How many folks can go through their local config files and explain all of entries, how many can even tell which ones they have changed, or why? Especially when they were changed by Frank, but he left 2 years ago. This is a non-issue with Git + Ansible, even without getting into tools like Terraform. At my dayjob, i set it up so that there's an Ansible playbook that does around 200 administrative tasks for each of the servers for a particular environment - all of the configuration is within Git repositories. Changes are applied by CI, all of the process also being thoroughly documented for local development, if necessary. Everything from installing packages, creating or removing user accounts, setting up firewall rules, setting up directories and sending the necessary configuration, systemd services for legacy stuff, container clusters, container deployments, monitoring, container registries, analytics, APM and so on are handled this way. Noone has write access on the server itself (unless explicitly given in the playbook, or the admins) and even the servers themselves can be wiped and reinstalled with a newer OS version (mostly thanks to Docker, in which the apps reside in), plus all of the changes are auditable, since they coincide with the Git repo history. It took me maybe 2 weeks to get that up and running, another 2 to handle all of the containerization and utility software aspects. I'm not even that knowledgeable or well paid, there's very few excuses for running setups that don't let you do things "the right way" in a somewhat easy manner, like Windows Server. That's like picking a hammer when you need to screw in a screw - it'll do the job but chances are that there will be serious drawbacks.

mannykannot 5y ago

All this "but it's so hard" whining should be balanced against one simple fact: If you get hit by a ransomware attack, you will be trusting the attackers to restore your system.

dspillett 5y ago
Yep. And if it is so hard to do, how come the attackers can restore your systems when you pay them?

stonemetal12 5y ago

A first step would be restore a backup, then check md5s of your executables\data, after all you are testing the backups not system functionality. A second step would be to run automated component level tests, or data integrity tests on the restore to verify that hashes aren't lying to you. The primary problem you are protecting against is that backups are broken, or corrupted. A broken, as in failure to write, backup won't restore. A corrupted backup probably wouldn't restore either, even if it did it would fail the MD5 checks. No one is expecting you to destroy and repair Prod on a weekly basis.

bluGill 5y ago

So you do it anyway! The less sure you are that you can do this, the more important it is to do it now: you are more likely to remember the change that was made (and that person is more like to be around) 30 years ago mainframe companies started realizing that their mainframes couldn't restart anymore - after many years of uptime all the on the fly configuration changes wouldn't be reapplied and so the whole couldn't restart. (all hardware had redundancies and backup power supplies, so any individual component could be replaced and most had been over time) So they started scheduling twice a year restarts to test that the whole could come back up. The mainframe itself is fully able to run for years without the restart, but the configuration wasn't.

tinus_hn 5y ago

If your systems are in this state better pray you never experience hardware failure.