44 ms·

Don’t Wanna Pay Ransom Gangs? Test Your Backups

599 points by parsecs 5y ago

LinuxBender 5y ago
Don't just test your backups. Make sure your automation can't clobber or tamper with your backups. This includes both local and disaster recovery sites. Give your pen-test team super-user privs on your automation and give them Amazon gift cards if they can tamper with your backups. If they can't mess with the backups, give the gift cards to whoever designed and hardened your infrastructure.
- wizzwizz4 5y ago
  Why not actual money? Amazon gift cards leak metadata to Amazon, and can only be used to buy stuff from Amazon.
  - jagged-chisel 5y ago
    And they support Amazon.
    - wizzwizz4 5y ago
      Oh, of course. Can't believe I forgot the biggest reason.
  - LinuxBender 5y ago
    Good point. Cash bonus and maybe RSU's if they company is public.
  - jonas21 5y ago
    It used to be that you could give employees gift cards up to a certain amount as awards and it would not be considered taxable income (but I believe that's no longer the case).
    - jagged-chisel 5y ago
      Any gift(s) up to a total value of ... $13k? -ish? I don't know what the limit is now. Google's cafeteria is (was? depending on that limit...) an example of how to benefit employees without causing the employee additional tax.
      - mikeyouse 5y ago
        That's a different issue - IRS clamped down on gift cards and non-cash compensation that used to be considered de minimis. Now most employers gross up and report any gift card type gift over ~$5. https://www.irs.gov/government-entities/federal-state-local-governments/de-minimis-fringe-benefits https://www.irs.gov/government-entities/federal-state-local-...
      - dmoy 5y ago
        Setting aside the gift card bit (addressed in above comment), $13k sounds way too high. Like two orders of magnitude too high. From irs.gov > Whether an item or service is de minimis depends on all the facts and circumstances. In addition, if a benefit is too large to be considered de minimis, the entire value of the benefit is taxable to the employee, not just the excess over a designated de minimis amount. The IRS has ruled previously in a particular case that items with a value exceeding $100 could not be considered de minimis, even under unusual circumstances. Which about matches with what I've seen at BigCo. $40 box of tools as a gift? Did not show up on my paycheck. $150 electronic device as a gift? Showed up on my paycheck.
        mikeyouse 5y ago
        There's another about other fringe benefits being taxed - with a prime example being tech cafeterias. https://taxnews.ey.com/news/2019-0493-employer-must-substantiate-necessity-of-employer-provided-meals-for-value-to-be-excluded-from-employee-income-irs-tam-emphasizes https://taxnews.ey.com/news/2019-0493-employer-must-substant... In the past few years, guidance has shifted toward accounting for employer-provided food with employee income as well.
  - bentcorner 5y ago
    I think logistically its easier for a team within an org to spend "their" money on gift cards for intermittent activities and hand them out as necessary. Getting stuff onto the actual payroll is probably more complicated.
    - edoceo 5y ago
      Hey Payroll, edoceo needs an off-cycle bonus of $$$. Your manager should be able to write a similar email.
      - Volundr 5y ago
        At least at the company I recently left, this kicks off an approval process within both the HR and accounting departments. Meanwhile an Amazon purchase (and thus an Amazon gift card) is something I could put on my card and expense, or approve someone else doing myself. I get it doesn't make sense, but that's corporate America for you. That said, be careful of the gift card route. Depending on the amount you can find yourself in the wrong side of the IRS that way.
        gowld 5y ago
        I wouldn't want to embezzle funds and commit income/payroll tax fraud just to bypass paperwork
      - rad_gruchalski 5y ago
        It doesn’t work like that in every place in the world. Those gift cards also aren’t that easy. It’s all taxable benefit to the employee and a cost for the company to put on the books. That needs to be justified and tax office may really slap you on the wrist for doing so.
      - Smoosh 5y ago
        Forget the backups - the pen test team can just produce fake emails requesting they get the $$$.
      - benhurmarcel 5y ago
- renewiltord 5y ago
  Which organizations currently do this?
jerhewet 5y ago
Isn't this more of a "We don't want our client / customer information released to The World At Large" question? I would think most business entities have backups of some kind (Scripps being the only exception I can think of), and will pay the ransom to keep any sensitive information off the market. Edit: Should have added that I find it hard to believe that companies have PB of data backed up. I could believe GB, and maybe even TB, but PB is pretty hard to swallow. The past three companies I've worked for (25 year span) had, at most, a couple of gigs of sensitive information that couldn't be easily replicated.
- stan_rogers 5y ago
  Ransomware attacks rarely indicate any data leakage; all they usually do is prevent you from accessing your own data (by encrypting your drive with a key you don't have access to).
  - intothev01d 5y ago
    O rly? https://www.trendmicro.com/vinfo/us/security/news/cybercrime-and-digital-threats/modern-ransomwares-double-extortion-tactics-and-how-to-protect-enterprises-against-them https://www.trendmicro.com/vinfo/us/security/news/cybercrime...
  - runnerup 5y ago
    These days attacks labeled “ransomware” in the news seem to be hybrid attacks. There usually is sensitive data exfiltration in addition to encrypt-in-place.
  - djrogers 5y ago
    The current trend is double-extortion ransomware attacks - encrypt your copy of your data, and threaten to release it publicly as well. [1] https://www.cybereason.com/blog/rise-of-double-extortion-shines-spotlight-on-ransomware-prevention https://www.cybereason.com/blog/rise-of-double-extortion-shi...
- nelgaard 5y ago
  I also find it hard to believe that a ransomware gang could encrypt 50 Petabytes without anyone noticing it. It would also take some time to decrypt 50 petabytes if you paid the criminals and got the key. And would you trust you data after criminals had access to it?
nickdothutton 5y ago
Not really just a backup and restore. You need to be able to rebuild from zero. I think of it more as a disaster recovery exercise, and for those… you are only as good as your last _real_ rehearsal. That may mean a suitcase of tapes, a sheet of paper, and a rack of blank servers. Then you have the problem of release of confidential information. For this reason, the sweetest target for ransomware is the company who can neither recover their data, nor can they afford to have it publicly posted or monetised by the gang. Oh and you do store those backups offline dont you? Ransomware gangs have been known to loiter and observe their target for weeks to learn how to sabotage backups when the time comes.
user3939382 5y ago
What sucks for HIPAA is that you can get fined for the breach itself, regardless of your backup management.
- edoceo 5y ago
  Not really a problem with HIPPA is it?
- MattGaiser 5y ago
  Seems appropriate.
slownews45 5y ago
There is another approach. Scrub old data you don't need. 2-3 year email retention on corp email. Paper files for sensitive client info (or don't keep it). We can reinstall office / windows / active director etc. Mandatory 2FA on google suite? Git codebases on github etc for LOB apps (we can rebuild and redeploy). We use the lock features in S3 for copies of data that must be kept. Not sure I can even unlock to delete as account owner without waiting for timeouts.
- orf 5y ago
  You can also enable versioning and deny all s3:DeleteObject actions via the bucket policy. It won’t stop a root account compromise unless you’ve got a multi-account setup going (as they could edit the bucket policy). But if you’ve not got any monitoring, they could also just remove the lock on the objects without you noticing and wait for the timeout themselves.
- TrueDuality 5y ago
  A lot of sensitive data is legally required to be kept around for much longer than that (email is up to 15 years if you're a publicly traded company). Most of the remaining suggestions aren't relevant to ransomware even if they're otherwise mostly fine recommendations. 2FA won't stop ransomware, or data destruction. Redeploying code and reinstalling active directory doesn't restore customer or user databases. Paper files are not accessible when they're needed, are easily damaged or misplaced, and cost a lot to store and index (if you're referring to keeping them as backups then yes you're making a form of the argument of the post just in a very expensive and inconvenient way). Read-only S3 copies are almost certainly falling in the realm of backups... but is also a relatively expensive way to do it for most organizations larger than a start-up. Offline and offsite backups are the cheapest, most effective tool for keeping copies of your data in your companies possession due to unforeseen circumstances and they protect against a huge number of potential events beyond just ransomware. It's negligent IMO for executive officers of a company to not have invested in a solid, tested, and comprehensive backup solution.
  - allset_ 5y ago
    Most attack campaigns start with compromised credentials, so MFA absolutely helps prevent ransomware.
Severian 5y ago
3-2-1 Backup Rule: Three copies of your data. Two "local" but on different mediums (disk/tape, disk/object storage), and at least one copy offsite. Then yes, absolutely perform a recovery and see how long it takes. RTOs need to be really low. Recovering from object storage is going to take at least a magnitude more time than on-prem. Also, storage snapshots/replications are not backups, stop using them as such. Replicating is good for instant failover, but if your environment is hacked they are probably going to be destroyed as well.
Waterluvian 5y ago
I’m a novice and am dealing with data that isn’t too complicated, large, or important. My approach is to build restore directly into the normal workflow. I test my backups by using them each week. A stack is spawned from a database backup and once it passes tests, replaces the previous one. Not sure how smart this all is but my goal is to learn through application.
- TheDong 5y ago
  The main reason I think this normally isn't done is that it requires downtime to do safely most of the time. In order to not lose data, you can't have any writes between the time when the backup was taken and the present, or you need code which reconciles additional state and adds it onto the backup before switching over. Normally, backup restoration is done during a maintenance window where the site is disabled so no writes can happen, and then usually a window of writes are lost anyway (i.e. 'last X hours, since the backup was taken') For your use-case, do you just have very few writes? Do you lose writes? Do you have some other clever strategy to deal with it?
  - dragontamer 5y ago
    It should be noted that not everyone is a global company. A typical bank / credit union may only serve one town. As such, it would be socially acceptable to designate 3am to 4am as a regular maintenance window where services are shutdown.
  - Waterluvian 5y ago
    Good point. The 5 minutes of downtime is simply tolerated. My captive audience are dozens of humans and thousands of robots all willing to try again.
    - mlac 5y ago
      Ticketmaster?
- innomatics 5y ago
  >A stack is spawned from a database backup and once it passes tests, replaces the previous one. I like this approach, although risky if you mean you routinely replace the production db. My preferred setup is to automate restores into a pre-prod environment, apply data masking and run tests there. It's not a replacement for full DR exercises, but at least it automates the backup verification process as part of your build system.
whartung 5y ago
"Test your backups" is so easy to say, but quite difficult for many to do. There are a lot of shops that probably don't know how to recreate a machine from scratch. How many systems are developed as balls of clay. Little bits added and smeared in over time until the ball just gets bigger, but each piece lost in the process. How many folks can go through their local config files and explain all of entries, how many can even tell which ones they have changed, or why? Especially when they were changed by Frank, but he left 2 years ago. You'd like to think you can just restore the system from backup and it'll just light back up. But how do you test this without cratering your existing system? Like a boat in a basement, many system are built in-situ and can be very rigid. Modern environments like cloud computing and creation scripts can mitigate this a bit organically, but how many of these systems are just a tower running Windows w/SQL Server and who knows what else? Plus whatever client software is on the client machines. How do you test that in isolation? At least read the media to see if it can be read (who doesn't love watching a backup tape fail halfway through the restore). Simply, it takes a lot of engineering to make a system that can be reliably restored, much less on a frequent basis. And this is all engineering that doesn't impact the actual project -- getting features to users and empowering the business. Which can make the task even more difficult.
- xwdv 5y ago
  This is like a sales pitch for just paying the ransom.
  - MattGaiser 5y ago
    It probably works more often than not.
    - shyn3 5y ago
      It works until it doesn't, such as in a hardware failure. [1] https://www.anandtech.com/show/15673/dell-hpe-updates-for-40k-hour-essd-failures https://www.anandtech.com/show/15673/dell-hpe-updates-for-40...
    - quickthrower2 5y ago
decebalus1 5y ago
If your disaster recovery process isn't tested, you actually don't have any disaster recovery. It's not only about 'how long it takes' it's also about whether or not it works at all. Can you rebuild from scratch? What happens if your entire infrastructure goes down at the same time? What happens if a datacenter you rely on just disappears? What happens if you lose access to your systems? Can you lose access to your systems? IMHO one of the only silver lining of these attacks is that organizations are starting to ask these questions more often.
PhantomGremlin 5y ago
There's been a lot of good advice here about backups and disaster recovery. But there's also a lot of other stuff to consider: Compartmentalization. Finance and Engineering and Sales only need to interact in limited ways. How about some firewalls between them, limiting types of access? Location isolation. Why does something that happens in Peoria affect Tuscaloosa? Once a ransomware gang breaches a perimeter, why is it allowed countrywide (or worldwide) access to a company? Monitoring. Aren't there tools that can alert on various anomalous patterns? All of a sudden, gigabytes of data start being exfiltrated? All of a sudden, processes fire up on a multitude of servers? Monitoring these things is hard to do at scale, but surely possible? Microsoft. In 2002, Bill Gates "Finally Discovers Security". How much longer will Microsoft be given a free pass? How many more "critical" vulnerabilities will their software have? https://www.wired.com/2002/01/gates-finally-discovers-security/ https://www.wired.com/2002/01/gates-finally-discovers-securi... I could go on and on. But why should I? Why can't MBA-type CEOs take IT seriously? Why can't they hire competent people and fund them and listen to them?
- blooalien 5y ago
  > … "and listen to them?" That's the part I've always had trouble gettin' out of most "management" types. They hire you for your expertise, and then undermine it at every opportunity to "save money" or to exert their "authoritah".
  - g_p 5y ago
    There's significant misalignment of incentives at play here (irrespective of company, generally). Many "management types" are measured solely and almost entirely by the bottom line - the share price in a traded company, or profits in a private company. Share price is based usually on profits. Usually they're delivering reporting to their boss or the board quarterly. Every dime they save this quarter is "profit" in their eyes. Obviously there's a line here - you don't want to save the dime that causes the factory to burst into flames and result in 4 months of production downtime. But if that dime can be saved this quarter when times are tough, you'll be seen as a well-performing hero, and next quarter you'll spend a few dimes getting the sprinkler system inspected... Until you need the boost to profits next quarter too(!) In management and generally in business it's hard to go backwards. Nobody wants to increase their spend on IT unless they are making more money. If they could sign a new deal this quarter, they'll probably give you a decent percentage of that deal as IT budget to get the deal signed (factoring in the risk of the customer not signing). But in an environment focused almost entirely on pursuit of unrelenting growth of profit or revenue or share price, security simply won't become a priority until you can convince manager-types that the issue is commercial and measurable and that it impacts the bottom line. Even if the issue is unexpected costs of recovery from a breach, the first questions they'll ask are "How likely is it to happen? What will it cost? Can we insure against it? What do our competitors do?" - it's not about preventing the breach, it's about defraying the potential loss of profits without compromising on growth or profits when it isn't a problem. Hence you'll see people hired to fill roles (new or existing) then being hamstrung by a total and outright refusal to act, because it isn't a commercial problem. Skilled tech leaders are good at turning the problems into language leaders can understand, but there is a limit and a line, and the solution in my view is clear regulation the leaders can "get", involving individual penalties and obligations of competency - even if your core business isn't safety, you have safety obligations as a business to your employees, contractors and the public. It's expected and required that you become suitably competent to do this, or get the expertise to do so, but with you still liable. We need the same for security (in my view). It's not all bad news - I'm a "management type", but from an almost entirely technical background. Bean counting isn't my style... Maybe we can infiltrate more organisations and bring some basic engineering understanding to decision making? It's frustrating to see most hide behind MBA-waffle though rather than try to actually do real things that make a difference.
throwawaysleep 5y ago
How does one learn how to do proper backups? Using my throwaway as I suspect my company doesn't do them (and even if they do, I don't know where they are or what to do with them as the main engineer left on my piece of software).
- dreadlordbone 5y ago
  That entirely depends on how your infrastructure is set up. If it's in AWS, just set up rolling snapshots of your EC2's and databases. If you can store everything in Github, that works too. If it's on local machines, something like Backblaze is cheap and easy to set up.
- deleted 5y ago
  [deleted]
- g_p 5y ago
  I'm not a backup specialist, but do run my own backup infrastructure, so a few starting points. Not saying they will work in all environments, or even any scale environment though. 1. Figure out what's valuable and what needs backed up. Some will say "back up everything", but then you don't know what you need to restore and how to restore it! You need to know what's valuable. Is that source code? A complex application environment? Files in S3 storage? User generated files stored locally on a server? Data in a database? 2. Figure out where it's stored. Not always straightforward, but again necessary. Otherwise you won't know how to back it up, or recover it. Remember the cloud isn't a backup - when someone steals your production S3 keys, absent some careful configuration, they can dump and delete all your bucket contents... Understand what you need to backup and how you'll recover it. If you are dealing with Windows based environments, consider the practicalities of getting these up again - windows isn't as nice as Linux (which can boot on any hardware within reason, modulo some bootloader settings in a few complex configurations) 3. Figure out how you're going to protect the data you back up. Now you've identified the crown jewels, don't just store them in plaintext and unencrypted on your laptop (!) If you're going to use encrypted backups, understand where the keys are stored and how they're used. More importantly, understand how you'll have the keys to decrypt the backups after a disaster. 4. Figure out how you'll stop a someone who gets in from deleting or accessing your backups. Assume they'll be attacked. Understand what the backup solutions offer in terms of security. Can you use asymmetric crypto so only public keys get stored on your servers? Can you use protection on an s3 storage bucket to prevent deletion, even with the access key? Etc. 5. Test doing backups. Then practice restoring to a non production environment to see what you missed. Document it all! 6. Set up monitoring so you know when something goes wrong with your backups. Gather and expose enough information to yourself that you know what's going on! This could be as simple as an automated email to yourself every morning, telling you the duration of the backup job, and the amount of data written, and amount changed since last time. But you need to monitor this. Don't just go for a failure alarm (have one of those, and make it fail-safe so it alerts if the backup didn't succeed, even if the backup script didn't run at all!) - also notify yourself on success so you are aware of the backups and what goes on. 7. Ensure you have enough copies. 1 isn't enough - what if the payment card on your storage account expires, or you get locked out? Or the company goes under? People normally say 3 backups in 2 locations, at least 1 off-premises. 8. Return to 1, because by now something has changed and you need to add more new things to the backup system. There's all sorts of other things to consider like data consistency and whether operations on your systems are atomic (what happens if the backup snapshot is taken mid-operation in your app?) that you'll want to think about too.
djrogers 5y ago
Lots of good info here - it's also worth pointing out that if you're compromised, you may not have all the backups you think you do. A lot of the attackers out there are adding the step of disabling and deleting local snapshot-style backups as part of their attack, because they don't want all their hard work to get thrown out the window with a simple OS-level rollback (side note - if your endpoint security vendor tries to sell you rollback as a ransomware protection feature, run). For this reason, data backed up to tape or some other physical media that gets removed is much more likely to survive a breach than volume shadow copies and snapshots. Test the hard stuff!
- SOLAR_FIELDS 5y ago
  With that point said, wouldn’t immutable snapshots in the cloud like what rsync.net offers be quite valuable in terms of a rollback strategy?
timoth3y 5y ago
"Test your backups" is very good advice, but it will do almost noting to protect you against ransom attacks. A ransom attack works because one of the first things attackers do when they gain entry to the system is locate and encrypt the backups. Having tested backups is great, but it will not protect you from ransom attacks.
- gowld 5y ago
  If your backup is writeable, it's not a backup, it's a mirror.
- jms703 5y ago
  Correct. Backups cannot “protect” you. However proper backups can help you recover from the effects of ransomware.
kazinator 5y ago
By the way, that includes the "ransom" paid to the good guys who provide hard drive recovery services. It runs into to the hundreds of dollars.
- easton 5y ago
  Thousands if you have to go to someone like DriveSavers because you really messed something up.
davemtl 5y ago
Yes, test your backups regularly. When I worked for a large insurance firm, we would run drills every 6 months to perform off-site disaster recover and operational recovery tests to validate our recovery processes. Everything was tested from WAN links, domain controllers, file backups, mainframe recovery and so much more. We were more or less ready for a nuke to drop. Obviously this costs money, but if you're an insurance firm, not being able to recover would cost way more than running DR and OR recovery drills every 6-12 months.
- ramraj07 5y ago
  Why would some companies be so diligent while others get caught with their pants down? Can we tell which is which? Might be a good etf to invest in.
  - xwdv 5y ago
    Just go through Cloudflare's list of customers.
  - corty 5y ago
    Typically only companies that had a disaster happen to them or their customers (like that insurance) will have the institutional awareness. All the rest will file the risk somewhere with alien abductions and toilet paper shortages. When you tell them what could and will happen they will just shrug it off like you are trying to sell them useless bs.
    - blooalien 5y ago
      > When you tell them what could and will happen they will just shrug it off like you are trying to sell them useless bs. Exactly. It's that mentality which drove me to small-scale contract IT work for smaller "mom and pop" organizations. Give them a fair price and do good work and most of them are happy to have your services, treat you with respect, and are often more than happy to trade knowledge and services for equivalent exchange of same. This can lead to much "win /win/everybody wins!" And if you take a contract that plays out in an unsatisfactory way, it's easy to simply turn down further contracts from the one problematic customer. More time to give your loyal customers, or hunt down a better customer to replace the bad one. ;)
tgsovlerkhgsel 5y ago
Don't test your backups. Test your recovery. This may seem like the same thing, but one of the reasons why those ransomware gangs are so successful is because paying the ransom promises to get you back into business now. You probably still want to do a full remediation afterwards, but paying means that you can do that while your business is running. To make it unattractive to pay the ransom, you don't have to only be able to recover, you have to be able to do so quickly.
- loteck 5y ago
  That's what the article is about, you should check it out!
  - tgsovlerkhgsel 5y ago
    You're completely right, sorry about that. I've read it now. To add, it's not just about the ability to get the data back from the backup. The effort required to re-setup every client machine, and possibly replace all of them on short notice, is a significant obstacle. It's also really hard to exercise without significant expenditure.
crazygringo 5y ago
To be fair, it seems like a lot of backup systems were (properly) designed to recover data for when a single computer or drive or database fails or gets overwritten or specifically attacked -- but not for an wide-ranging attack where every networked computer gets wiped. All the stuff in this article is great scenarios to think about (recovery time, key location, required tools), but it's still all at the backup design phase. The headline of "test your backups" seems misleading -- you need to design all these things in before you even try to test them. It seems like a real problem here is simply that backup strategies were often designed before Bitcoin ransomware became prevalent, and execs have been told "we have backups" without probing deeper into whether they're the right kind of backup. In other words, there's no such single thing as "having backups", but rather different types of backup+recovery scenarios that either are or aren't covered. (And then yes, test them.)
- hnick 5y ago
  IIRC in the Maersk NotPetya disaster they had to look worldwide for a domain controller in Africa that happened to be off at the time, but fix and patch it before bringing it online. Restoring from backups would leave you vulnerable if a worm is still bouncing around. It takes a big coordinated effort for larger companies. Also the article doesn't seem to consider the fact that some hackers are now threatening release, not just destruction. Embarrassing emails, source code, and trade secrets. Backups won't help at all.
nickstinemates 5y ago
I always thought paying the Ransom was about the customer, PII, financial, and HR records/systems that are breached, less about getting the business back online. What a sorry state of affairs that it's both.
jiggawatts 5y ago
One thing that has irked me about everyone's flippant comments about moving to the cloud is that the "devops as a recovery mechanism" generally only works for single-app startups or small shops with only a few dozen simple VMs at most. Some of my customers have thousands of VMs in their cloud, and they aren't cloned cattle! They're pets. Thousands upon thousands of named pets. Each with their individual, special recovery requirements. This then has a nice thick crust of PaaS and SaaS layered on top in a tangle of interdependencies that no human can unravel. Some resources were built using ARM templates. Some with PowerShell scripts. Some with Terraform. A handful with Bicep. Most with click-ops. These are kept in any one of a dozen source control systems, and deployed mostly manually by some consultant that has quit his consulting company and can't be reached. Most cloud vendors "solve" this by providing snapshots of virtual machines as a backup product. Congratulations big vendors! We can now recover exactly one type of resource out out of hundreds of IaaS, PaaS, and SaaS offerings. Well done. For everything else: WARNING: THIS ACTION IS IRREVERSIBLE! Fantastic. No worries though, I can... err... export the definition, right? Wrong. That doesn't work for something like 50% of all resource types. Even if it "works", good luck restoring inter-dependent resources in the right order. Okay, you got things restored! Good job! Except now your DNS Zones have picked a different random pool of name servers and are inaccessible for days. Your PaaS systems are now on different random IP addresses and noone can access them because legacy firewall systems don't like the cloud. All your managed identities have reset their GUIDs and lost their role assignments. The dynamically assigned NIC IP addresses have been scrambled. Your certificates have evaporated. "But, but, the cloud is redundant! And replicated!" you're just itching to say. Repeat after me: A synchronous replica is not a backup. A synchronous replica is not a backup. A synchronous replica is not a backup. Repeat it. Do you know what it takes to obliterate a cloud-only business, permanently and irreparably? Two commands. I won't repeat them here, because like Voldemort's name it simply invites trouble to speak them out loud.
- Drblessing 5y ago
  This is a work of literature.
- 908B64B197 5y ago
  Some companies just do Cloud so their yearly report contains the word Cloud you know?
WheelsAtLarge 5y ago
I see the point of testing backups and it's good advice. But the real problem is that preparing for an emergency that never comes is very expensive in both productivity and costs. You can be 99.99% ready for a ransomware attack that would cost a lot but it would hit your organization's productivity hard. Yet there's a large possibility that the preparedness will go to waste because it will never be used. We need to find solutions that are very inexpensive but effective. I can only think of it being a cloud based solution where it will be trivial to reset and start over. I suspect that disaster recovery as a service(RaaS) should be part of any cloud based service. I get that some companies are so large and complicated that it would be impossible to provide that service for them but there are plenty of small to mid-size companies for which the service would be possible. So it's possible to offer it as part of any service package. RaaS has the great advantage that the costs can be shared among many companies so no one company needs to deal with the large costs that may never be used. It will also solve the problem associated with the constant up keep. It's hard to prepare for a disaster but it's even harder to keep it going for as long as it's needed. In addition, the increase in complexity for bad actors would decrease the incentives for ransomware in general. This won't happen now but given time it would largely fix the current situation for all.
- herpderperator 5y ago
  > Yet there's a large possibility that the preparedness will go to waste because it will never be used. Is this something companies want to gamble their future on?
  - lazide 5y ago
    Is it something that is easy to do by ignoring the possibility of it happening - and therefore is the default? Sure seems like it!
- csomar 5y ago
  > Yet there's a large possibility that the preparedness will go to waste because it will never be used. Human mistakes, hardware errors, and the increase of ransomware. That, and many companies relies on their IT infrastructure for their existence. I think backups are well worth it.
collsni 5y ago
Hot segregated onsite backup with a different authentication mechanism
WaitWaitWha 5y ago
Reminiscing - I once worked on a "backup server" that ran NetWare 3.11 and QIC80 drive. The customer was quite convinced their backup was fine. They could hear the QIC going with the standard weeeeee, tick, tick, tick, weeeeeee, tick, tick, tick sound every night as they were leaving. When I ejected the drive, the tape was nearly clear. All the material have been wiped clean off of it, and was sitting at the bottom of the server in a neat gray, brown pile (at least that is how I remember it, & I am sticking to it). Since they never had to restore, they never checked.
sillysaurusx 5y ago
One thing I've always wondered: How do you prevent ransomware from ruining your backups, too? Lots of ransomware tends to be "delayed" – it can't encrypt everything instantly, so it encrypts a little bit each minute. During those minutes, isn't it conceivable that the now-encrypted files replace your old backups? I suppose this isn't really a "backup" but rather a "mirroring strategy." But for certain kinds of data -- photos, video, large media -- can you really afford to do any other kind of strategy? The other question I have is related to that first point: since ransomware can't encrypt everything all at once, how is it possible for your system to continue to function? As you can tell, I'm a ransomware noob, but it's quite interesting to me from an engineering standpoint. Does the system get into a "half encrypted" state where, if you rebooted it, it would fail to boot at all? Or does ransomware targeted at businesses tend to be more of a surgical strike, where it targets and wipes out specific datastores before anyone notices? (It's the "before anyone notices" part that I'm especially curious about. Isn't there some kind of alarm that could be raised more or less instantly, because something detects unexpected binary blobs being created on your disk?)
- goldenkey 5y ago
  Linux executables can be modified while still running, this is what differentiates windows updates, many requiring rebooting, from Linux updates. Once rebooted, then you realize your executables are tainted.
- derekp7 5y ago
  Your general backup strategy should follow something like doing full/incremental and/or snapshot based backups. So in the case of your media, if you do a daily snapshot then your daily backups would be very small. And if the media doesn't change that often you can keep weekly copies for several weeks, monthly copies for many months, and several years of yearly snapshots. The other strategy is with tape rotation. You need to have about 30x the tape storage as you have online storage, so you can keep 7 yearly backups, 12 monthly, 6 weekly (all full backups), and 7 - 14 daily incremental backups.
- Dylan16807 5y ago
  > I suppose this isn't really a "backup" but rather a "mirroring strategy." Correct. > But for certain kinds of data -- photos, video, large media -- can you really afford to do any other kind of strategy? Yes. Make the backup system for that big slow-changing data a moderate amount bigger than the primary data store, and then you can have months of retention at low cost. If too much data changes at once then it should go read-only and send out a barrage of alerts.
SMAAART 5y ago
There's no glamour in back ups. Even less glamour in great back up. Even less in testing back-ups. And there's a lot of glory in "slashing the IT budget with no disruptions in operations, cutting the fat is good for business".
- rhizome 5y ago
  This is a good nutshell of the evolution in internet systems operations over the past 10 years. The boss didn't go to GSB just to watch some unfungible college-dropout make their own schedule every day.
aborsy 5y ago
Even with successful tests and recovery, you may still have to pay the ransom, because hackers often threaten to publish or sell data.
flowersjeff 5y ago
This really hits home, decades ago - I was working at this place that did daily tape backups. I remember thinking, this is unreal - there's literally a room filled with tapes. One day, I asked if they ever had performed a recovery off of the tapes, as I questioned if the tapes were even being written to. (NOTE: Backups was not my job at all. ) Why had I brought this up? I would be in the server room and never saw the blinky lights on the tape...well.. blink. Everyone literally laughed at me, thought was a grade A moron. A year later, servers died... Pop'ed in the tape... Blank. No worries, they had thousands more of these tapes. Sadly, they were all MT. They had to ship hard drives to a recovery shop, and it was rather expensive. I left shortly after this.
- wyldfire 5y ago
  > Everyone literally laughed at me, thought was a grade A moron. A note for anyone else in a similar situation - a good team doesn't ridicule someone for questions like these. A responsible leader should have cited a time in the past that they did a restore or a spot check, and no one should have laughed. The laughter sounds like masked fear or embarrassment. This goes for any team. "How do we know this function of our job does what we think it does?" You should have an answer. Now, I've only worked in R&D software and not in IT. But IMO IT teams should work the same way in this regard.
  - yosito 5y ago
    > a good team doesn't ridicule someone for questions like these A good team won't ridicule any questions. If you're on a team that ridicules your questions, that's a huge red flag. Get out as soon as possible!
  - 908B64B197 5y ago
    > A responsible leader should have cited a time in the past that they did a restore or a spot check, and no one should have laughed. The laughter sounds like masked fear or embarrassment. ... or assigned the engineer asking questions the task of figuring it out!
  - amatecha 5y ago
k12sosse 5y ago
It's so easy to backup (and test!) these days. There's no excuse other than incompetence.
nonfamous 5y ago
Fourth, lack of backups may not even be the threat. What if the hackers demand a ransom to prevent them from releasing sensitive data to the public?
- TrueDuality 5y ago
  Those are much rarer, and in several cases the attackers didn't actually have the data when those threats are made. That aside, what you're talking about is substantively different crime and one that is much easier to prosecute internationally than computer based crime. Espionage, and blackmail are both crimes that have been around for a very long time and have international treaties around them. That doesn't stop it from happening, but when someone is caught it's much more likely they're going to be extradited and prosecuted than for a crippling attack against a computer system. Either way, if an attacker is threatening to release your internal confidential data, would you rather have a copy of that data as well or not? Your course of action is also the same in either case, notify the FBI if you're in the US.
_wldu 5y ago
They also threaten to leak your data if you don't pay. They know a lot of orgs can restore the data and won't need it decrypted. There's no real defense against this (other than good security practices).
- SamBam 5y ago
  I mean, that's a whole nother kettle of fish. At that point they have you in the hook for indefinite blackmail, because after you pay they still have the days. Is that actually common, though? I think most of these ransomware cases are simply pay-to-decrypt.
  - allset_ 5y ago
    Almost all of the ransomware gangs now also exfil data and use that as additional leverage.
  - mm983 5y ago
    yes, double extortions are getting common because people rely on backups
- squeaky-clean 5y ago
  You can never prove that they won't leak your data after paying though. I'm not a CEO/CTO and haven't had to make these decisions, but from my perspective it's an empty promise that by paying them they will actually keep their word and not leak your data.
  - thekyle 5y ago
    If they do leak your data after you pay, then it'll ruin their reputation and make it less likely for other victims to pay in the future.
    - axiosgunnar 5y ago
      Which is why we should do „fake“ ransomware attacks where a company „pays“ and gets „betrayed“ when the attacker still leaks some „super important“ data after the payment. What are the hackers gonna do? Sue you?
efitz 5y ago
My very first mentor when I started my IT career 30 years ago told me “your job is to make sure the backups are working. Everything else is icing on the cake.” Still true. But one caveat: your back ups can get screwed up if you back up data that has already been encrypted by ransomware. One easy way to defend against this is to have a tier 2 back up with a time delay, eg it backs up the backup files from a week ago. Tear 1 backup is just normal back up. You test it frequently of course to make sure that you can restore it but you don’t have to do any extraordinary work to detect ransomware in tier 1. Tier 2 backs up the back up, but only after a certain number of days have passed. That number of days is the window that you have to detect that ransomware has infected you. If you ever find a ransomware infection, you isolate and turn off tier 2 immediately to preserve a known clean state, and once you have everything rebuilt clean and patched, you restore from tier 2. You use tier 1 for restores in non ransomware situations because it’s necessarily more up-to-date.
Jedd 5y ago
Depressingly few organisations I've worked in or with have a clearly defined set of RPOs and RTOs for their essential systems, and similarly few have regular processes to test their backups and archives to either confirm the process works, or to determine how long a recovery would take. This stuff is all conceptually very easy to do - but politically extremely difficult to agree on the definitions, and then obtain the resources on the ops side, presumably because it's yet another of those things that fits in the category of being hidden and non-urgent, and therefore a low priority, right up until the moment it isn't.
deleted 5y ago
[deleted]
basicplus2 5y ago
This is why i have my own tape backup system in addition to cloud (someone elses computer) backup. Ready to restore everything quickly.
deleted 5y ago
[deleted]
leke 5y ago
I didn't realise backups were normally kept off site.
- whartung 5y ago
  I love disaster recovery disaster stories. When all the good intentions go wrong. Two from the '89 quake. 1) The smart thinking operator who equipped their machines with UPSes to keep them up through a power failure. The quake cut the power, the UPSes kicked in, the systems stayed up, the drives kept spinning, the quake kept going and buried the drive heads in to the platters. 2) System crashed, backup tapes were kept in the basement. Quake set off the sprinkler system in the storage room, soaking the tapes with rusty water. "I'm sorry, did I break your concentration? I didn't mean to do that. Please, continue, you were saying something about best intentions." - Jules
Aulig 5y ago
What if the ransomware has been hiding for a few months before it activated? You'd either have overwritten the last clean backup by then or you'd restore the ransomware too. Or am I missing something?
- KirillPanov 5y ago
  The article addresses this: “That is still somewhat rare,” Wosar said. “It does happen but it’s more the exception than the rule. It's something that very high-value targets will eventually have to worry about. But figuring out how to corrupt the backups without disrupting the production systems (and being discovered) is not likely to ever be a fully automatic process for the randomware goons. They'll have to invest serious time in understanding how the victim's systems work. For high-value targets, that will happen. I just walked through a telecom colo last week and saw a weird new box in somebody else's cage I'd never seen before. Later, I did a web search for the brand name. Apparently this company sells a network filer that keeps local daily/monthly/annual backups and doesn't let you delete them. Probably a good idea for an organization that just wants to throw money at the problem. Of course you can build this yourself with "btrfs sub snap" except for the part about not letting yourself (even as root) delete them.
  - Aulig 5y ago
    Ah, I interpreted "corrupt backups aswell" as encrypting them aswell.
AtlasBarfed 5y ago
If you have a cloud, it's not "test your backup". It's "have an automated restore" ready. Maybe on a different cloud. With clouds, you can test standing up your entire infrastructure/system stack across even a couple hundred machines or more in automated fashion, and then tear the whole thing down.
dijit 5y ago
Is this the death of ops? Common ops mantra used to be “A backup does not exist until you’ve restored it”. Having a blob of data means nothing- being able to continually restore it and integrity check it is everything.
joefife 5y ago
That's not how modern ransomware works. It's now common to see data extracted and for the ransom to cover not disclosing your corporate data. But yes, agree sky backups in terms of restoring operations.
nazrulmum10 5y ago
We recommend creating an entire playbook so you know what you need to do to recover from a ransomware attack.
varelaz 5y ago
I think we need to treat such threats as a rear disease which could happen whenever you ready or not. We need to check for it but be prepared to have it anyway. Industry should start thinking more about insurance, negotiation and investigation services.
timwis 5y ago
Unfortunately this doesn’t cover the case where the ransomware group is threatening to leak your data unless they pay :/ It’s also good to consider—if a ransomware allows attackers to access your network—whether there’s anything stopping them from accessing (and encrypting/overwriting/deleting) your backups.
sandreas 5y ago
A huge step forward was using zfs with snapshots for my system and backup devices... zfs local fs might be still a bit dangerous, because theoretically ransomware could delete snapshots, but if you use a server / share that underlies zfs, you are pretty save. There are some decent guides for Arch: Official: https://wiki.archlinux.org/title/Install_Arch_Linux_on_ZFS https://wiki.archlinux.org/title/Install_Arch_Linux_on_ZFS Encrypted: https://forum.level1techs.com/t/manjaro-root-on-zfs-with-encryption/170428 https://forum.level1techs.com/t/manjaro-root-on-zfs-with-enc...
technimad 5y ago
"A backup not restored is a backup not made." Should be on the wall in every IT department. Together with "Snapshots are NOT backups".
teekert 5y ago
I heard from companies using ZFS, that suddenly see a massive increase in disk space usage... A results from copy-on-write for all the new encrypted files. Then they restored to a snapshot before the encryption started and were able to resume business (after a purge and decoupling of all suspicious machines). Could it be that simple? Sure snapshots are not backups, but this is not a hardware failure. In my home situation I do a periodic rsync over ssh to a Pi3 at my parents place. It's super simple but I can just browse the sshfs to check if stuff is there and working. And when I need it, I drive there and get the disk itself. Sure, this is not relevant for a large business, but for us self-hosters it is a nice solution. The backup is manual, by choice, so I never overwrites accidentally.
raxxorrax 5y ago
Plainly the best defense against ransomware. Had two attacks that probably could not have been avoided. Some employee will at some point open that funny looking mail. Scammers know our names and mail signatures (the colorful text, not the crypt sig for mails). Maybe they get some detail wrong, like department or a contact address, but otherwise the mails looked genuine. A sensible backup solution made us come back within 30 minutes of severe attacks. The police tried to negotiate with the attackers but sadly couldn't get them. But at least they didn't get any money. We have a bought backup solution that is quite expensive that does snapshots every 15 minutes I believe. Worth it.
mvkel 5y ago
Paying the ransom and then plugging the hole is probably far cheaper than spinning up and maintaining a hardcore backup-perfect stack.
marto1 5y ago
I believe we are unfortunately at the “have backups” stage still civilizationally.
HeavyStorm 5y ago
You mean, my cloud provider isn't doing it for me?

599 points by parsecs 5y ago

Don't just test your backups. Make sure your automation can't clobber or tamper with your backups. This includes both local and disaster recovery sites. Give your pen-test team super-user privs on your automation and give them Amazon gift cards if they can tamper with your backups. If they can't mess with the backups, give the gift cards to whoever designed and hardened your infrastructure.

wizzwizz4 5y ago
Why not actual money? Amazon gift cards leak metadata to Amazon, and can only be used to buy stuff from Amazon.
- jagged-chisel 5y ago
  And they support Amazon.
  - wizzwizz4 5y ago
    Oh, of course. Can't believe I forgot the biggest reason.
- LinuxBender 5y ago
  Good point. Cash bonus and maybe RSU's if they company is public.
- jonas21 5y ago
  It used to be that you could give employees gift cards up to a certain amount as awards and it would not be considered taxable income (but I believe that's no longer the case).
  - jagged-chisel 5y ago
    Any gift(s) up to a total value of ... $13k? -ish? I don't know what the limit is now. Google's cafeteria is (was? depending on that limit...) an example of how to benefit employees without causing the employee additional tax.
    - mikeyouse 5y ago
      That's a different issue - IRS clamped down on gift cards and non-cash compensation that used to be considered de minimis. Now most employers gross up and report any gift card type gift over ~$5. https://www.irs.gov/government-entities/federal-state-local-governments/de-minimis-fringe-benefits https://www.irs.gov/government-entities/federal-state-local-...
    - dmoy 5y ago
      Setting aside the gift card bit (addressed in above comment), $13k sounds way too high. Like two orders of magnitude too high. From irs.gov > Whether an item or service is de minimis depends on all the facts and circumstances. In addition, if a benefit is too large to be considered de minimis, the entire value of the benefit is taxable to the employee, not just the excess over a designated de minimis amount. The IRS has ruled previously in a particular case that items with a value exceeding $100 could not be considered de minimis, even under unusual circumstances. Which about matches with what I've seen at BigCo. $40 box of tools as a gift? Did not show up on my paycheck. $150 electronic device as a gift? Showed up on my paycheck.
      - mikeyouse 5y ago
        There's another about other fringe benefits being taxed - with a prime example being tech cafeterias. https://taxnews.ey.com/news/2019-0493-employer-must-substantiate-necessity-of-employer-provided-meals-for-value-to-be-excluded-from-employee-income-irs-tam-emphasizes https://taxnews.ey.com/news/2019-0493-employer-must-substant... In the past few years, guidance has shifted toward accounting for employer-provided food with employee income as well.
- bentcorner 5y ago
  I think logistically its easier for a team within an org to spend "their" money on gift cards for intermittent activities and hand them out as necessary. Getting stuff onto the actual payroll is probably more complicated.
  - edoceo 5y ago
    Hey Payroll, edoceo needs an off-cycle bonus of $$$. Your manager should be able to write a similar email.
    - Volundr 5y ago
      At least at the company I recently left, this kicks off an approval process within both the HR and accounting departments. Meanwhile an Amazon purchase (and thus an Amazon gift card) is something I could put on my card and expense, or approve someone else doing myself. I get it doesn't make sense, but that's corporate America for you. That said, be careful of the gift card route. Depending on the amount you can find yourself in the wrong side of the IRS that way.
      - gowld 5y ago
        I wouldn't want to embezzle funds and commit income/payroll tax fraud just to bypass paperwork
    - rad_gruchalski 5y ago
      It doesn’t work like that in every place in the world. Those gift cards also aren’t that easy. It’s all taxable benefit to the employee and a cost for the company to put on the books. That needs to be justified and tax office may really slap you on the wrist for doing so.
    - Smoosh 5y ago
      Forget the backups - the pen test team can just produce fake emails requesting they get the $$$.
    - benhurmarcel 5y ago
renewiltord 5y ago
Which organizations currently do this?

jerhewet 5y ago

Isn't this more of a "We don't want our client / customer information released to The World At Large" question? I would think most business entities have backups of some kind (Scripps being the only exception I can think of), and will pay the ransom to keep any sensitive information off the market. Edit: Should have added that I find it hard to believe that companies have PB of data backed up. I could believe GB, and maybe even TB, but PB is pretty hard to swallow. The past three companies I've worked for (25 year span) had, at most, a couple of gigs of sensitive information that couldn't be easily replicated.

stan_rogers 5y ago
Ransomware attacks rarely indicate any data leakage; all they usually do is prevent you from accessing your own data (by encrypting your drive with a key you don't have access to).
- intothev01d 5y ago
  O rly? https://www.trendmicro.com/vinfo/us/security/news/cybercrime-and-digital-threats/modern-ransomwares-double-extortion-tactics-and-how-to-protect-enterprises-against-them https://www.trendmicro.com/vinfo/us/security/news/cybercrime...
- runnerup 5y ago
  These days attacks labeled “ransomware” in the news seem to be hybrid attacks. There usually is sensitive data exfiltration in addition to encrypt-in-place.
- djrogers 5y ago
  The current trend is double-extortion ransomware attacks - encrypt your copy of your data, and threaten to release it publicly as well. [1] https://www.cybereason.com/blog/rise-of-double-extortion-shines-spotlight-on-ransomware-prevention https://www.cybereason.com/blog/rise-of-double-extortion-shi...
nelgaard 5y ago
I also find it hard to believe that a ransomware gang could encrypt 50 Petabytes without anyone noticing it. It would also take some time to decrypt 50 petabytes if you paid the criminals and got the key. And would you trust you data after criminals had access to it?

nickdothutton 5y ago

Not really just a backup and restore. You need to be able to rebuild from zero. I think of it more as a disaster recovery exercise, and for those… you are only as good as your last _real_ rehearsal. That may mean a suitcase of tapes, a sheet of paper, and a rack of blank servers. Then you have the problem of release of confidential information. For this reason, the sweetest target for ransomware is the company who can neither recover their data, nor can they afford to have it publicly posted or monetised by the gang. Oh and you do store those backups offline dont you? Ransomware gangs have been known to loiter and observe their target for weeks to learn how to sabotage backups when the time comes.

user3939382 5y ago

What sucks for HIPAA is that you can get fined for the breach itself, regardless of your backup management.

edoceo 5y ago
Not really a problem with HIPPA is it?
MattGaiser 5y ago
Seems appropriate.

slownews45 5y ago

There is another approach. Scrub old data you don't need. 2-3 year email retention on corp email. Paper files for sensitive client info (or don't keep it). We can reinstall office / windows / active director etc. Mandatory 2FA on google suite? Git codebases on github etc for LOB apps (we can rebuild and redeploy). We use the lock features in S3 for copies of data that must be kept. Not sure I can even unlock to delete as account owner without waiting for timeouts.

orf 5y ago
You can also enable versioning and deny all s3:DeleteObject actions via the bucket policy. It won’t stop a root account compromise unless you’ve got a multi-account setup going (as they could edit the bucket policy). But if you’ve not got any monitoring, they could also just remove the lock on the objects without you noticing and wait for the timeout themselves.
TrueDuality 5y ago
A lot of sensitive data is legally required to be kept around for much longer than that (email is up to 15 years if you're a publicly traded company). Most of the remaining suggestions aren't relevant to ransomware even if they're otherwise mostly fine recommendations. 2FA won't stop ransomware, or data destruction. Redeploying code and reinstalling active directory doesn't restore customer or user databases. Paper files are not accessible when they're needed, are easily damaged or misplaced, and cost a lot to store and index (if you're referring to keeping them as backups then yes you're making a form of the argument of the post just in a very expensive and inconvenient way). Read-only S3 copies are almost certainly falling in the realm of backups... but is also a relatively expensive way to do it for most organizations larger than a start-up. Offline and offsite backups are the cheapest, most effective tool for keeping copies of your data in your companies possession due to unforeseen circumstances and they protect against a huge number of potential events beyond just ransomware. It's negligent IMO for executive officers of a company to not have invested in a solid, tested, and comprehensive backup solution.
- allset_ 5y ago
  Most attack campaigns start with compromised credentials, so MFA absolutely helps prevent ransomware.

Severian 5y ago

3-2-1 Backup Rule: Three copies of your data. Two "local" but on different mediums (disk/tape, disk/object storage), and at least one copy offsite. Then yes, absolutely perform a recovery and see how long it takes. RTOs need to be really low. Recovering from object storage is going to take at least a magnitude more time than on-prem. Also, storage snapshots/replications are not backups, stop using them as such. Replicating is good for instant failover, but if your environment is hacked they are probably going to be destroyed as well.

Waterluvian 5y ago

I’m a novice and am dealing with data that isn’t too complicated, large, or important. My approach is to build restore directly into the normal workflow. I test my backups by using them each week. A stack is spawned from a database backup and once it passes tests, replaces the previous one. Not sure how smart this all is but my goal is to learn through application.

TheDong 5y ago
The main reason I think this normally isn't done is that it requires downtime to do safely most of the time. In order to not lose data, you can't have any writes between the time when the backup was taken and the present, or you need code which reconciles additional state and adds it onto the backup before switching over. Normally, backup restoration is done during a maintenance window where the site is disabled so no writes can happen, and then usually a window of writes are lost anyway (i.e. 'last X hours, since the backup was taken') For your use-case, do you just have very few writes? Do you lose writes? Do you have some other clever strategy to deal with it?
- dragontamer 5y ago
  It should be noted that not everyone is a global company. A typical bank / credit union may only serve one town. As such, it would be socially acceptable to designate 3am to 4am as a regular maintenance window where services are shutdown.
- Waterluvian 5y ago
  Good point. The 5 minutes of downtime is simply tolerated. My captive audience are dozens of humans and thousands of robots all willing to try again.
  - mlac 5y ago
    Ticketmaster?
innomatics 5y ago
>A stack is spawned from a database backup and once it passes tests, replaces the previous one. I like this approach, although risky if you mean you routinely replace the production db. My preferred setup is to automate restores into a pre-prod environment, apply data masking and run tests there. It's not a replacement for full DR exercises, but at least it automates the backup verification process as part of your build system.

whartung 5y ago

"Test your backups" is so easy to say, but quite difficult for many to do. There are a lot of shops that probably don't know how to recreate a machine from scratch. How many systems are developed as balls of clay. Little bits added and smeared in over time until the ball just gets bigger, but each piece lost in the process. How many folks can go through their local config files and explain all of entries, how many can even tell which ones they have changed, or why? Especially when they were changed by Frank, but he left 2 years ago. You'd like to think you can just restore the system from backup and it'll just light back up. But how do you test this without cratering your existing system? Like a boat in a basement, many system are built in-situ and can be very rigid. Modern environments like cloud computing and creation scripts can mitigate this a bit organically, but how many of these systems are just a tower running Windows w/SQL Server and who knows what else? Plus whatever client software is on the client machines. How do you test that in isolation? At least read the media to see if it can be read (who doesn't love watching a backup tape fail halfway through the restore). Simply, it takes a lot of engineering to make a system that can be reliably restored, much less on a frequent basis. And this is all engineering that doesn't impact the actual project -- getting features to users and empowering the business. Which can make the task even more difficult.

xwdv 5y ago
This is like a sales pitch for just paying the ransom.
- MattGaiser 5y ago
  It probably works more often than not.
  - shyn3 5y ago
    It works until it doesn't, such as in a hardware failure. [1] https://www.anandtech.com/show/15673/dell-hpe-updates-for-40k-hour-essd-failures https://www.anandtech.com/show/15673/dell-hpe-updates-for-40...
  - quickthrower2 5y ago

decebalus1 5y ago

If your disaster recovery process isn't tested, you actually don't have any disaster recovery. It's not only about 'how long it takes' it's also about whether or not it works at all. Can you rebuild from scratch? What happens if your entire infrastructure goes down at the same time? What happens if a datacenter you rely on just disappears? What happens if you lose access to your systems? Can you lose access to your systems? IMHO one of the only silver lining of these attacks is that organizations are starting to ask these questions more often.

PhantomGremlin 5y ago

There's been a lot of good advice here about backups and disaster recovery. But there's also a lot of other stuff to consider: Compartmentalization. Finance and Engineering and Sales only need to interact in limited ways. How about some firewalls between them, limiting types of access? Location isolation. Why does something that happens in Peoria affect Tuscaloosa? Once a ransomware gang breaches a perimeter, why is it allowed countrywide (or worldwide) access to a company? Monitoring. Aren't there tools that can alert on various anomalous patterns? All of a sudden, gigabytes of data start being exfiltrated? All of a sudden, processes fire up on a multitude of servers? Monitoring these things is hard to do at scale, but surely possible? Microsoft. In 2002, Bill Gates "Finally Discovers Security". How much longer will Microsoft be given a free pass? How many more "critical" vulnerabilities will their software have? https://www.wired.com/2002/01/gates-finally-discovers-security/ https://www.wired.com/2002/01/gates-finally-discovers-securi... I could go on and on. But why should I? Why can't MBA-type CEOs take IT seriously? Why can't they hire competent people and fund them and listen to them?

blooalien 5y ago
> … "and listen to them?" That's the part I've always had trouble gettin' out of most "management" types. They hire you for your expertise, and then undermine it at every opportunity to "save money" or to exert their "authoritah".
- g_p 5y ago
  There's significant misalignment of incentives at play here (irrespective of company, generally). Many "management types" are measured solely and almost entirely by the bottom line - the share price in a traded company, or profits in a private company. Share price is based usually on profits. Usually they're delivering reporting to their boss or the board quarterly. Every dime they save this quarter is "profit" in their eyes. Obviously there's a line here - you don't want to save the dime that causes the factory to burst into flames and result in 4 months of production downtime. But if that dime can be saved this quarter when times are tough, you'll be seen as a well-performing hero, and next quarter you'll spend a few dimes getting the sprinkler system inspected... Until you need the boost to profits next quarter too(!) In management and generally in business it's hard to go backwards. Nobody wants to increase their spend on IT unless they are making more money. If they could sign a new deal this quarter, they'll probably give you a decent percentage of that deal as IT budget to get the deal signed (factoring in the risk of the customer not signing). But in an environment focused almost entirely on pursuit of unrelenting growth of profit or revenue or share price, security simply won't become a priority until you can convince manager-types that the issue is commercial and measurable and that it impacts the bottom line. Even if the issue is unexpected costs of recovery from a breach, the first questions they'll ask are "How likely is it to happen? What will it cost? Can we insure against it? What do our competitors do?" - it's not about preventing the breach, it's about defraying the potential loss of profits without compromising on growth or profits when it isn't a problem. Hence you'll see people hired to fill roles (new or existing) then being hamstrung by a total and outright refusal to act, because it isn't a commercial problem. Skilled tech leaders are good at turning the problems into language leaders can understand, but there is a limit and a line, and the solution in my view is clear regulation the leaders can "get", involving individual penalties and obligations of competency - even if your core business isn't safety, you have safety obligations as a business to your employees, contractors and the public. It's expected and required that you become suitably competent to do this, or get the expertise to do so, but with you still liable. We need the same for security (in my view). It's not all bad news - I'm a "management type", but from an almost entirely technical background. Bean counting isn't my style... Maybe we can infiltrate more organisations and bring some basic engineering understanding to decision making? It's frustrating to see most hide behind MBA-waffle though rather than try to actually do real things that make a difference.

throwawaysleep 5y ago

How does one learn how to do proper backups? Using my throwaway as I suspect my company doesn't do them (and even if they do, I don't know where they are or what to do with them as the main engineer left on my piece of software).

dreadlordbone 5y ago
That entirely depends on how your infrastructure is set up. If it's in AWS, just set up rolling snapshots of your EC2's and databases. If you can store everything in Github, that works too. If it's on local machines, something like Backblaze is cheap and easy to set up.
deleted 5y ago
[deleted]
g_p 5y ago
I'm not a backup specialist, but do run my own backup infrastructure, so a few starting points. Not saying they will work in all environments, or even any scale environment though. 1. Figure out what's valuable and what needs backed up. Some will say "back up everything", but then you don't know what you need to restore and how to restore it! You need to know what's valuable. Is that source code? A complex application environment? Files in S3 storage? User generated files stored locally on a server? Data in a database? 2. Figure out where it's stored. Not always straightforward, but again necessary. Otherwise you won't know how to back it up, or recover it. Remember the cloud isn't a backup - when someone steals your production S3 keys, absent some careful configuration, they can dump and delete all your bucket contents... Understand what you need to backup and how you'll recover it. If you are dealing with Windows based environments, consider the practicalities of getting these up again - windows isn't as nice as Linux (which can boot on any hardware within reason, modulo some bootloader settings in a few complex configurations) 3. Figure out how you're going to protect the data you back up. Now you've identified the crown jewels, don't just store them in plaintext and unencrypted on your laptop (!) If you're going to use encrypted backups, understand where the keys are stored and how they're used. More importantly, understand how you'll have the keys to decrypt the backups after a disaster. 4. Figure out how you'll stop a someone who gets in from deleting or accessing your backups. Assume they'll be attacked. Understand what the backup solutions offer in terms of security. Can you use asymmetric crypto so only public keys get stored on your servers? Can you use protection on an s3 storage bucket to prevent deletion, even with the access key? Etc. 5. Test doing backups. Then practice restoring to a non production environment to see what you missed. Document it all! 6. Set up monitoring so you know when something goes wrong with your backups. Gather and expose enough information to yourself that you know what's going on! This could be as simple as an automated email to yourself every morning, telling you the duration of the backup job, and the amount of data written, and amount changed since last time. But you need to monitor this. Don't just go for a failure alarm (have one of those, and make it fail-safe so it alerts if the backup didn't succeed, even if the backup script didn't run at all!) - also notify yourself on success so you are aware of the backups and what goes on. 7. Ensure you have enough copies. 1 isn't enough - what if the payment card on your storage account expires, or you get locked out? Or the company goes under? People normally say 3 backups in 2 locations, at least 1 off-premises. 8. Return to 1, because by now something has changed and you need to add more new things to the backup system. There's all sorts of other things to consider like data consistency and whether operations on your systems are atomic (what happens if the backup snapshot is taken mid-operation in your app?) that you'll want to think about too.

djrogers 5y ago

Lots of good info here - it's also worth pointing out that if you're compromised, you may not have all the backups you think you do. A lot of the attackers out there are adding the step of disabling and deleting local snapshot-style backups as part of their attack, because they don't want all their hard work to get thrown out the window with a simple OS-level rollback (side note - if your endpoint security vendor tries to sell you rollback as a ransomware protection feature, run). For this reason, data backed up to tape or some other physical media that gets removed is much more likely to survive a breach than volume shadow copies and snapshots. Test the hard stuff!

SOLAR_FIELDS 5y ago
With that point said, wouldn’t immutable snapshots in the cloud like what rsync.net offers be quite valuable in terms of a rollback strategy?

timoth3y 5y ago

"Test your backups" is very good advice, but it will do almost noting to protect you against ransom attacks. A ransom attack works because one of the first things attackers do when they gain entry to the system is locate and encrypt the backups. Having tested backups is great, but it will not protect you from ransom attacks.

gowld 5y ago
If your backup is writeable, it's not a backup, it's a mirror.
jms703 5y ago
Correct. Backups cannot “protect” you. However proper backups can help you recover from the effects of ransomware.

kazinator 5y ago

By the way, that includes the "ransom" paid to the good guys who provide hard drive recovery services. It runs into to the hundreds of dollars.

easton 5y ago
Thousands if you have to go to someone like DriveSavers because you really messed something up.

davemtl 5y ago

Yes, test your backups regularly. When I worked for a large insurance firm, we would run drills every 6 months to perform off-site disaster recover and operational recovery tests to validate our recovery processes. Everything was tested from WAN links, domain controllers, file backups, mainframe recovery and so much more. We were more or less ready for a nuke to drop. Obviously this costs money, but if you're an insurance firm, not being able to recover would cost way more than running DR and OR recovery drills every 6-12 months.

ramraj07 5y ago
Why would some companies be so diligent while others get caught with their pants down? Can we tell which is which? Might be a good etf to invest in.
- xwdv 5y ago
  Just go through Cloudflare's list of customers.
- corty 5y ago
  Typically only companies that had a disaster happen to them or their customers (like that insurance) will have the institutional awareness. All the rest will file the risk somewhere with alien abductions and toilet paper shortages. When you tell them what could and will happen they will just shrug it off like you are trying to sell them useless bs.
  - blooalien 5y ago
    > When you tell them what could and will happen they will just shrug it off like you are trying to sell them useless bs. Exactly. It's that mentality which drove me to small-scale contract IT work for smaller "mom and pop" organizations. Give them a fair price and do good work and most of them are happy to have your services, treat you with respect, and are often more than happy to trade knowledge and services for equivalent exchange of same. This can lead to much "win /win/everybody wins!" And if you take a contract that plays out in an unsatisfactory way, it's easy to simply turn down further contracts from the one problematic customer. More time to give your loyal customers, or hunt down a better customer to replace the bad one. ;)

tgsovlerkhgsel 5y ago

Don't test your backups. Test your recovery. This may seem like the same thing, but one of the reasons why those ransomware gangs are so successful is because paying the ransom promises to get you back into business now. You probably still want to do a full remediation afterwards, but paying means that you can do that while your business is running. To make it unattractive to pay the ransom, you don't have to only be able to recover, you have to be able to do so quickly.

loteck 5y ago
That's what the article is about, you should check it out!
- tgsovlerkhgsel 5y ago
  You're completely right, sorry about that. I've read it now. To add, it's not just about the ability to get the data back from the backup. The effort required to re-setup every client machine, and possibly replace all of them on short notice, is a significant obstacle. It's also really hard to exercise without significant expenditure.

crazygringo 5y ago

To be fair, it seems like a lot of backup systems were (properly) designed to recover data for when a single computer or drive or database fails or gets overwritten or specifically attacked -- but not for an wide-ranging attack where every networked computer gets wiped. All the stuff in this article is great scenarios to think about (recovery time, key location, required tools), but it's still all at the backup design phase. The headline of "test your backups" seems misleading -- you need to design all these things in before you even try to test them. It seems like a real problem here is simply that backup strategies were often designed before Bitcoin ransomware became prevalent, and execs have been told "we have backups" without probing deeper into whether they're the right kind of backup. In other words, there's no such single thing as "having backups", but rather different types of backup+recovery scenarios that either are or aren't covered. (And then yes, test them.)

hnick 5y ago
IIRC in the Maersk NotPetya disaster they had to look worldwide for a domain controller in Africa that happened to be off at the time, but fix and patch it before bringing it online. Restoring from backups would leave you vulnerable if a worm is still bouncing around. It takes a big coordinated effort for larger companies. Also the article doesn't seem to consider the fact that some hackers are now threatening release, not just destruction. Embarrassing emails, source code, and trade secrets. Backups won't help at all.

nickstinemates 5y ago

I always thought paying the Ransom was about the customer, PII, financial, and HR records/systems that are breached, less about getting the business back online. What a sorry state of affairs that it's both.

jiggawatts 5y ago

One thing that has irked me about everyone's flippant comments about moving to the cloud is that the "devops as a recovery mechanism" generally only works for single-app startups or small shops with only a few dozen simple VMs at most. Some of my customers have thousands of VMs in their cloud, and they aren't cloned cattle! They're pets. Thousands upon thousands of named pets. Each with their individual, special recovery requirements. This then has a nice thick crust of PaaS and SaaS layered on top in a tangle of interdependencies that no human can unravel. Some resources were built using ARM templates. Some with PowerShell scripts. Some with Terraform. A handful with Bicep. Most with click-ops. These are kept in any one of a dozen source control systems, and deployed mostly manually by some consultant that has quit his consulting company and can't be reached. Most cloud vendors "solve" this by providing snapshots of virtual machines as a backup product. Congratulations big vendors! We can now recover exactly one type of resource out out of hundreds of IaaS, PaaS, and SaaS offerings. Well done. For everything else: WARNING: THIS ACTION IS IRREVERSIBLE! Fantastic. No worries though, I can... err... export the definition, right? Wrong. That doesn't work for something like 50% of all resource types. Even if it "works", good luck restoring inter-dependent resources in the right order. Okay, you got things restored! Good job! Except now your DNS Zones have picked a different random pool of name servers and are inaccessible for days. Your PaaS systems are now on different random IP addresses and noone can access them because legacy firewall systems don't like the cloud. All your managed identities have reset their GUIDs and lost their role assignments. The dynamically assigned NIC IP addresses have been scrambled. Your certificates have evaporated. "But, but, the cloud is redundant! And replicated!" you're just itching to say. Repeat after me: A synchronous replica is not a backup. A synchronous replica is not a backup. A synchronous replica is not a backup. Repeat it. Do you know what it takes to obliterate a cloud-only business, permanently and irreparably? Two commands. I won't repeat them here, because like Voldemort's name it simply invites trouble to speak them out loud.

Drblessing 5y ago
This is a work of literature.
908B64B197 5y ago
Some companies just do Cloud so their yearly report contains the word Cloud you know?

WheelsAtLarge 5y ago

I see the point of testing backups and it's good advice. But the real problem is that preparing for an emergency that never comes is very expensive in both productivity and costs. You can be 99.99% ready for a ransomware attack that would cost a lot but it would hit your organization's productivity hard. Yet there's a large possibility that the preparedness will go to waste because it will never be used. We need to find solutions that are very inexpensive but effective. I can only think of it being a cloud based solution where it will be trivial to reset and start over. I suspect that disaster recovery as a service(RaaS) should be part of any cloud based service. I get that some companies are so large and complicated that it would be impossible to provide that service for them but there are plenty of small to mid-size companies for which the service would be possible. So it's possible to offer it as part of any service package. RaaS has the great advantage that the costs can be shared among many companies so no one company needs to deal with the large costs that may never be used. It will also solve the problem associated with the constant up keep. It's hard to prepare for a disaster but it's even harder to keep it going for as long as it's needed. In addition, the increase in complexity for bad actors would decrease the incentives for ransomware in general. This won't happen now but given time it would largely fix the current situation for all.

herpderperator 5y ago
> Yet there's a large possibility that the preparedness will go to waste because it will never be used. Is this something companies want to gamble their future on?
- lazide 5y ago
  Is it something that is easy to do by ignoring the possibility of it happening - and therefore is the default? Sure seems like it!
csomar 5y ago
> Yet there's a large possibility that the preparedness will go to waste because it will never be used. Human mistakes, hardware errors, and the increase of ransomware. That, and many companies relies on their IT infrastructure for their existence. I think backups are well worth it.

collsni 5y ago

Hot segregated onsite backup with a different authentication mechanism

WaitWaitWha 5y ago

Reminiscing - I once worked on a "backup server" that ran NetWare 3.11 and QIC80 drive. The customer was quite convinced their backup was fine. They could hear the QIC going with the standard weeeeee, tick, tick, tick, weeeeeee, tick, tick, tick sound every night as they were leaving. When I ejected the drive, the tape was nearly clear. All the material have been wiped clean off of it, and was sitting at the bottom of the server in a neat gray, brown pile (at least that is how I remember it, & I am sticking to it). Since they never had to restore, they never checked.

sillysaurusx 5y ago

One thing I've always wondered: How do you prevent ransomware from ruining your backups, too? Lots of ransomware tends to be "delayed" – it can't encrypt everything instantly, so it encrypts a little bit each minute. During those minutes, isn't it conceivable that the now-encrypted files replace your old backups? I suppose this isn't really a "backup" but rather a "mirroring strategy." But for certain kinds of data -- photos, video, large media -- can you really afford to do any other kind of strategy? The other question I have is related to that first point: since ransomware can't encrypt everything all at once, how is it possible for your system to continue to function? As you can tell, I'm a ransomware noob, but it's quite interesting to me from an engineering standpoint. Does the system get into a "half encrypted" state where, if you rebooted it, it would fail to boot at all? Or does ransomware targeted at businesses tend to be more of a surgical strike, where it targets and wipes out specific datastores before anyone notices? (It's the "before anyone notices" part that I'm especially curious about. Isn't there some kind of alarm that could be raised more or less instantly, because something detects unexpected binary blobs being created on your disk?)

goldenkey 5y ago
Linux executables can be modified while still running, this is what differentiates windows updates, many requiring rebooting, from Linux updates. Once rebooted, then you realize your executables are tainted.
derekp7 5y ago
Your general backup strategy should follow something like doing full/incremental and/or snapshot based backups. So in the case of your media, if you do a daily snapshot then your daily backups would be very small. And if the media doesn't change that often you can keep weekly copies for several weeks, monthly copies for many months, and several years of yearly snapshots. The other strategy is with tape rotation. You need to have about 30x the tape storage as you have online storage, so you can keep 7 yearly backups, 12 monthly, 6 weekly (all full backups), and 7 - 14 daily incremental backups.
Dylan16807 5y ago
> I suppose this isn't really a "backup" but rather a "mirroring strategy." Correct. > But for certain kinds of data -- photos, video, large media -- can you really afford to do any other kind of strategy? Yes. Make the backup system for that big slow-changing data a moderate amount bigger than the primary data store, and then you can have months of retention at low cost. If too much data changes at once then it should go read-only and send out a barrage of alerts.

SMAAART 5y ago

There's no glamour in back ups. Even less glamour in great back up. Even less in testing back-ups. And there's a lot of glory in "slashing the IT budget with no disruptions in operations, cutting the fat is good for business".

rhizome 5y ago
This is a good nutshell of the evolution in internet systems operations over the past 10 years. The boss didn't go to GSB just to watch some unfungible college-dropout make their own schedule every day.

aborsy 5y ago

Even with successful tests and recovery, you may still have to pay the ransom, because hackers often threaten to publish or sell data.

flowersjeff 5y ago

This really hits home, decades ago - I was working at this place that did daily tape backups. I remember thinking, this is unreal - there's literally a room filled with tapes. One day, I asked if they ever had performed a recovery off of the tapes, as I questioned if the tapes were even being written to. (NOTE: Backups was not my job at all. ) Why had I brought this up? I would be in the server room and never saw the blinky lights on the tape...well.. blink. Everyone literally laughed at me, thought was a grade A moron. A year later, servers died... Pop'ed in the tape... Blank. No worries, they had thousands more of these tapes. Sadly, they were all MT. They had to ship hard drives to a recovery shop, and it was rather expensive. I left shortly after this.

wyldfire 5y ago
> Everyone literally laughed at me, thought was a grade A moron. A note for anyone else in a similar situation - a good team doesn't ridicule someone for questions like these. A responsible leader should have cited a time in the past that they did a restore or a spot check, and no one should have laughed. The laughter sounds like masked fear or embarrassment. This goes for any team. "How do we know this function of our job does what we think it does?" You should have an answer. Now, I've only worked in R&D software and not in IT. But IMO IT teams should work the same way in this regard.
- yosito 5y ago
  > a good team doesn't ridicule someone for questions like these A good team won't ridicule any questions. If you're on a team that ridicules your questions, that's a huge red flag. Get out as soon as possible!
- 908B64B197 5y ago
  > A responsible leader should have cited a time in the past that they did a restore or a spot check, and no one should have laughed. The laughter sounds like masked fear or embarrassment. ... or assigned the engineer asking questions the task of figuring it out!
- amatecha 5y ago

k12sosse 5y ago

It's so easy to backup (and test!) these days. There's no excuse other than incompetence.

nonfamous 5y ago

Fourth, lack of backups may not even be the threat. What if the hackers demand a ransom to prevent them from releasing sensitive data to the public?

TrueDuality 5y ago
Those are much rarer, and in several cases the attackers didn't actually have the data when those threats are made. That aside, what you're talking about is substantively different crime and one that is much easier to prosecute internationally than computer based crime. Espionage, and blackmail are both crimes that have been around for a very long time and have international treaties around them. That doesn't stop it from happening, but when someone is caught it's much more likely they're going to be extradited and prosecuted than for a crippling attack against a computer system. Either way, if an attacker is threatening to release your internal confidential data, would you rather have a copy of that data as well or not? Your course of action is also the same in either case, notify the FBI if you're in the US.

_wldu 5y ago

They also threaten to leak your data if you don't pay. They know a lot of orgs can restore the data and won't need it decrypted. There's no real defense against this (other than good security practices).

SamBam 5y ago
I mean, that's a whole nother kettle of fish. At that point they have you in the hook for indefinite blackmail, because after you pay they still have the days. Is that actually common, though? I think most of these ransomware cases are simply pay-to-decrypt.
- allset_ 5y ago
  Almost all of the ransomware gangs now also exfil data and use that as additional leverage.
- mm983 5y ago
  yes, double extortions are getting common because people rely on backups
squeaky-clean 5y ago
You can never prove that they won't leak your data after paying though. I'm not a CEO/CTO and haven't had to make these decisions, but from my perspective it's an empty promise that by paying them they will actually keep their word and not leak your data.
- thekyle 5y ago
  If they do leak your data after you pay, then it'll ruin their reputation and make it less likely for other victims to pay in the future.
  - axiosgunnar 5y ago
    Which is why we should do „fake“ ransomware attacks where a company „pays“ and gets „betrayed“ when the attacker still leaks some „super important“ data after the payment. What are the hackers gonna do? Sue you?

efitz 5y ago

My very first mentor when I started my IT career 30 years ago told me “your job is to make sure the backups are working. Everything else is icing on the cake.” Still true. But one caveat: your back ups can get screwed up if you back up data that has already been encrypted by ransomware. One easy way to defend against this is to have a tier 2 back up with a time delay, eg it backs up the backup files from a week ago. Tear 1 backup is just normal back up. You test it frequently of course to make sure that you can restore it but you don’t have to do any extraordinary work to detect ransomware in tier 1. Tier 2 backs up the back up, but only after a certain number of days have passed. That number of days is the window that you have to detect that ransomware has infected you. If you ever find a ransomware infection, you isolate and turn off tier 2 immediately to preserve a known clean state, and once you have everything rebuilt clean and patched, you restore from tier 2. You use tier 1 for restores in non ransomware situations because it’s necessarily more up-to-date.

Jedd 5y ago

Depressingly few organisations I've worked in or with have a clearly defined set of RPOs and RTOs for their essential systems, and similarly few have regular processes to test their backups and archives to either confirm the process works, or to determine how long a recovery would take. This stuff is all conceptually very easy to do - but politically extremely difficult to agree on the definitions, and then obtain the resources on the ops side, presumably because it's yet another of those things that fits in the category of being hidden and non-urgent, and therefore a low priority, right up until the moment it isn't.

deleted 5y ago

[deleted]

basicplus2 5y ago

This is why i have my own tape backup system in addition to cloud (someone elses computer) backup. Ready to restore everything quickly.

deleted 5y ago

[deleted]

leke 5y ago

I didn't realise backups were normally kept off site.

whartung 5y ago
I love disaster recovery disaster stories. When all the good intentions go wrong. Two from the '89 quake. 1) The smart thinking operator who equipped their machines with UPSes to keep them up through a power failure. The quake cut the power, the UPSes kicked in, the systems stayed up, the drives kept spinning, the quake kept going and buried the drive heads in to the platters. 2) System crashed, backup tapes were kept in the basement. Quake set off the sprinkler system in the storage room, soaking the tapes with rusty water. "I'm sorry, did I break your concentration? I didn't mean to do that. Please, continue, you were saying something about best intentions." - Jules

Aulig 5y ago

What if the ransomware has been hiding for a few months before it activated? You'd either have overwritten the last clean backup by then or you'd restore the ransomware too. Or am I missing something?

KirillPanov 5y ago
The article addresses this: “That is still somewhat rare,” Wosar said. “It does happen but it’s more the exception than the rule. It's something that very high-value targets will eventually have to worry about. But figuring out how to corrupt the backups without disrupting the production systems (and being discovered) is not likely to ever be a fully automatic process for the randomware goons. They'll have to invest serious time in understanding how the victim's systems work. For high-value targets, that will happen. I just walked through a telecom colo last week and saw a weird new box in somebody else's cage I'd never seen before. Later, I did a web search for the brand name. Apparently this company sells a network filer that keeps local daily/monthly/annual backups and doesn't let you delete them. Probably a good idea for an organization that just wants to throw money at the problem. Of course you can build this yourself with "btrfs sub snap" except for the part about not letting yourself (even as root) delete them.
- Aulig 5y ago
  Ah, I interpreted "corrupt backups aswell" as encrypting them aswell.

AtlasBarfed 5y ago

If you have a cloud, it's not "test your backup". It's "have an automated restore" ready. Maybe on a different cloud. With clouds, you can test standing up your entire infrastructure/system stack across even a couple hundred machines or more in automated fashion, and then tear the whole thing down.

dijit 5y ago

Is this the death of ops? Common ops mantra used to be “A backup does not exist until you’ve restored it”. Having a blob of data means nothing- being able to continually restore it and integrity check it is everything.

joefife 5y ago

That's not how modern ransomware works. It's now common to see data extracted and for the ransom to cover not disclosing your corporate data. But yes, agree sky backups in terms of restoring operations.

nazrulmum10 5y ago

We recommend creating an entire playbook so you know what you need to do to recover from a ransomware attack.

varelaz 5y ago

I think we need to treat such threats as a rear disease which could happen whenever you ready or not. We need to check for it but be prepared to have it anyway. Industry should start thinking more about insurance, negotiation and investigation services.

timwis 5y ago

Unfortunately this doesn’t cover the case where the ransomware group is threatening to leak your data unless they pay :/ It’s also good to consider—if a ransomware allows attackers to access your network—whether there’s anything stopping them from accessing (and encrypting/overwriting/deleting) your backups.

sandreas 5y ago

A huge step forward was using zfs with snapshots for my system and backup devices... zfs local fs might be still a bit dangerous, because theoretically ransomware could delete snapshots, but if you use a server / share that underlies zfs, you are pretty save. There are some decent guides for Arch: Official: https://wiki.archlinux.org/title/Install_Arch_Linux_on_ZFS https://wiki.archlinux.org/title/Install_Arch_Linux_on_ZFS Encrypted: https://forum.level1techs.com/t/manjaro-root-on-zfs-with-encryption/170428 https://forum.level1techs.com/t/manjaro-root-on-zfs-with-enc...

technimad 5y ago

"A backup not restored is a backup not made." Should be on the wall in every IT department. Together with "Snapshots are NOT backups".

teekert 5y ago

I heard from companies using ZFS, that suddenly see a massive increase in disk space usage... A results from copy-on-write for all the new encrypted files. Then they restored to a snapshot before the encryption started and were able to resume business (after a purge and decoupling of all suspicious machines). Could it be that simple? Sure snapshots are not backups, but this is not a hardware failure. In my home situation I do a periodic rsync over ssh to a Pi3 at my parents place. It's super simple but I can just browse the sshfs to check if stuff is there and working. And when I need it, I drive there and get the disk itself. Sure, this is not relevant for a large business, but for us self-hosters it is a nice solution. The backup is manual, by choice, so I never overwrites accidentally.

raxxorrax 5y ago

Plainly the best defense against ransomware. Had two attacks that probably could not have been avoided. Some employee will at some point open that funny looking mail. Scammers know our names and mail signatures (the colorful text, not the crypt sig for mails). Maybe they get some detail wrong, like department or a contact address, but otherwise the mails looked genuine. A sensible backup solution made us come back within 30 minutes of severe attacks. The police tried to negotiate with the attackers but sadly couldn't get them. But at least they didn't get any money. We have a bought backup solution that is quite expensive that does snapshots every 15 minutes I believe. Worth it.

mvkel 5y ago

Paying the ransom and then plugging the hole is probably far cheaper than spinning up and maintaining a hardcore backup-perfect stack.

marto1 5y ago

I believe we are unfortunately at the “have backups” stage still civilizationally.

HeavyStorm 5y ago

You mean, my cloud provider isn't doing it for me?