38 ms·
How to survive a ransomware attack without paying the ransom
- lobster45 6y agoThis is not really surviving. What do you need to do is prepare and to have off line backups
- ntucker 6y agoPaywall didn't let me read it, but as I clicked, I thought "this article should be one sentence about doing backups."
- FlyMoreRockets 6y agoOffline backups have been a thing for decades. Why is this not standard practice? Especially for a technology company like Garmin. It can't be about cost savings, businesses still pay for insurance and security systems. For that matter, offsite backups should also be saved in case of fires, floods, tornadoes, theft, etc...
- cpeterso 6y agoOffline backups are not a complete solution. What if your backups are infected with the virus? Even if the backups are uninfected, your IT department has to manually scrap and rebuild all your computers from data centers to the warehouse to the receptionist. And in the meantime, like the article described, you have to pay your employees and suppliers and continue to ship products to customers.
- FlyMoreRockets 6y agoAn important part of any backup strategy is testing your backups on a regular basis. Perhaps it could even be automated...
- cranekam 6y agoI think the point here is that it's not as trivial as having an offline copy of your SQL DB or whatever. If the ransomware has encrypted a huge chunk of your infra the chances are that you no longer have anything to restore the backup to — maybe your configurations are encrypted, your DB hosts aren't up, user accounts etc are missing. Assuming that only user data is affected and can be easily restored likely falls very short of the full picture. I expect the folks at Garmin are faced with an infrastructure that looks like a grenade fell into it.
- noir_lord 6y agoWired did a phenomenal article on the Maersk attack https://www.wired.com/story/notpetya-cyberattack-ukraine-russia-code-crashed-the-world/ https://www.wired.com/story/notpetya-cyberattack-ukraine-rus... company I worked for was affected as we had shipping containers on the water at the time and it was chaos but they recovered.
- SV_BubbleTime 6y agoBackup price for 8TB is cheap enough. Backup price for 8PT does not scale well. I don’t know how much data Garmin has company wide. But it’s a lot different for me to consider offline backups as a simple service than a company this size and complexity.
- gpm 6y agoThere are plenty of companies that can backup 8PB of data from a wide variety of sources for you, and make it a relatively staitforward task to interegrate with them. There is complexity, yes, but it's mostly a solved problem. Disclaimer: I work for one.
- saalweachter 6y agoThe easiest way to back up 8PiB of business critical data is to be a business with a need for exabytes of non-business-critical data. I think the people who are the worst off have petabytes of business critical customer data, but don't do massive datamining projects on top of this. Then you end up with a data center that is 90% (business critical) prod, and triplicating that becomes much more relatively expensive than having a data center that is 20% prod.
- bloomingeek 6y agoExactly, Although not always practical, if kept virus free (which of course is possible), offline back ups are the best solution because it's never broken into. ANYTHING online is risky, we all know this, so why do corporations still practically ignore the obvious?
- beefhash 6y agoGiven there's been a recent trend about ransomware not only encrypting, but also exfiltrating data, backups won't save you from the bad PR of the leak.
- piokoch 6y agoGarmin CEO at al must be reading this impatiently, looking for some clever-magic clue, which is not gonna arrive, I am afraid. Meanwhile Garmin watches users (like me) are wondering how it is that syncing my watch that I have bought with an application on my smartphone that I have bought requires presence of some distant online service. I can understand that some parts like "social" stuff might depend on some central service but, hey, something simple like syncing ones excercise achievements between phone and watch? Really? Who has designed that?
- pengaru 6y agoObviously there's a business incentive to maximize gatekeeping. Even if the gate's left wide open (for now), having control of a gate is of tremendous potential value. Just imagine how much more $$ one can elicit in an acquisition if potential buyers may add tolls to an already established and well-traveled gate.
- wlll 6y agoWhat's the backstory for this?
- latchkey 6y agoPart of the backstory is that in 2010, they fired all the Garmin Connect staff [0]. More recently, they've been shut down [1]. [0] https://news.ycombinator.com/item?id=1196996 https://news.ycombinator.com/item?id=1196996 [1] https://news.ycombinator.com/item?id=23926289 https://news.ycombinator.com/item?id=23926289
- dehrmann 6y agoI forgot about 0. I had to double check, but a high school friend was on that team.
- rectang 6y agohttps://www.garmin.com/en-US/outage/ https://www.garmin.com/en-US/outage/
- larrymcp 6y agoHow is ransomware able to spread to all the PCs in a company? (Especially PCs at different locations around the globe) The malware needs to execute itself on each computer. But I would think this would be thwarted by hardware firewalls as well as apps like Windows Firewall. If my PC at work gets infected, somehow it can magically infect the guy down the hall's PC too? I thought that was made impossible years ago.
- core_dumped 6y agoFirewalls can only protect against what's known. Once you've invented or discovered a method the firewall doesn't know about, you're trusted as much as any regular program. Sometimes even changing the binary or payload slightly will thwart some firewalls because they're precise machines looking for precise signatures. It's not super easy to get past a firewall with a known vulnerability, but not impossible. With a 0day the firewall is almost irrelevant.
- packet_nerd 6y agoThis is true regarding "next-gen" firewalls. But, if you design a plain old segmentation strategy with simple but well thought out allow/deny rules, then a firewall will be pretty valuable in many situations. Extreme example: you can think of an air gap as a "firewall" with all deny rules. Air gaps are pretty secure. (Yes, there are still way's in but finding them will be many orders of magnitude harder than finding a 0day in a "next-gen" firewall). Another example: I put all printers in a dedicated VLAN and block all traffic in and out except specific print ports from the print server IP only. In practice, way more secure than any "next-gen" firewall will ever be.
- halfcat 6y agoProbably through Active Directory, which has the ability to deploy software. If a domain controller was compromised, the payload could be pushed out across the board. Endpoints like PCs and servers check in with domain controllers at recurring intervals, so even if all endpoints are behind firewalls and can’t talk to one another, they still reach out to domain controllers periodically to pull down configuration updates and so forth.
- neonate 6y agohttps://archive.is/Rfcha https://archive.is/Rfcha
- Tiltowait-- 6y agoEasy: restore from backups.
- ourmandave 6y agoAnd format every hard drive in the building first. Also replace your IT security provider and/or person.
- gav 6y agoSometimes not that simple, the attack could have corrupted or deleted the backups, or the backups themselves could be the source of reinfection.
- matsemann 6y agoWhat if they hacked you months before pulling the trigger? The article mentions they were hacked in December and the attack launched in March. Restoring a backup would then still leave the hackers inside. And even if most data were backed up, most computers still have to be wiped and reinstalled. I don't think most companies backup the entire disks off all employees, it's normally just a dedicated file area. So while the data can be restored, the IT department still have to set up hundreds of computers for all kinds of different workers or machines on the spot. Nothing is ever easy, don't be so dismissive about things you haven't thought through.
- viraptor 6y agoCompanies of non-trivial size often have (and should have) a system allowing for remote device management. Which means: - It should be easy to reinstall to a known good image with all the relevant software, settings, drivers, etc. then restore the backed up data. This is relatively common in corps. - Once you observe the malware and know how it reaches the C&C server, you can push rules blocking that host or block the bad binary network-wide. Of course there will be companies that didn't have good enough system in place and once exploited are doomed.
- marcinzm 6y ago
- LifeLiverTransp 6y agoDont give into your lower instincts as a company to centralize all the things. Air-Gap them in the architecture?
- einrealist 6y agoI wonder how many of such attacks can be prevented / starved in a zero-trust network.
- fortran77 6y agoIsn't a ransomware attack no different from a catastrophic disk drive failure? You reformat and restore from backup. Of course, the companies profiled in that article had all their computers infected, so it could take some time. Still a recovery boot disk could be distributed and a clean image restored over the network.
- PeterisP 6y agoNo, because you can't consider your backups as a "known good state". A malicious attack is fundamentally different from a disaster or accident. You should expect that any backup of systems (instead of backups of 100% pure data) will contain backdoors, that any weird systems (routers, printers, phone centrals) may be compromised even if they seem fine, and that the credentials of all the employees and any private keys/certificates have been exfiltrated, so they need to be changed.
- pmiller2 6y agoEven “100% pure data” isn’t necessarily safe. Word documents, Excel sheets, PowerPoint decks, etc. (and their Google Docs counterparts) are all suspect, because they can contain embedded code. Some “data” formats are really not data formats at all, but code which produces the data you use (e.g. PDF, Postscript, or any Excel sheet with formulas). It’s even possible to corrupt certain otherwise inert data files in such a way as to cause malicious behavior by exploiting bugs in the software that reads them. So, yes, while you’re technically correct that 100% inert and uncorrupted data files are safe, you have to prove that those files are not corrupted. And, so many data formats either are code or contain embedded code, so these need to be treated as suspect until proven otherwise, as well.
- bobbylarrybobby 6y agoIs there some other solution to this than to cryptographically sign 100% of the files in your system with a key that is only written down on paper?
- PeterisP 6y ago
- naple 6y agoI consider the modal on the bloomberg site a ransomware. Can't close till you pay. Joking :)
- dheera 6y agoYeah screw that paywall. Paste this into the console document.querySelector('.paywall-inline-tout').remove(); document.querySelectorAll('p').forEach(e => e.style.display='');
- dencodev 6y agoI prefer control-W
- coronadisaster 6y agoOne of my extension appear to be blocking it because I didn't know it existed
- zelly 6y agoDisable JavaScript and it goes away
- rectang 6y ago> In other words, it’s less a question of how to stop hackers from breaking in than how to best survive the inevitable damage. There doesn't seem to be conventional wisdom about how to build systems that are easy to restore. How do you optimize for recovery after an attack? How do you ensure that you've eliminated all the backdoors? My guess is a combination of "continuous restoration", version controlled code, and a complete separation of code from data. I want to read books about this but they don't seem to exist.
- t-shaped 6y agoIf there's a book that you want to read, but it hasn't been written yet, then you must write it. - Toni Morrison
- jeffbee 6y agoJust having a decent and reasonable way to nuke and pave machines goes a long way. Most organizations don't have a good way to shoot a machine in the face and have it back up and serving in 2 minutes. Most organizations are absolutely married to "stateful services" like SQL databases with local storage, that are hard to kill, hard to restore, and give attackers a place to hang out. If you can take all your hosts down and bring them all back up quickly, that gives you at least one tool for disrupting the attackers.
- jonpurdy 6y agoI was thinking about this recently. Tight, centralized control over servers, employees computers, and devices is hard to set up, hard to manage, and a huge surface where misconfigurations can allow attackers to jump right in. Decentralization is the key. Microservices, or segregated services, stateless (as much as possible), and perhaps even partitioning groups of users into totally separated instances. One group gets attacked and service only goes out for the 1000 users in the group. Of course, infrastructure costs would go up but maybe not that much (since you need less resources for 1000 users than 100,000). This is relatively easy to build from scratch these days thanks to various IaaS providers and DevOps tools (obviously hard for established companies with legacy tech). Then, keep employee computers totally separate from production servers. Let employees back them up themselves, especially since so much can just be stored in the cloud (I'll get there next). Forget about VPNs where everyone can talk to everyone else. Don't try to save money by hosting your own Jira and Bitbucket instances. Pay extra for Atlassian to host for you, then let them deal with security (actually it's cheaper from what I remember). Companies already pay for Office 365 and don't self-host that. Don't host your own email servers. Just focus on the core of what the company needs to do and that's it. This way you spread your attack surface across a whole bunch of services that are better than you at security, and you get the benefit of not having to deal with other issues. If Atlassian or Cloudflare goes down, no big deal; they'll fix it. And all your other stuff still works.
- linsomniac 6y agoThe unfortunate thing is that the ransom probably is priced such that it's cheaper than the company resolving the problem on their own. Or the company would just resolve it without paying the ransom. On the other hand, it's bad on many fronts if the company just decides it is the cost of doing business, and doesn't do a great job of securing their systems in the future... Many companies just get by, rather than doing serious security design. How do you change that culture in a company? Will paying the ransom do that? Probably if it only costs $1M to do. If it costs them $100M to do, would they do it?
- abcok1 6y agoYou can't because of bitcoin. Bitcoin is what criminals dream of.
- hprotagonist 6y agoIt is always a temptation for a rich and lazy nation, To puff and look important and to say: – "Though we know we should defeat you, we have not the time to meet you. We will therefore pay you cash to go away." And that is called paying the Dane-geld; But we've proved it again and again, That if once you have paid him the Dane-geld You never get rid of the Dane. http://www.kiplingsociety.co.uk/poems_danegeld.htm http://www.kiplingsociety.co.uk/poems_danegeld.htm parenthetically, this is equally true of toddler-geld.
- dzonga 6y agosome of these companies, need to start suing microsoft. since it's usually windows affected by these malware attacks. if microsoft wants to keep serving the majority of the corp world, they need to have an os, based on user space system. i.e each program runs in it's own sandbox. and any data passed is via message passing.
- user5994461 6y agoMicrosoft has pretty good security nowadays. It's the fault of companies for never upgrading their machines, giving full administrators access to every employee and using Admin123 as the domain administrator password account. If we believe the article, the virus came from an attachment in an email to a random employee. Why are executable attachments not blocked? Why is an executable running as an unprivileged user able to storm through every computer in the company?
- raverbashing 6y agoBut did they? Were their machines updated? Did they have some "corporate AV" solution that was useless? > Why are executable attachments not blocked? Because there are dozens of weird Windows extension types that execute automatically on Windows, though yeah the attachment should have been blocked and a customer service machine should have a whitelist of programs that need to run (or only run signed ones)
- jankiehodgpodge 6y agoI've seen hundreds of enterprise networks and you'd be surprised how many of them have passwords in public shares, no configured firewalls or software policies and only file level backups. We obviously can't know right now what Garmin's setup was but I'd wager they were guilty of something.
- WrtCdEvrydy 6y agoI find it very interesting the Volume Shadow Copies and VLANs are basic tools that have been around forever, cost very little and can mitigate a lot of ransomware attempts. There's no reason for the secretary's computer to be able to connect to the onsite SQL server... unless she uses an application that uses that SQL server.
- zamadatix 6y agoThe complicated part isn't figuring out that you should segment access or finding technology that lets you do it it's actually knowing what to segment in a way that balances risk with speed and cost. The same is true for most things. Problems are often well known, solutions are often understood, but doing things is where the actual work is.
- arthurofbabylon 6y agoJudging by comments hackers aren’t commenting on hacker news anymore.
- sytelus 6y agoGood story, bad title. There is no real lesson on how to survive ransomware except that company called Hydro was able to use employees to rebuild some of its data needed for running plants. Most companies deal with databases that cannot be similarly rebuilt in just days like that.
- tluyben2 6y agoI know this is always contentious but are there any of these ransomware attacks on non Windows machine? I mean prominent ones? I understand everyone is running Windows on the desktop, but why are Linux servers not targetted by the same thing as they are prominent? I know they get hacked all the time, but I never read stories like this about them. I read that mongo was hacked (and yeah, using mongo, sorry but...) which probably ran on Linux; however pure ransomware attacks I cannot find outside Windows. People keep saying that if other devices would be as popular, they would attack them; but for instance my mother has an ipad, android phone and a windows laptop, and the only (penis enlarger.....) malware is in windows which has an up to date AV. Android is more popular than Windows, Linux on servers is as well, iOS is as well. And yet all the crap is always Windows. I do not get it.
- dangelov 6y agoHere's an article about a recent Mac ransomware, it was on HN a while back. https://www.wired.com/story/new-mac-ransomware-thiefquest-evilquest/ https://www.wired.com/story/new-mac-ransomware-thiefquest-ev... But still, a lot less of these stories than for Windows.
- tluyben2 6y agoYes, but my question was specifically about a high profile one; so huge company or huge money stolen. This is just 'yes it exists' but nothing was done with it.
- PeterisP 6y agoRansomware attacks absolutely do target Linux servers because one needs to take down all the servers to have a proper business disruption for which someone will pay a million dollar ransom; in all the recent prominent attacks Linux servers were taken down as well. Perhaps there's some issue with what you mean by "pure ransomware" - if you mean automatically spreading worms, then those aren't that relevant, prominent examples like Petya was four years ago; NotPetya was not ransomware but a destructive weapon, etc. In the current environment, and also in the attack described in this article, a "ransomware attack" means a takeover of your systems by a ransomware crew of hackers manually working on your specific network. They generally start with a spearphishing which targets Windows desktop machines because usually the easiest way to target Linux servers is through client-side attacks, obtaining user credentials and a foothold inside the network that helps with firewall restrictions.
- deleted 6y ago[deleted]
- unnouinceput 6y agoThis quote left me dumbfounded, marveling at sheer level of incompetence (or bureaucracy) some places have: "Three weeks after the attack, Hydro had a total of four functioning PCs in all of the U.S."