19 ms·

US city rejects $5.3M ransom demand and restores encrypted files from backup

467 points by GiulioS 7y ago

hartator 7y ago
> "I decided to make a counter-offer using insurance proceeds in the amount of $400,000, which I determined to be consistent with ransoms recently paid by other municipalities," They didn't say no. It's odd there is so little recourse against things like this.
- cardamomo 7y ago
  You're right, they didn't outright give a hard "no," but the story gets more interesting after New Bedford's counter offer. By attempting to negotiate, the city bought valuable time that they used to harden and restore their systems.
  - hartator 7y ago
    Yes, but my understanding of the article is the city was negotiating in good faith. They were willing to pay the $400k. Still a lot of money.
    - logicallee 7y ago
      Any reason laws can't be introduced making it highly illegal to pay a ransom, with serious criminal repercussions (same for government officials doing so)? After all paying a ransom is very literally funding terrorism, in the most direct possible way. "Whosoever knowingly pays a ransom", etc. Of course those seeking a ransom could make their language less and less clear, so it's no longer clearly a ransom. But isn't that still an improvement over the status quo? Maybe I'm missing something, but I don't see why it should be legal to pay.
      - TaExtra 7y ago
        >After all paying a ransom is very literally funding terrorism, in the most direct possible way. Not everything is related to terrorism.
      - parsimo2010 7y ago
        Because you shouldn't criminalize being a victim. I agree that it's probably a bad idea to pay a ransom, since that just supports the success rate and makes ransomware more popular. But do you really think that you should throw a grandparent in jail because paying a ransom is the only way they can get their digital memories back? Are you going to throw all of City Hall in jail because they needed to pay to get their tax records back? The best option is to have good backups and not give into the demands, and that should be encouraged. But criminalizing people because they didn't adhere to the best digital practices is a bad idea.
        paulryanrogers 7y ago
        For private individuals perhaps it shouldn't be criminal. But for businesses responsible for other people's private information? Yes, I think there is professional negligence if they cannot even restore backups in a timely manner. (Or cannot detect infections before they become so widespread the whole thing falls down.)
        logicallee 7y ago
        The part I would criminalize is the money-wiring part.
        parsimo2010 7y ago
        The action that a business should be punished for is not keeping their customer's information secure. Paying a ransom to recover after they are the victim of a crime should not be what they are punished for. If they were doing everything right but still got hit by ransomware I don't think they should be punished. If you can prove that they wrote their login info on a sticky note and left it in the main lobby then you should punish the bad security practice if it exposes other people's information. Several parts of the world have kidnapping and piracy issues, and you can buy kidnapping/piracy insurance to pay ransoms in case you are the victim of such a crime. I think most people in the world acknowledge that sometimes bad things happen even when you take reasonable precautions, and you shouldn't be punished just because you were the unlucky one. Most security experts agree that no computer system is un-crackable, there are just varying levels of sophistication and access needed to do so. We've even seen that Stuxnet was capable of jumping air gaps. If a business had such good security that their database and backups were air-gapped but still was hit by ransomware do you think that they should still be fined?
    - deleted 7y ago
      [deleted]
  - solidasparagus 7y ago
    That didn't make sense to me. What did negotiating actually buy them? I don't see any indication that the attackers paused the attack.
    - rdc12 7y ago
      Little to gain in restoring from backup if the same attack vector is still open. So it was really just a bit of misdirection.
    - blaser-waffle 7y ago
      It bought them time to figure out the root cause, fix it, and check if restoring backups is a viable option. Smart play.
  - rolltiide 7y ago
    Restoring a backup they already had? Get outta here
  - rebuilder 7y ago
    That sounds like whitewashing to me, to be honest: The city were willing to pay a ransom, and are now trying to make it sound like a shrewd move instead of negotiating with criminals.
- deleted 7y ago
  [deleted]
- VectorLock 7y ago
  I wonder if they just didn't check their email or were on vacation that day. That is a HUGE payoff to just ignore.
- mkagenius 7y ago
  > They didn't say no. Most likely they don't know english and only have pre-written formats to talk to the victim.
jumelles 7y ago
"Restore from backup" really ought to be the default response to ransomware. That will probably never happen though...
- TallGuyShort 7y ago
  It's very liberating to know you can do this. You can sleep at night knowing your confidential data is safe. From a business continuity standpoint, worst case scenario you get a new machine, restore, and you're back in business. When I trade-in my work laptop for a new one I love the feeling of handing them my old one, picking up my new one, and not looking back. Works just as well for ransomware, other viruses, ruined hard-drives, etc.
  - new_guy 7y ago
    But whatever attack vector was used is probably still wide open on your machine even after restoring a backup. Most people will just get infected again straight away.
    - technofiend 7y ago
      Well, maybe. On the other hand forcing someone to update often reveals that updating does not in fact break the world, particularly in linux-land where API updates are supposed to not break userland. Many times users who don't understand the risk incorrectly assume worst case scenario.
  - reustle 7y ago
    > It's very liberating to know you can do this. You can sleep at night knowing your confidential data is safe If ransomware was able to infect the machine, then I doubt the data was safe either
    - hobs 7y ago
      Exfiltrating is insanely more difficult especially of "the useful stuff" - and even then you have to sell it or use it to embarrass your target, it's pretty difficult unless you target people ahead of time - easier to just interrupt business.
wmf 7y ago
I don't understand what I'm reading here. Why did they offer to pay $400K to recover ~100 PCs if they had backups? Was it so expensive to restore those PCs? I guess the good news is that we can now sell backups as "anti-ransomware" cybersecurity.
- warent 7y ago
  Maybe they first had to verify the backups actually existed like they should and were up-to-date enough
  - everybodyknows 7y ago
    Indeed. I've worked at a big corp where it was belatedly discovered that backups were broken. A second reason would be to give the attackers reason to pause, in case they somehow had a second bomb to throw, while IT hardened defenses: >Since the attack, the city has installed additional security software and is developing new protocols.
    - AnIdiotOnTheNet 7y ago
      > Indeed. I've worked at a big corp where it was belatedly discovered that backups were broken. Ancient wisdom: If you don't test your backups then you don't have any backups.
  - frankharv 7y ago
    My thoughts exactly. I bet they did not have any backups for the Forth of July weekend so anyone doing city business that weekend may have to redo any paperwork or payments. Small price to pay to laugh at these clowns. What exactly are the FBI doing about this. Perhaps if they got off their asses and quit investigating Presidential piss parties they could find this email recipient. How hard is it for the FBI/NSA to trace an email. They just want to cry about going dark instead. If foreign put a CIA hit on the computer terrorist.
- fuzz4lyfe 7y ago
  >Why did they offer to pay $400K to recover ~100 PCs if they had backups? "Backup is lets says weekly and someone high up the chain really wants the power point he put together on Tuesday." is probably what happened in my experience.
dsfyu404ed 7y ago
Meanwhile Springfield payed with a check but the check bounced. Fitchburg just decided that the affected portions of government didn't actually need doing and nobody noticed. Worcester didn't even know they got cryptowalled because they couldn't see their screens through the pot cloud. Providence didn't pay a dime since the mayor knew a guy who knew a guy. (And before anyone gets offended, this is a joke based on the stereotypical attributes of the other local cities.) Edit: You people have no sense of humor.
quickthrower2 7y ago
Why not make it a crime to pay the ransom demand for cryptolocker attacks? This way the flow of money to these gangsters will stop. If you get hit with a cryptolocker and you have no backup you simply lose that data. Or you can pay the ransom, get your data back and go to jail. While this might seem unfair, it might stop 1000 other attacks because there will be no money in it. It would be for the greater good.
- pfisch 7y ago
  What happens when a hospital gets hacked? This law could kill people.
- slg 7y ago
  Why don't we make getting mugged illegal while we are prosecuting victims. If you give your wallet to a mugger, you go to jail. Then no one mugs anyone anymore, right? /s People pay ransoms because they want back whatever is being ransomed. Making it illegal to pay the ransom isn't going to stop that. It will just push everything underground and make it harder to catch the ransomers.
  - arcticbull 7y ago
    Mugging involves threat of physical harm. Getting ransomeware'd doesn't, and if cities stopped paying then this kind of attack would stop. It's more like outlawing giving muggers your wallet when you're dressed like Iron Man.
    - dmix 7y ago
      What happens if the ransomwares don't stop? Do you spend a bunch of money on investigations to see if some thousands of small towns or little agency somewhere kept if quiet because the cost otherwise is huge? Up the fines the tax payers will pay when they get caught?
    - olliej 7y ago
      I don’t like this argument: if the ransomware takes out emergency dispatch, hospital systems, etc it gets much closer to what you’re claiming.
deleted 7y ago
[deleted]
H8crilA 7y ago
See also Matt Levine's take on ransoms (scroll down): https://www.bloomberg.com/opinion/articles/2019-08-27/the-libor-change-is-coming https://www.bloomberg.com/opinion/articles/2019-08-27/the-li... And the linked article: https://www.propublica.org/article/the-extortion-economy-how-insurance-companies-are-fueling-a-rise-in-ransomware-attacks https://www.propublica.org/article/the-extortion-economy-how... Tl;dr: perverse incentives, paying some ransoms is in the interest of cyber crime insurers, as it expands the cyber crime insurance market. Also there's more ransomware crime now that the word is out that insurers do pay out ransoms.
zelly 7y ago
The real solution is for them to hire out for their ops. No reason these rag tag city governments should store their stuff on-prem. Until then these okie dokie city bureaucrats will keep getting pwned.
- cat199 7y ago
  not sure how you are getting that from this one - sounds like this 'rag tag' govenment didn't get powned (completely) and was able to rely on good it practice to recover with minimal impact..
  - zelly 7y ago
    They got the victims' hard drives. Regardless of what happened besides that, that's a pwn. What if the police chief had emails admitting to bribes or worse stuff on his pc? What if the hacker had an enemy he wanted framed and arrested?
    - dsfyu404ed 7y ago
      >What if the police chief had emails admitting to bribes or worse stuff Unless he's admitting to doing something that is directly counter to the platform of the ruling party (e.g. aiding ICE, rubber stamping CCW permits, racial profiling, etc) nobody would care. There would be some token outrage but it would mostly be business as usual. This is just how MA works. There's very much a "well that's government, nothing you can do, no sense getting bent out of shape over it" attitude by the majority of the population in MA. For example, the state police overtime fraud is a recurring (every 1-3yr or so) "scandal". The latest instance is basically out of the news already.
- snazz 7y ago
  I don’t know about that. They’re always going to have endpoints; maybe it’s better that they don’t get sucked into a multimillion-dollar AWS contract they didn’t need? I agree that they need better IT consulting (similar to many small businesses, I would think).
olliej 7y ago
Honestly this success might encourage city governments to make sure their backup systems work better. The counter offer of 400k was presumably the break even point for the cost of losing X days of work (depending on what was lost that could involve manually recovering things like tax payments, billing, tickets, etc)
hello_tyler 7y ago
Why is a backup system so hard for people (especially businesses) to understand????
- stanski 7y ago
  Only other people need it.
Gustomaximus 7y ago
I wonder if a law that you can't pay ransom money for files should be passed. This should work to take any government and larger companies off the target of these groups as they are likely to obey these laws. Kinda like the we dont negotiate with terrorists approach.
- jasonhansel 7y ago
  Yep. I'm actually surprised it isn't already illegal, given that the money is likely being used to fund organized crime, rogue states, etc.
  - blueadept111 7y ago
    Not only that, there's money to be made by insiders leaking (or even creating) security holes for the hackers to exploit and then getting a kickback.
- d357r0y3r 7y ago
  Absolutely. It's sort of the same as the "We don't negotiate with hostage-takers" policy. Once you negotiate, it's now a strategy known to work and you'll signal how good of a tactic it is.
  - elchin 7y ago
    Exactly. It's going to incentivize more hacking
shameshame 7y ago
The city was extremely lucky the attackers played their hand too early. If servers had been encrypted it could have a lot worse.
auslander 7y ago
No Windows no cry :) Seriously, how can one compromised machine (inevitable) infect all others? Isn't there host firewalls on every machine?
- SmellyGeekBoy 7y ago
  If you believe that ransomware doesn't exist for macOS and Linux I'm afraid you've been misinformed.
  - auslander 7y ago
    Well, there is no SMB v1 on either, main propagation bug. Next propagation tools are SSH bruteforce, subnet scan for vuln apps, Domain admin creds stored locally and sniffing NTLM hashes from network interface. Again only SSH and app vulns are viable in non-windows. I was asked once to do reference design of Windows on AWS. After I learned how many ports has to be open for every machine and all of them had to be in same network as Domain Controller, I quit my job. > If you believe that ransomware doesn't exist for macOS Example? I have not seen any.
    - lgl 7y ago
      Absence of evidence != Evidence of absence
      - auslander 7y ago
        but is a pretty good guideline :) If Windows got ransomed every time and MacOS is never, that is interesting, right? Windows holds perceived monopoly on company-wide identity control. There are Mac solutions: https://www.jamf.com/products/jamf-pro/deployment/ https://www.jamf.com/products/jamf-pro/deployment/
      - swebs 7y ago
        unfalsifiability != evidence https://en.wikipedia.org/wiki/Russell%27s_teapot https://en.wikipedia.org/wiki/Russell%27s_teapot
gesman 7y ago
Key: backup Do you ... backup?
rsync 7y ago
ZFS snapshots are immutable (read-only) for normal users. So, if you had your data stored on a cloud storage platform that created and maintained ZFS snapshots, Mallory could gain all of the credentials and still not be able to touch your daily/weekly/monthly snapshots. Now, if only there were a cloud backup platform that included zfs snapshots ...
- masklinn 7y ago
  > ZFS snapshots are immutable (read-only) for normal users. Unless you've given them "destroy" that is. > Now, if only there were a cloud backup platform that included zfs snapshots ... I see what you did there...
  - nikolay 7y ago
    Yup, he's the guy behind https://www.rsync.net/ https://www.rsync.net/
- icebraining 7y ago
  What if someone uses the credentials to file a support request in that platform asking the snapshots to be deleted?
  - jdnenej 7y ago
    Hopefully anyone with the ability to delete them would decline such a request.
    - icebraining 7y ago
      So the platform would keep data against the wishes of the client? That doesn't sound good.
      - TomMarius 7y ago
        More like it would include a more elaborate, person to person process.
peter_retief 7y ago
Restore from backup isn't as easy as it sounds, well done to a disaster recovery plan that actually works
auslander 7y ago
Great read: https://www.wired.com/story/notpetya-cyberattack-ukraine-russia-code-crashed-the-world/ https://www.wired.com/story/notpetya-cyberattack-ukraine-rus...
mtgx 7y ago
Cities need to stop wanting to become "smart" with little thought put into it or by using vendors that are friends of friends.
tastroder 7y ago
Could somebody contextualize how targeted this particular attack really was? According to random Google result #1: https://www.2-spyware.com/remove-ryuk-ransomware.html#qm-h2-4 https://www.2-spyware.com/remove-ryuk-ransomware.html#qm-h2-... the specific malware distribution is unclear but likely involves email attachments and/or vulnerable and exposed RDP. While reporting on ransomware cases often sounds like targeted APTs, more often than not the details in these stories read like "we didn't bother to pay enough admins to actually patch and secure our systems" and "we didn't train our users not to click on every random attachment".
pilif 7y ago
Do I read this right? They offered to pay $400K even though they had backups to restore their data from? Does this mean that the restore operation cost more than 400K or was this an incredible sample of laziness? They'd rather pay the ransom to a criminal than invest the time it takes to restore backups?
- bluescrn 7y ago
  Even with good backups and a prompt recovery, you could be looking at losing a day or so of data/work across the affected organisation. If the system had been backing up encrypted files for days/weeks then there’s a much higher cost to restoring from older backups
  - SahAssar 7y ago
    Making sure the systems are actually clean after getting the files decrypted costs too though. The comparison would be (due diligence to make sure systems are clean + ransom) vs. (restoring backups + lost work).
iandinwoodie 7y ago
So they offered $400,000 for restoring what was most likely a week of lost work for 158 employees. That works out to $2,531.65 per employee for that week. Is the average salary of the compromised employee $131,645.57 ($2,531.65*52)? It sounds like the town was just willing to throw a lump of cash at the attacker based on what other victims had paid with no regard to what the lost work was actually worth. I know the attacker did not accept and it allowed them to strengthen their security and etc., but it bothers me that the attacker could have just gotten a $400,000 payout. It’s almost as if the moral of the story is “even if a town government had backup system in place, you can still pull in a few years of income with a ransomware attack!”
- duiker101 7y ago
  I guess accepting it would set the precedent that they are easy to lowball? Also, it might even be a lower sum for the city, if they need to redo the work, the employees will already know more or less what to do and it might be faster.
- Zenbit_UX 7y ago
  You would lose more than a week of work if your entire team is no longer doing this week's work to make up for last week's. Let's say they instead of doing this week's work, they do last week's, next week they're still a week behind and you have to start paying 100+ government employees overtime in order to regain the catch up on the remaining week. This could take a while to catch up and be incredibly expensive. The $400,000 wasn't taxpayer money it was their insurance companies offer to make this go away.
  - dsfyu404ed 7y ago
    Local government work is mostly cyclical and the "customer" is captive. It's not the end of the world if they bill someone a day late or have to re-add the new librarian to the payroll system. It's not like people can up and switch to a new government because they don't like the quality of service. Based on having grown up in the region and having, um, "connections" to state and local government I think it is highly likely that their desire to not throw out work was based around reasoning around how their public image would be affected if a batch of fines/taxes/fees/bills had to be waived. Cutting people (collectively, waiving something on a case by case basis is fine) a break because the government screwed up is kind of a non-starter because of the possibility of setting a precedent.
ashelmire 7y ago
Paying hundreds of thousands to restore 158 backed up machines would be absurd. Surely a week of work for those 158 employees isn't worth that. They've gotta put a financial analyst on these issues, and not leave it up to panicking executives. And look at how greedy the attacker was. Missed out on a 400k payday.
- raxxorrax 7y ago
  I would like to see your calculation for that. My current company is much smaller in size and had 2 ransom attacks that I saw happening (in ~2 years). We use nimble as a backup and it is quite expensive (hundreds of thousands actually, depending on infrastructure) but worked flawlessly to restore all our systems in a short time. You will loose a few hours of work though. Attack vectors are the usual: mail with infected attachment and users with too many rights on documents. The attackers even know names of people working at the company and use correct mail signatures (not actual mail signatures, just the colorful stuff you put at the end of every business mail). If production stopped for a week, the contractual penalties alone would probably eat up sums like that.
CryptoPunk 7y ago
Cryptocurrency is hardening IT infrastructure.
mxuribe 7y ago
Kudos to the city of New Bedford!
aitchnyu 7y ago
> Because the attack happened at night, most of the city's systems were turned off and the ransomware was unable to spread. I used to complain Indian banks used to allow fund transfers only during working days and office hours. Now I see why it was useful.
elchin 7y ago
Why isn't it mandatory for government orgs to use cloud?
- Shivetya 7y ago
  I do not agree they should use "the cloud" but what is all too common is local, city, state, and even federal governments, love to enforce regulations and laws upon others they do not obey themselves. the first rule of laws and regulations should be, government must adhere to the letter if they are to be enforced upon others seeking penalties if compliance is not met. it is really difficult to hold government agencies responsible, it doesn't even have to be computer security, just look at the number of cities in the US who violate lead levels in water. even better, in the US, a lot of what happens can be protected from suits by sovereign immunity. So keep it simple, what is required of private organizations and people is the minimum standard that a government agency must meet
- SahAssar 7y ago
  Why would it be and how would you even define "the cloud"?
msmerberry 7y ago
It sounds like they played every card right, and that's probably not by accident. I'd be fascinated to see a case study made out of this - especially with a breakdown of how they performed against their CSIRP - because there's no way they didn't tabletop that plan at some point, and it paid off in the end. Sort of like insurance, eh?
avivdeg 7y ago
NetApp has a solution for all of that: Cloud Volumes ONTAP is a virtualized storage platform running on AWS, Azure, and Google,it consumes native cloud resources and it provides NFS/CIFS/iSCSI. Cloud Volumes ONTAP has the best snapshots out there, immutable and without any resource penalty. you can take any size snapshots (or restore) in seconds. you can also create clones out of these snapshots, so you can check if that data been affected or not, again in seconds. Adding to that Cloud Manager's Ransomeware protection that blocks known Ransomware files. In short- this is the best solution out there for any hybrid/ cloud and it can actually be cheaper than free, if you have enough capacity, due to all of its storage efficiencies like dedup, compressions and compaction, with auto-tiering of unused blocks to the checper object storage,.

467 points by GiulioS 7y ago

> "I decided to make a counter-offer using insurance proceeds in the amount of $400,000, which I determined to be consistent with ransoms recently paid by other municipalities," They didn't say no. It's odd there is so little recourse against things like this.

cardamomo 7y ago
You're right, they didn't outright give a hard "no," but the story gets more interesting after New Bedford's counter offer. By attempting to negotiate, the city bought valuable time that they used to harden and restore their systems.
- hartator 7y ago
  Yes, but my understanding of the article is the city was negotiating in good faith. They were willing to pay the $400k. Still a lot of money.
  - logicallee 7y ago
    Any reason laws can't be introduced making it highly illegal to pay a ransom, with serious criminal repercussions (same for government officials doing so)? After all paying a ransom is very literally funding terrorism, in the most direct possible way. "Whosoever knowingly pays a ransom", etc. Of course those seeking a ransom could make their language less and less clear, so it's no longer clearly a ransom. But isn't that still an improvement over the status quo? Maybe I'm missing something, but I don't see why it should be legal to pay.
    - TaExtra 7y ago
      >After all paying a ransom is very literally funding terrorism, in the most direct possible way. Not everything is related to terrorism.
    - parsimo2010 7y ago
      Because you shouldn't criminalize being a victim. I agree that it's probably a bad idea to pay a ransom, since that just supports the success rate and makes ransomware more popular. But do you really think that you should throw a grandparent in jail because paying a ransom is the only way they can get their digital memories back? Are you going to throw all of City Hall in jail because they needed to pay to get their tax records back? The best option is to have good backups and not give into the demands, and that should be encouraged. But criminalizing people because they didn't adhere to the best digital practices is a bad idea.
      - paulryanrogers 7y ago
        For private individuals perhaps it shouldn't be criminal. But for businesses responsible for other people's private information? Yes, I think there is professional negligence if they cannot even restore backups in a timely manner. (Or cannot detect infections before they become so widespread the whole thing falls down.)
        logicallee 7y ago
        The part I would criminalize is the money-wiring part.
        parsimo2010 7y ago
        The action that a business should be punished for is not keeping their customer's information secure. Paying a ransom to recover after they are the victim of a crime should not be what they are punished for. If they were doing everything right but still got hit by ransomware I don't think they should be punished. If you can prove that they wrote their login info on a sticky note and left it in the main lobby then you should punish the bad security practice if it exposes other people's information. Several parts of the world have kidnapping and piracy issues, and you can buy kidnapping/piracy insurance to pay ransoms in case you are the victim of such a crime. I think most people in the world acknowledge that sometimes bad things happen even when you take reasonable precautions, and you shouldn't be punished just because you were the unlucky one. Most security experts agree that no computer system is un-crackable, there are just varying levels of sophistication and access needed to do so. We've even seen that Stuxnet was capable of jumping air gaps. If a business had such good security that their database and backups were air-gapped but still was hit by ransomware do you think that they should still be fined?
  - deleted 7y ago
    [deleted]
- solidasparagus 7y ago
  That didn't make sense to me. What did negotiating actually buy them? I don't see any indication that the attackers paused the attack.
  - rdc12 7y ago
    Little to gain in restoring from backup if the same attack vector is still open. So it was really just a bit of misdirection.
  - blaser-waffle 7y ago
    It bought them time to figure out the root cause, fix it, and check if restoring backups is a viable option. Smart play.
- rolltiide 7y ago
  Restoring a backup they already had? Get outta here
- rebuilder 7y ago
  That sounds like whitewashing to me, to be honest: The city were willing to pay a ransom, and are now trying to make it sound like a shrewd move instead of negotiating with criminals.
deleted 7y ago
[deleted]
VectorLock 7y ago
I wonder if they just didn't check their email or were on vacation that day. That is a HUGE payoff to just ignore.
mkagenius 7y ago
> They didn't say no. Most likely they don't know english and only have pre-written formats to talk to the victim.

jumelles 7y ago

"Restore from backup" really ought to be the default response to ransomware. That will probably never happen though...

TallGuyShort 7y ago
It's very liberating to know you can do this. You can sleep at night knowing your confidential data is safe. From a business continuity standpoint, worst case scenario you get a new machine, restore, and you're back in business. When I trade-in my work laptop for a new one I love the feeling of handing them my old one, picking up my new one, and not looking back. Works just as well for ransomware, other viruses, ruined hard-drives, etc.
- new_guy 7y ago
  But whatever attack vector was used is probably still wide open on your machine even after restoring a backup. Most people will just get infected again straight away.
  - technofiend 7y ago
    Well, maybe. On the other hand forcing someone to update often reveals that updating does not in fact break the world, particularly in linux-land where API updates are supposed to not break userland. Many times users who don't understand the risk incorrectly assume worst case scenario.
- reustle 7y ago
  > It's very liberating to know you can do this. You can sleep at night knowing your confidential data is safe If ransomware was able to infect the machine, then I doubt the data was safe either
  - hobs 7y ago
    Exfiltrating is insanely more difficult especially of "the useful stuff" - and even then you have to sell it or use it to embarrass your target, it's pretty difficult unless you target people ahead of time - easier to just interrupt business.

wmf 7y ago

I don't understand what I'm reading here. Why did they offer to pay $400K to recover ~100 PCs if they had backups? Was it so expensive to restore those PCs? I guess the good news is that we can now sell backups as "anti-ransomware" cybersecurity.

warent 7y ago
Maybe they first had to verify the backups actually existed like they should and were up-to-date enough
- everybodyknows 7y ago
  Indeed. I've worked at a big corp where it was belatedly discovered that backups were broken. A second reason would be to give the attackers reason to pause, in case they somehow had a second bomb to throw, while IT hardened defenses: >Since the attack, the city has installed additional security software and is developing new protocols.
  - AnIdiotOnTheNet 7y ago
    > Indeed. I've worked at a big corp where it was belatedly discovered that backups were broken. Ancient wisdom: If you don't test your backups then you don't have any backups.
- frankharv 7y ago
  My thoughts exactly. I bet they did not have any backups for the Forth of July weekend so anyone doing city business that weekend may have to redo any paperwork or payments. Small price to pay to laugh at these clowns. What exactly are the FBI doing about this. Perhaps if they got off their asses and quit investigating Presidential piss parties they could find this email recipient. How hard is it for the FBI/NSA to trace an email. They just want to cry about going dark instead. If foreign put a CIA hit on the computer terrorist.
fuzz4lyfe 7y ago
>Why did they offer to pay $400K to recover ~100 PCs if they had backups? "Backup is lets says weekly and someone high up the chain really wants the power point he put together on Tuesday." is probably what happened in my experience.

dsfyu404ed 7y ago

Meanwhile Springfield payed with a check but the check bounced. Fitchburg just decided that the affected portions of government didn't actually need doing and nobody noticed. Worcester didn't even know they got cryptowalled because they couldn't see their screens through the pot cloud. Providence didn't pay a dime since the mayor knew a guy who knew a guy. (And before anyone gets offended, this is a joke based on the stereotypical attributes of the other local cities.) Edit: You people have no sense of humor.

quickthrower2 7y ago

Why not make it a crime to pay the ransom demand for cryptolocker attacks? This way the flow of money to these gangsters will stop. If you get hit with a cryptolocker and you have no backup you simply lose that data. Or you can pay the ransom, get your data back and go to jail. While this might seem unfair, it might stop 1000 other attacks because there will be no money in it. It would be for the greater good.

pfisch 7y ago
What happens when a hospital gets hacked? This law could kill people.
slg 7y ago
Why don't we make getting mugged illegal while we are prosecuting victims. If you give your wallet to a mugger, you go to jail. Then no one mugs anyone anymore, right? /s People pay ransoms because they want back whatever is being ransomed. Making it illegal to pay the ransom isn't going to stop that. It will just push everything underground and make it harder to catch the ransomers.
- arcticbull 7y ago
  Mugging involves threat of physical harm. Getting ransomeware'd doesn't, and if cities stopped paying then this kind of attack would stop. It's more like outlawing giving muggers your wallet when you're dressed like Iron Man.
  - dmix 7y ago
    What happens if the ransomwares don't stop? Do you spend a bunch of money on investigations to see if some thousands of small towns or little agency somewhere kept if quiet because the cost otherwise is huge? Up the fines the tax payers will pay when they get caught?
  - olliej 7y ago
    I don’t like this argument: if the ransomware takes out emergency dispatch, hospital systems, etc it gets much closer to what you’re claiming.

deleted 7y ago

[deleted]

H8crilA 7y ago

See also Matt Levine's take on ransoms (scroll down): https://www.bloomberg.com/opinion/articles/2019-08-27/the-libor-change-is-coming https://www.bloomberg.com/opinion/articles/2019-08-27/the-li... And the linked article: https://www.propublica.org/article/the-extortion-economy-how-insurance-companies-are-fueling-a-rise-in-ransomware-attacks https://www.propublica.org/article/the-extortion-economy-how... Tl;dr: perverse incentives, paying some ransoms is in the interest of cyber crime insurers, as it expands the cyber crime insurance market. Also there's more ransomware crime now that the word is out that insurers do pay out ransoms.

zelly 7y ago

The real solution is for them to hire out for their ops. No reason these rag tag city governments should store their stuff on-prem. Until then these okie dokie city bureaucrats will keep getting pwned.

cat199 7y ago
not sure how you are getting that from this one - sounds like this 'rag tag' govenment didn't get powned (completely) and was able to rely on good it practice to recover with minimal impact..
- zelly 7y ago
  They got the victims' hard drives. Regardless of what happened besides that, that's a pwn. What if the police chief had emails admitting to bribes or worse stuff on his pc? What if the hacker had an enemy he wanted framed and arrested?
  - dsfyu404ed 7y ago
    >What if the police chief had emails admitting to bribes or worse stuff Unless he's admitting to doing something that is directly counter to the platform of the ruling party (e.g. aiding ICE, rubber stamping CCW permits, racial profiling, etc) nobody would care. There would be some token outrage but it would mostly be business as usual. This is just how MA works. There's very much a "well that's government, nothing you can do, no sense getting bent out of shape over it" attitude by the majority of the population in MA. For example, the state police overtime fraud is a recurring (every 1-3yr or so) "scandal". The latest instance is basically out of the news already.
snazz 7y ago
I don’t know about that. They’re always going to have endpoints; maybe it’s better that they don’t get sucked into a multimillion-dollar AWS contract they didn’t need? I agree that they need better IT consulting (similar to many small businesses, I would think).

olliej 7y ago

Honestly this success might encourage city governments to make sure their backup systems work better. The counter offer of 400k was presumably the break even point for the cost of losing X days of work (depending on what was lost that could involve manually recovering things like tax payments, billing, tickets, etc)

hello_tyler 7y ago

Why is a backup system so hard for people (especially businesses) to understand????

stanski 7y ago
Only other people need it.

Gustomaximus 7y ago

I wonder if a law that you can't pay ransom money for files should be passed. This should work to take any government and larger companies off the target of these groups as they are likely to obey these laws. Kinda like the we dont negotiate with terrorists approach.

jasonhansel 7y ago
Yep. I'm actually surprised it isn't already illegal, given that the money is likely being used to fund organized crime, rogue states, etc.
- blueadept111 7y ago
  Not only that, there's money to be made by insiders leaking (or even creating) security holes for the hackers to exploit and then getting a kickback.
d357r0y3r 7y ago
Absolutely. It's sort of the same as the "We don't negotiate with hostage-takers" policy. Once you negotiate, it's now a strategy known to work and you'll signal how good of a tactic it is.
- elchin 7y ago
  Exactly. It's going to incentivize more hacking

shameshame 7y ago

The city was extremely lucky the attackers played their hand too early. If servers had been encrypted it could have a lot worse.

auslander 7y ago

No Windows no cry :) Seriously, how can one compromised machine (inevitable) infect all others? Isn't there host firewalls on every machine?

SmellyGeekBoy 7y ago
If you believe that ransomware doesn't exist for macOS and Linux I'm afraid you've been misinformed.
- auslander 7y ago
  Well, there is no SMB v1 on either, main propagation bug. Next propagation tools are SSH bruteforce, subnet scan for vuln apps, Domain admin creds stored locally and sniffing NTLM hashes from network interface. Again only SSH and app vulns are viable in non-windows. I was asked once to do reference design of Windows on AWS. After I learned how many ports has to be open for every machine and all of them had to be in same network as Domain Controller, I quit my job. > If you believe that ransomware doesn't exist for macOS Example? I have not seen any.
  - lgl 7y ago
    Absence of evidence != Evidence of absence
    - auslander 7y ago
      but is a pretty good guideline :) If Windows got ransomed every time and MacOS is never, that is interesting, right? Windows holds perceived monopoly on company-wide identity control. There are Mac solutions: https://www.jamf.com/products/jamf-pro/deployment/ https://www.jamf.com/products/jamf-pro/deployment/
    - swebs 7y ago
      unfalsifiability != evidence https://en.wikipedia.org/wiki/Russell%27s_teapot https://en.wikipedia.org/wiki/Russell%27s_teapot

gesman 7y ago

Key: backup Do you ... backup?

rsync 7y ago

ZFS snapshots are immutable (read-only) for normal users. So, if you had your data stored on a cloud storage platform that created and maintained ZFS snapshots, Mallory could gain all of the credentials and still not be able to touch your daily/weekly/monthly snapshots. Now, if only there were a cloud backup platform that included zfs snapshots ...

masklinn 7y ago
> ZFS snapshots are immutable (read-only) for normal users. Unless you've given them "destroy" that is. > Now, if only there were a cloud backup platform that included zfs snapshots ... I see what you did there...
- nikolay 7y ago
  Yup, he's the guy behind https://www.rsync.net/ https://www.rsync.net/
icebraining 7y ago
What if someone uses the credentials to file a support request in that platform asking the snapshots to be deleted?
- jdnenej 7y ago
  Hopefully anyone with the ability to delete them would decline such a request.
  - icebraining 7y ago
    So the platform would keep data against the wishes of the client? That doesn't sound good.
    - TomMarius 7y ago
      More like it would include a more elaborate, person to person process.

peter_retief 7y ago

Restore from backup isn't as easy as it sounds, well done to a disaster recovery plan that actually works

auslander 7y ago

Great read: https://www.wired.com/story/notpetya-cyberattack-ukraine-russia-code-crashed-the-world/ https://www.wired.com/story/notpetya-cyberattack-ukraine-rus...

mtgx 7y ago

Cities need to stop wanting to become "smart" with little thought put into it or by using vendors that are friends of friends.

tastroder 7y ago

Could somebody contextualize how targeted this particular attack really was? According to random Google result #1: https://www.2-spyware.com/remove-ryuk-ransomware.html#qm-h2-4 https://www.2-spyware.com/remove-ryuk-ransomware.html#qm-h2-... the specific malware distribution is unclear but likely involves email attachments and/or vulnerable and exposed RDP. While reporting on ransomware cases often sounds like targeted APTs, more often than not the details in these stories read like "we didn't bother to pay enough admins to actually patch and secure our systems" and "we didn't train our users not to click on every random attachment".

pilif 7y ago

Do I read this right? They offered to pay $400K even though they had backups to restore their data from? Does this mean that the restore operation cost more than 400K or was this an incredible sample of laziness? They'd rather pay the ransom to a criminal than invest the time it takes to restore backups?

bluescrn 7y ago
Even with good backups and a prompt recovery, you could be looking at losing a day or so of data/work across the affected organisation. If the system had been backing up encrypted files for days/weeks then there’s a much higher cost to restoring from older backups
- SahAssar 7y ago
  Making sure the systems are actually clean after getting the files decrypted costs too though. The comparison would be (due diligence to make sure systems are clean + ransom) vs. (restoring backups + lost work).

iandinwoodie 7y ago

So they offered $400,000 for restoring what was most likely a week of lost work for 158 employees. That works out to $2,531.65 per employee for that week. Is the average salary of the compromised employee $131,645.57 ($2,531.65*52)? It sounds like the town was just willing to throw a lump of cash at the attacker based on what other victims had paid with no regard to what the lost work was actually worth. I know the attacker did not accept and it allowed them to strengthen their security and etc., but it bothers me that the attacker could have just gotten a $400,000 payout. It’s almost as if the moral of the story is “even if a town government had backup system in place, you can still pull in a few years of income with a ransomware attack!”

duiker101 7y ago
I guess accepting it would set the precedent that they are easy to lowball? Also, it might even be a lower sum for the city, if they need to redo the work, the employees will already know more or less what to do and it might be faster.
Zenbit_UX 7y ago
You would lose more than a week of work if your entire team is no longer doing this week's work to make up for last week's. Let's say they instead of doing this week's work, they do last week's, next week they're still a week behind and you have to start paying 100+ government employees overtime in order to regain the catch up on the remaining week. This could take a while to catch up and be incredibly expensive. The $400,000 wasn't taxpayer money it was their insurance companies offer to make this go away.
- dsfyu404ed 7y ago
  Local government work is mostly cyclical and the "customer" is captive. It's not the end of the world if they bill someone a day late or have to re-add the new librarian to the payroll system. It's not like people can up and switch to a new government because they don't like the quality of service. Based on having grown up in the region and having, um, "connections" to state and local government I think it is highly likely that their desire to not throw out work was based around reasoning around how their public image would be affected if a batch of fines/taxes/fees/bills had to be waived. Cutting people (collectively, waiving something on a case by case basis is fine) a break because the government screwed up is kind of a non-starter because of the possibility of setting a precedent.

ashelmire 7y ago

Paying hundreds of thousands to restore 158 backed up machines would be absurd. Surely a week of work for those 158 employees isn't worth that. They've gotta put a financial analyst on these issues, and not leave it up to panicking executives. And look at how greedy the attacker was. Missed out on a 400k payday.

raxxorrax 7y ago
I would like to see your calculation for that. My current company is much smaller in size and had 2 ransom attacks that I saw happening (in ~2 years). We use nimble as a backup and it is quite expensive (hundreds of thousands actually, depending on infrastructure) but worked flawlessly to restore all our systems in a short time. You will loose a few hours of work though. Attack vectors are the usual: mail with infected attachment and users with too many rights on documents. The attackers even know names of people working at the company and use correct mail signatures (not actual mail signatures, just the colorful stuff you put at the end of every business mail). If production stopped for a week, the contractual penalties alone would probably eat up sums like that.

CryptoPunk 7y ago

Cryptocurrency is hardening IT infrastructure.

mxuribe 7y ago

Kudos to the city of New Bedford!

aitchnyu 7y ago

> Because the attack happened at night, most of the city's systems were turned off and the ransomware was unable to spread. I used to complain Indian banks used to allow fund transfers only during working days and office hours. Now I see why it was useful.

elchin 7y ago

Why isn't it mandatory for government orgs to use cloud?

Shivetya 7y ago
I do not agree they should use "the cloud" but what is all too common is local, city, state, and even federal governments, love to enforce regulations and laws upon others they do not obey themselves. the first rule of laws and regulations should be, government must adhere to the letter if they are to be enforced upon others seeking penalties if compliance is not met. it is really difficult to hold government agencies responsible, it doesn't even have to be computer security, just look at the number of cities in the US who violate lead levels in water. even better, in the US, a lot of what happens can be protected from suits by sovereign immunity. So keep it simple, what is required of private organizations and people is the minimum standard that a government agency must meet
SahAssar 7y ago
Why would it be and how would you even define "the cloud"?

msmerberry 7y ago

It sounds like they played every card right, and that's probably not by accident. I'd be fascinated to see a case study made out of this - especially with a breakdown of how they performed against their CSIRP - because there's no way they didn't tabletop that plan at some point, and it paid off in the end. Sort of like insurance, eh?

avivdeg 7y ago

NetApp has a solution for all of that: Cloud Volumes ONTAP is a virtualized storage platform running on AWS, Azure, and Google,it consumes native cloud resources and it provides NFS/CIFS/iSCSI. Cloud Volumes ONTAP has the best snapshots out there, immutable and without any resource penalty. you can take any size snapshots (or restore) in seconds. you can also create clones out of these snapshots, so you can check if that data been affected or not, again in seconds. Adding to that Cloud Manager's Ransomeware protection that blocks known Ransomware files. In short- this is the best solution out there for any hybrid/ cloud and it can actually be cheaper than free, if you have enough capacity, due to all of its storage efficiencies like dedup, compressions and compaction, with auto-tiering of unused blocks to the checper object storage,.