9 ms·

Baltimore Blew Off Ransomware Demand Only to Find Data Had Never Been Backed Up

190 points by miles 7y ago

duxup 7y ago
For a while I worked with some products that were popular with municipalities, and school districts in a previous career. The scale of under funding, and incompetent leadership in those roles is staggering, and sadly not surprising to me. Internal politics are always an issue of course, especially at schools where a handful of luddite teachers / administrators can kill good idea, but that's also an issue of IT leadership too. I was glad to move away from those products.
- covercash 7y ago
  I briefly worked for a school district that had 8000 Macs to support. They fired the entire technical team (for reasons unknown) except for me and then proceeded to hire people who had never used a Mac in their life. The woman in charge of the department was like Lorraine Bracco’s character in Hackers - extremely overpaid, could barely figure out how to work a computer, and used God as her password. I’m still disgusted when I think about how corrupt and incompetent that district was and likely still is. A few of the fired guys wound up with a nice settlement, I’m sure the webcam recordings of the superintendent exchanging teaching positions for “favors” probably worked to their advantage.
  - listenallyall 7y ago
    Reasons unknown? For starters, they were conspiring to covertly spy on their employer.
    - covercash 7y ago
      Nope, the district had no idea they were doing that until after the firing. I did hear the actual reason was because they were considering joining a union, but I didn’t stick around long enough to dig any deeper.
      - listenallyall 7y ago
        Conspiring criminals tend to do lots of conspiracy-ish and criminal-ly things. If they secretly installed webcams, they likely also were reading users' email and viewing confidential documents like performance reviews, budgets, salaries and student test scores. Their firing was likely highly justified. Since you seem to know all kinds of facts that discredit the school district, the superintendent, and the IT supervisor, yet "didn't stick around" to absorb info that might look bad for the group of employees, makes you a narrator who is difficult to believe is being objective and non-biased.
Ancalagon 7y ago
The irony in this is absolutely hilarious. I remember we were all praising the city for having such forethought and for doing their due diligence (unlike most of these cases) and keeping backups of their data.
- coldcode 7y ago
  I wonder how the IT director was hired and who monitored what the department was doing. I worked at a place that hired a MS system support person but no on one knew anything about it and he seemed to know more than they did. He was fired a couple months later as he actually knew nothing and spent all his time making support calls trying to get someone to tell him what to do. He was on the phone for hours every day. Then when he was fired no one changed passwords so he logged into the DNS server and randomized all the IPs, leaving the whole company without networking. Hopefully not the same person...
  - fortran77 7y ago
    Twice I've been at companies where we lost some storage due to a hardware failure and it turns out the backups were missing or didn't work. And both times, nobody was fired.
    - JoeAltmaier 7y ago
      In fact, if you haven't tried to recover from a backup, it probably doesn't work. Like most recovery/data integrity hardware/software, if it isn't being (regularly) tested, it probably is misconfigured, not running or broken. Like all software. I've seen many PC-raid hardware solutions that simply didn't work. A drive fails, and the machine becomes unusable, fail to recover when a good drive is inserted, or crashes/hangs and the data is lost. Yet people keep buying raid solutions for PCs. I recommend: pull a drive from your raid hardware and see what happens. If you're afraid to do that, then you need a different solution.
      - bluGill 7y ago
        I used to work at a RAID hardware company (since bought by SUN, now Oracle though I have no idea if that model is still sold). Salesmen were told to demonstrate the system by pulling a random drive from the production disk and put it on top. 15 minutes latter someone from support showed up with a new disk (not IT, support), the salesmen made sure they were still next to the machine to show off that support knew about the failure and could replace it quickly. If you don't have the same confidence in your systems you need to fix that.
blakesterz 7y ago
I remember hearing lots of back and forth on whether they actually used EternalBlue or not, I'm still not sure if that's settled or not, is it? "Given the fact that $6 million has already been pulled from parks and public utilities funds to "harden" city systems, the $76,000 demand now seems like a bargain." That doesn't seem fair at all. They'd still have to harden everything and it would still likely cost millions. From the looks of it they'd at least have their stuff back, probably, but they'd still need to put all the same time/money/work in, wouldn't they?
- joncrane 7y ago
  First of all, it's a terrible article, it's full of typos; why aren't we linking to the source article on ArsTecnica? Secondly, Later on in the article is the more relevant >The city figures it will cost $18 million to recover from a rejected $76,000 ransom demand. I guess if you're going to play chicken with extortionists, you might want to make sure your backup plans at least meet min spec. Either way, it's a terrible article and I'm on the fence about flagging it.
  - dmix 7y ago
    It's RobinHood ransomware, I don't think it was EternalBlue either... https://www.bleepingcomputer.com/news/security/a-closer-look-at-the-robbinhood-ransomware/ https://www.bleepingcomputer.com/news/security/a-closer-look...
  - totaldude87 7y ago
    on the contrary, i kinda liked this sarcastic writing.. >>The person in charge of the city's systems was Frank Johnson, who went on leave (presumably permanently) after a post-attack audit found the IT director hadn't done much IT directing.
- shantly 7y ago
  I wonder if, by the time we factor in all the costs of putting everything on a computer and online, securely, it's going to look like it would have been a pretty alright idea to leave most of it the way it had been in the distant past of 30ish years ago.
fredley 7y ago
If you don't test your backups, they're not backups. But I guess in this case if you don't have backups at all, you also don't have backups... 1 is 0, 2 is 1, etc...
- LinuxBender 7y ago
  I would take that a step further and say, "If anything, including automation, can tamper with your backups, they are not backups" Several companies have gone out of business this year due to getting their servers wiped by an attacker. Their backups were on live servers as well.
- mdip 7y ago
  I used to manage a set of tape backup robots (huge square tapes, I want to say 25 glorious GB a piece, and some Sony DAT devices @50-75GB a piece) that backed up sister data centers over a few DS-3s back in the early 2000s. The hardware/servers involved were around a 3/4 million. I only provide that context because I've been as far away from that experience as I can possibly put myself and I can only hope it's gotten a lot better. Prior to me and during the time that I managed backups, the statement "if you don't have backups at all, you don't have backups" was true for the entire backup set probably about once a week[0]. Those miserable robotic devices would crush tapes, the drives would eat tapes or otherwise fail, the backup servers would be over-whelmed (despite being nearly the largest hosts we had), the software would crash (Backup Exec), the server would just be ... skipped? And the tapes had a lifetime that was 1/10th what was advertised (and 1/2 what we had planned on[1]). Our process included a monthly restore test of a few critical pieces of infrastructure and a rotating list of other hosts. I think the one time that we passed might have been worthy of opening a bottle of wine. Most of the time we could get enough data back that the restore wouldn't result in more than a rounding-error financial loss, but for the most part our server team spent a lot of time praying. That system gave me nightmares for years after I left that position. [0] It was so bad that our initial plan was to backup DC A using DC B and vice versa, but we opted to backup both from both data-centers (one of our requirements was off-site backups). [1] I think I read the headline to an article a month ago about why "tape is still king" in backups. I couldn't bring myself to click-in. I assume things have gotten better, but if they're still selling this product the way they did back then, I'd want real-world numbers on MTBF and the like.
Leader2light 7y ago
Baltimore. Trump was right all along. That is exatly why you don't hear the media giving him more crap about that old black dudes death at 68. They knew he didn't do crap for that place for decades. Good riddance. Maybe now they can get in someone who will. That is a dream. MLK dream.
jiveturkey 7y ago
It makes perfect sense! The same kind of people that would say no to a pocket change ransom, are the same kind of people that would use taxpayer money to not just do a poor job, but to not do it at all! The kind of people whose shit don't stink.
- bluGill 7y ago
  Never pay a ransom: it just encourages the bad guys to try harder. Soon they will figure out how to corrupt your backups as well (if there is anyway to do that - now we are in an arms race). Better to write off the loss now and ensure the bad guys don't continue to think of new attacks. There is one exception: if the ransom is paid in such a way that the FBI (or equivalent) can track where the money goes and thus arrest the criminals.
  - dangerboysteve 7y ago
    Well when the FYI recommends paying what options do you have? https://www.nytimes.com/2019/08/14/opinion/ransomware.html https://www.nytimes.com/2019/08/14/opinion/ransomware.html
whack 7y ago
You never know who's swimming naked until the tide goes out. Would be great if every organization conducted targeted attacks in order to probe the reliability of their assumed safeguards. If your system hasn't been tested by someone who has a real incentive to break it, you have no idea if it is really as secure as you think it is.
- dangerboysteve 7y ago
  We do this. Every place I have worked at I've had the senior admin explain and show the backup plans for servers and desktops and demonstrate various restore requests. I have caught a few with their pants down. Another important thing is fine granular permissions on network shares.
  - wil421 7y ago
    We have our own pen testing team in addition to the 3rd parties who also do it. There’s a big DR test this weekend for our 2 largest data centers as well to make sure backups and fail overs happen. Plus multiple networks for different products that’s completely different from the regular employee network. You even have to sign in if you plug into our Ethernet.
- bduerst 7y ago
  Penetration tests are a huge deal right now. Many companies outsource it to security agencies who then perform the pen tests and report results.
- Allower 7y ago
  Seems like an actually useful thing for our so called 'defense' agencies to do, so im sure it will never happen.
DoofusOfDeath 7y ago
I have to wonder if, in macro economic terms, we'd be better off spending whatever is necessary to successfully investigate and prosecute scammers/telemarketers/ransomers.
- bluGill 7y ago
  We are best off not paying them regardless: if there is no payout the bad actors will give up the attack. If you are in a group that pays out target them. It doesn't pay to target someone who has good backups (though it might not be worth checking if infecting everyone is easy)
  - crankylinuxuser 7y ago
    >if there is no payout the bad actors will give up the attack I'd argue that isn't true. Ransomware is a form of terror in the form of "losing everything". And that threat of impending loss combined with 'backups are hard wahhhh' and the fact that no Corp I know backs up user machines... As a numbers game, it pays. Well. And even if it didn't, some still hack for the "lulz".
    - wpietri 7y ago
      Definitely. Look at all the people pursuing "make money fast" scams. Some people will happily do anything they think might make them a buck. Not paying out will help, of course, but it doesn't eliminate the problem.
      - bluGill 7y ago
        When there is no money in it nobody will be developing the exploits that make it easy. There is a large difference between buying a recent vulnerability and exploiting it vs finding a vulnerability.
    - davinic 7y ago
      >>no Corp I know backs up user machines Most of the orgs I've worked at (for at least the last decade) map the local profile to a network share, which is definitely backed up. The only things not backed up would be saved outside that folder structure, but the organizations have always been clear about that.
mdip 7y ago
"That can't be real?" Really? I'll never forget the description given to me by a close friend who had left a "Government IT" job for the private sector: (1) You have standards and practices like everywhere, but forget one and you're explaining yourself to a judge instead of a boss and (2) there's never enough money for doing all of the things required to meet regulations let alone make things better which is why you'll see silly things you haven't seen on banking websites in a decade still prominent as "security features" on state treasury websites. I found the story about "important data on a desktop drive", and the shock it caused, surprising. Maybe my past life (a decade ago in infrastructure) was unique, but I specifically recall an incident where I was called down to make an old IBM NetVista (mind you, Lenovo had owned that line for a while at this point) boot up[0]. I noticed some numbers on a printed label and a boot error about the CMOS battery, realized it was drive telemetry, realized that nobody who was looking at the problem had ever heard of plugging in those values (or probably had touched an IDE controller -- server guys -- and it was ancient technology). The rest of the story I might not have completely correct as parts of it are assembled third-hand, but this desktop was located in our data center, hooked up to a modem (2400bps) and it handled submitting charges to another carrier to the tune of "a few layoffs" for every week it wasn't functional. How does this happen? Well, the company went bankrupt and emerged, then was purchased by another company. During that time, a large part of our operations was moved from one state to another, hardware and all (but mostly not the people). This predates all of that, of course. At some point, a NetVista was put in place to test setting up an automated process for billing this carrier -- something carrier imposed (must have been one of the big guys). The developer who set up the test system was successful ... on his final week of work before being laid off. A few months later, the carrier continued working the migration plan and switched things over, and after a short delay, accounting rang the alarm bells. A busy developer stepped in, found the offending system was connected to test and re-configured it to point to prod. Everyone went on their day. And hey, when desktop migrated everyone to Windows XP, they put UPSes at every desk so it literally ran in a cubicle until the Data Center migration (where it failed the first time and the label was printed). Rather than figuring out what, on earth, it would take to fix it, they put it on a shelf and plugged it into the UPS in the rack until I was called several years later. [0] I was one of two people who were called when everything else was tried. This sort of incident happened to me in very similar ways at least 4 times (once with an old Thinkpad Laptop).
swebs 7y ago
How do we increase IT competence in the public sector? There's no reason important servers should be running on Windows machines with "backups" consisting of making a copy of a file on the same hard drive as the original.
- AnIdiotOnTheNet 7y ago
  First you'd have to solve politics. Good luck with that.
- magashna 7y ago
  You can't do anything but let it publicly fail. These incompetencies are usually the result of the top not believing that the cost is worth it. Also a well run IT department often gets "nothing's ever broken, what do they actually do?" so you get rewarded for doing break/fix.
  - toomuchtodo 7y ago
    I am an infosec professional who pitches utilities and public government. This is spot on. The second call back is always more expensive for them, but sometimes someone needs to touch the stove themselves to see it’s hot. If you genuinely can’t afford it, I’ll help for free, but there’s only so much time in the day and I won’t eat the cost of stupid or politics.
    - ben_jones 7y ago
      Is it not also because the government contracting system is FUBAR? I.e. only certain kinds of companies can contract on the big projects, you need certain clearances for certain kinds of works, and the pay gap between public and private sector SWE & security? Also aren't their some pretty tough mandates like x% of companies must be minority owned etc. I imagine it allows those companies to charge extortionist rates because they hit the relevant quotas and nobody else does.
      - iguy 7y ago
rednerrus 7y ago
This headline gives me anxiety.
duckqlz 7y ago
TLDR; if your company / city has millions invested in a cyber infrastructure get cyber insurance. :facepalm: Also $76,000 is oddly low... like low enough to be the price of a well established firm to do a pen test on your network...
- davinic 7y ago
  True, but managers also usually fail to account for the time and cost of remediation after a pentest. Had the city engaged in the pentest, would there have been an appetite to spend 10-20x that amount on remediation?
  - NullPrefix 7y ago
    Paying the ransom doesn't cover the need to spend money on fixing the problems. Having a ransom payer mark on the organization raises the security requirement, doesn't it?
- magashna 7y ago
  For personal ransomware, the requests are average around $250, lower than you'd think.
rsync 7y ago
I guess a failure to do backups of any kind is beyond any solution we could offer, but if one could take even a single step forward beyond nothing, a read-only (immutable) offsite destination is well worth your consideration.[1] Most end users looking for "ransomware protection" probably just want to drag and drop some files, like Dropbox, which is why a simple SFTP (filezilla) solution is nice, but of course you could point any old thing[2] if you were more sophisticated ... [1] https://www.rsync.net/products/ransomware.html https://www.rsync.net/products/ransomware.html [2] borg, restic, rclone, git-annex ... rsync ...
- korzun 7y ago
  I subscribed to your service a couple of years back, and you were leaking other customers' home directories. The whole thing was just a shared, dedicated server with shell access. Super sketchy. Back then, you also stated that you offer super-secure access and all of that jazz. Funny that you are double downing on all the compliance bullshit and ransomware buzz words now. The compliance statements are pretty questionable and make no sense for the services you are offering. Surprised you don't provide 'military-grade' encryption. Just throw all of that in there keep the thing. Buyers beware. Do your homework.
- vultour 7y ago
  Surprised someone would use or recommend FileZilla after their numerous adware issues.
  - rsync 7y ago
    I'm just thinking of simple SFTP clients ... WinSCP ? psftp.exe ? Whatever works for you.
deleted 7y ago
[deleted]
jrochkind1 7y ago
The techdirt article author suggest they should have just paid the ransom. Even knowing that paying the ransom is no _guarantee_ you actually recover, and that you have to then harden your system anyway after that (which you should have been doing all along)... I tend to agree. I am a Baltimore resident, and I've also had that opinion since the day after the ransomware attack was announced -- just pay the ransom. (In part because from what I experience of Baltimore City government, i was pretty sure the backups and recovery/continuity plans were going to be basically nonexistent). But on previous HN threads, it was a very unpopular opinion, in the articles right after the attacks were announced, very few in the comments threads seemed to think it was acceptable or a good idea to pay the ransom. I'm curious if that remains true?
- tempestn 7y ago
  Just my personal opinion, but I figure the logical response would be to audit 1) the state of their backups, and 2) the source of the malware. After confirming that all necessary data had indeed been backed up and could be restored, and after blocking the vector used to deliver the malware, they could refuse to pay the ransom and restore from backups. Then they could go about further hardening. If it was discovered that the backups were faulty (as in this case), or if it were not possible to discover how the infection happened prior to the ransom deadline, I would have advised paying the ransom, since 76k is a relatively small amount to the city of Baltimore. (And then of course continuing to investigate and harden to avoid having to do so again.)
- nickpeterson 7y ago
  People have principles when it isn't their money. The best route would be to pay it, then divert the money and fix the issues. Think of the 76k as cheap penetration testing.
  - itake 7y ago
    The problem with this thinking is: 1/ You have to trust the criminals are "good" people and actually took the time to write code to decrypt the data. 2/ Paying the ransom paints you an easy target for the next exploit.
totaldude87 7y ago
>>The person in charge of the city's systems was Frank Johnson, who went on leave (presumably permanently) after a post-attack audit found the IT director hadn't done much IT directing. love this writing man :) :) Dont know why would a city's chief digital officer go backup free?!!! even at worst , backblaze would have helped :|
ping_pong 7y ago
To be fair, the CIO was only on the job for 1.5 years when the ransomware hit. It's not known whether or not he had the budget for reorganizing all of the data, etc. Maybe it was on his list of priorities, but it didn't have the funding, etc, and it was pushed off until the next year. I'm not absolving him of this because backups are a first order object when you're dealing with IT, but it could be part of the reason why it wasn't implemented yet. This is the reality of working in a bureaucracy like city government.
linuxftw 7y ago
Baltimore (like many cities) is run by crooks and morons. They are a shining example of ineptitude and failure in all things.

190 points by miles 7y ago

For a while I worked with some products that were popular with municipalities, and school districts in a previous career. The scale of under funding, and incompetent leadership in those roles is staggering, and sadly not surprising to me. Internal politics are always an issue of course, especially at schools where a handful of luddite teachers / administrators can kill good idea, but that's also an issue of IT leadership too. I was glad to move away from those products.

covercash 7y ago
I briefly worked for a school district that had 8000 Macs to support. They fired the entire technical team (for reasons unknown) except for me and then proceeded to hire people who had never used a Mac in their life. The woman in charge of the department was like Lorraine Bracco’s character in Hackers - extremely overpaid, could barely figure out how to work a computer, and used God as her password. I’m still disgusted when I think about how corrupt and incompetent that district was and likely still is. A few of the fired guys wound up with a nice settlement, I’m sure the webcam recordings of the superintendent exchanging teaching positions for “favors” probably worked to their advantage.
- listenallyall 7y ago
  Reasons unknown? For starters, they were conspiring to covertly spy on their employer.
  - covercash 7y ago
    Nope, the district had no idea they were doing that until after the firing. I did hear the actual reason was because they were considering joining a union, but I didn’t stick around long enough to dig any deeper.
    - listenallyall 7y ago
      Conspiring criminals tend to do lots of conspiracy-ish and criminal-ly things. If they secretly installed webcams, they likely also were reading users' email and viewing confidential documents like performance reviews, budgets, salaries and student test scores. Their firing was likely highly justified. Since you seem to know all kinds of facts that discredit the school district, the superintendent, and the IT supervisor, yet "didn't stick around" to absorb info that might look bad for the group of employees, makes you a narrator who is difficult to believe is being objective and non-biased.

Ancalagon 7y ago

The irony in this is absolutely hilarious. I remember we were all praising the city for having such forethought and for doing their due diligence (unlike most of these cases) and keeping backups of their data.

coldcode 7y ago
I wonder how the IT director was hired and who monitored what the department was doing. I worked at a place that hired a MS system support person but no on one knew anything about it and he seemed to know more than they did. He was fired a couple months later as he actually knew nothing and spent all his time making support calls trying to get someone to tell him what to do. He was on the phone for hours every day. Then when he was fired no one changed passwords so he logged into the DNS server and randomized all the IPs, leaving the whole company without networking. Hopefully not the same person...
- fortran77 7y ago
  Twice I've been at companies where we lost some storage due to a hardware failure and it turns out the backups were missing or didn't work. And both times, nobody was fired.
  - JoeAltmaier 7y ago
    In fact, if you haven't tried to recover from a backup, it probably doesn't work. Like most recovery/data integrity hardware/software, if it isn't being (regularly) tested, it probably is misconfigured, not running or broken. Like all software. I've seen many PC-raid hardware solutions that simply didn't work. A drive fails, and the machine becomes unusable, fail to recover when a good drive is inserted, or crashes/hangs and the data is lost. Yet people keep buying raid solutions for PCs. I recommend: pull a drive from your raid hardware and see what happens. If you're afraid to do that, then you need a different solution.
    - bluGill 7y ago
      I used to work at a RAID hardware company (since bought by SUN, now Oracle though I have no idea if that model is still sold). Salesmen were told to demonstrate the system by pulling a random drive from the production disk and put it on top. 15 minutes latter someone from support showed up with a new disk (not IT, support), the salesmen made sure they were still next to the machine to show off that support knew about the failure and could replace it quickly. If you don't have the same confidence in your systems you need to fix that.

blakesterz 7y ago

I remember hearing lots of back and forth on whether they actually used EternalBlue or not, I'm still not sure if that's settled or not, is it? "Given the fact that $6 million has already been pulled from parks and public utilities funds to "harden" city systems, the $76,000 demand now seems like a bargain." That doesn't seem fair at all. They'd still have to harden everything and it would still likely cost millions. From the looks of it they'd at least have their stuff back, probably, but they'd still need to put all the same time/money/work in, wouldn't they?

joncrane 7y ago
First of all, it's a terrible article, it's full of typos; why aren't we linking to the source article on ArsTecnica? Secondly, Later on in the article is the more relevant >The city figures it will cost $18 million to recover from a rejected $76,000 ransom demand. I guess if you're going to play chicken with extortionists, you might want to make sure your backup plans at least meet min spec. Either way, it's a terrible article and I'm on the fence about flagging it.
- dmix 7y ago
  It's RobinHood ransomware, I don't think it was EternalBlue either... https://www.bleepingcomputer.com/news/security/a-closer-look-at-the-robbinhood-ransomware/ https://www.bleepingcomputer.com/news/security/a-closer-look...
- totaldude87 7y ago
  on the contrary, i kinda liked this sarcastic writing.. >>The person in charge of the city's systems was Frank Johnson, who went on leave (presumably permanently) after a post-attack audit found the IT director hadn't done much IT directing.
shantly 7y ago
I wonder if, by the time we factor in all the costs of putting everything on a computer and online, securely, it's going to look like it would have been a pretty alright idea to leave most of it the way it had been in the distant past of 30ish years ago.

fredley 7y ago

If you don't test your backups, they're not backups. But I guess in this case if you don't have backups at all, you also don't have backups... 1 is 0, 2 is 1, etc...

LinuxBender 7y ago
I would take that a step further and say, "If anything, including automation, can tamper with your backups, they are not backups" Several companies have gone out of business this year due to getting their servers wiped by an attacker. Their backups were on live servers as well.
mdip 7y ago
I used to manage a set of tape backup robots (huge square tapes, I want to say 25 glorious GB a piece, and some Sony DAT devices @50-75GB a piece) that backed up sister data centers over a few DS-3s back in the early 2000s. The hardware/servers involved were around a 3/4 million. I only provide that context because I've been as far away from that experience as I can possibly put myself and I can only hope it's gotten a lot better. Prior to me and during the time that I managed backups, the statement "if you don't have backups at all, you don't have backups" was true for the entire backup set probably about once a week[0]. Those miserable robotic devices would crush tapes, the drives would eat tapes or otherwise fail, the backup servers would be over-whelmed (despite being nearly the largest hosts we had), the software would crash (Backup Exec), the server would just be ... skipped? And the tapes had a lifetime that was 1/10th what was advertised (and 1/2 what we had planned on[1]). Our process included a monthly restore test of a few critical pieces of infrastructure and a rotating list of other hosts. I think the one time that we passed might have been worthy of opening a bottle of wine. Most of the time we could get enough data back that the restore wouldn't result in more than a rounding-error financial loss, but for the most part our server team spent a lot of time praying. That system gave me nightmares for years after I left that position. [0] It was so bad that our initial plan was to backup DC A using DC B and vice versa, but we opted to backup both from both data-centers (one of our requirements was off-site backups). [1] I think I read the headline to an article a month ago about why "tape is still king" in backups. I couldn't bring myself to click-in. I assume things have gotten better, but if they're still selling this product the way they did back then, I'd want real-world numbers on MTBF and the like.

Leader2light 7y ago

Baltimore. Trump was right all along. That is exatly why you don't hear the media giving him more crap about that old black dudes death at 68. They knew he didn't do crap for that place for decades. Good riddance. Maybe now they can get in someone who will. That is a dream. MLK dream.

jiveturkey 7y ago

It makes perfect sense! The same kind of people that would say no to a pocket change ransom, are the same kind of people that would use taxpayer money to not just do a poor job, but to not do it at all! The kind of people whose shit don't stink.

bluGill 7y ago
Never pay a ransom: it just encourages the bad guys to try harder. Soon they will figure out how to corrupt your backups as well (if there is anyway to do that - now we are in an arms race). Better to write off the loss now and ensure the bad guys don't continue to think of new attacks. There is one exception: if the ransom is paid in such a way that the FBI (or equivalent) can track where the money goes and thus arrest the criminals.
- dangerboysteve 7y ago
  Well when the FYI recommends paying what options do you have? https://www.nytimes.com/2019/08/14/opinion/ransomware.html https://www.nytimes.com/2019/08/14/opinion/ransomware.html

whack 7y ago

You never know who's swimming naked until the tide goes out. Would be great if every organization conducted targeted attacks in order to probe the reliability of their assumed safeguards. If your system hasn't been tested by someone who has a real incentive to break it, you have no idea if it is really as secure as you think it is.

dangerboysteve 7y ago
We do this. Every place I have worked at I've had the senior admin explain and show the backup plans for servers and desktops and demonstrate various restore requests. I have caught a few with their pants down. Another important thing is fine granular permissions on network shares.
- wil421 7y ago
  We have our own pen testing team in addition to the 3rd parties who also do it. There’s a big DR test this weekend for our 2 largest data centers as well to make sure backups and fail overs happen. Plus multiple networks for different products that’s completely different from the regular employee network. You even have to sign in if you plug into our Ethernet.
bduerst 7y ago
Penetration tests are a huge deal right now. Many companies outsource it to security agencies who then perform the pen tests and report results.
Allower 7y ago
Seems like an actually useful thing for our so called 'defense' agencies to do, so im sure it will never happen.

DoofusOfDeath 7y ago

I have to wonder if, in macro economic terms, we'd be better off spending whatever is necessary to successfully investigate and prosecute scammers/telemarketers/ransomers.

bluGill 7y ago
We are best off not paying them regardless: if there is no payout the bad actors will give up the attack. If you are in a group that pays out target them. It doesn't pay to target someone who has good backups (though it might not be worth checking if infecting everyone is easy)
- crankylinuxuser 7y ago
  >if there is no payout the bad actors will give up the attack I'd argue that isn't true. Ransomware is a form of terror in the form of "losing everything". And that threat of impending loss combined with 'backups are hard wahhhh' and the fact that no Corp I know backs up user machines... As a numbers game, it pays. Well. And even if it didn't, some still hack for the "lulz".
  - wpietri 7y ago
    Definitely. Look at all the people pursuing "make money fast" scams. Some people will happily do anything they think might make them a buck. Not paying out will help, of course, but it doesn't eliminate the problem.
    - bluGill 7y ago
      When there is no money in it nobody will be developing the exploits that make it easy. There is a large difference between buying a recent vulnerability and exploiting it vs finding a vulnerability.
  - davinic 7y ago
    >>no Corp I know backs up user machines Most of the orgs I've worked at (for at least the last decade) map the local profile to a network share, which is definitely backed up. The only things not backed up would be saved outside that folder structure, but the organizations have always been clear about that.

mdip 7y ago

"That can't be real?" Really? I'll never forget the description given to me by a close friend who had left a "Government IT" job for the private sector: (1) You have standards and practices like everywhere, but forget one and you're explaining yourself to a judge instead of a boss and (2) there's never enough money for doing all of the things required to meet regulations let alone make things better which is why you'll see silly things you haven't seen on banking websites in a decade still prominent as "security features" on state treasury websites. I found the story about "important data on a desktop drive", and the shock it caused, surprising. Maybe my past life (a decade ago in infrastructure) was unique, but I specifically recall an incident where I was called down to make an old IBM NetVista (mind you, Lenovo had owned that line for a while at this point) boot up[0]. I noticed some numbers on a printed label and a boot error about the CMOS battery, realized it was drive telemetry, realized that nobody who was looking at the problem had ever heard of plugging in those values (or probably had touched an IDE controller -- server guys -- and it was ancient technology). The rest of the story I might not have completely correct as parts of it are assembled third-hand, but this desktop was located in our data center, hooked up to a modem (2400bps) and it handled submitting charges to another carrier to the tune of "a few layoffs" for every week it wasn't functional. How does this happen? Well, the company went bankrupt and emerged, then was purchased by another company. During that time, a large part of our operations was moved from one state to another, hardware and all (but mostly not the people). This predates all of that, of course. At some point, a NetVista was put in place to test setting up an automated process for billing this carrier -- something carrier imposed (must have been one of the big guys). The developer who set up the test system was successful ... on his final week of work before being laid off. A few months later, the carrier continued working the migration plan and switched things over, and after a short delay, accounting rang the alarm bells. A busy developer stepped in, found the offending system was connected to test and re-configured it to point to prod. Everyone went on their day. And hey, when desktop migrated everyone to Windows XP, they put UPSes at every desk so it literally ran in a cubicle until the Data Center migration (where it failed the first time and the label was printed). Rather than figuring out what, on earth, it would take to fix it, they put it on a shelf and plugged it into the UPS in the rack until I was called several years later. [0] I was one of two people who were called when everything else was tried. This sort of incident happened to me in very similar ways at least 4 times (once with an old Thinkpad Laptop).

swebs 7y ago

How do we increase IT competence in the public sector? There's no reason important servers should be running on Windows machines with "backups" consisting of making a copy of a file on the same hard drive as the original.

AnIdiotOnTheNet 7y ago
First you'd have to solve politics. Good luck with that.
magashna 7y ago
You can't do anything but let it publicly fail. These incompetencies are usually the result of the top not believing that the cost is worth it. Also a well run IT department often gets "nothing's ever broken, what do they actually do?" so you get rewarded for doing break/fix.
- toomuchtodo 7y ago
  I am an infosec professional who pitches utilities and public government. This is spot on. The second call back is always more expensive for them, but sometimes someone needs to touch the stove themselves to see it’s hot. If you genuinely can’t afford it, I’ll help for free, but there’s only so much time in the day and I won’t eat the cost of stupid or politics.
  - ben_jones 7y ago
    Is it not also because the government contracting system is FUBAR? I.e. only certain kinds of companies can contract on the big projects, you need certain clearances for certain kinds of works, and the pay gap between public and private sector SWE & security? Also aren't their some pretty tough mandates like x% of companies must be minority owned etc. I imagine it allows those companies to charge extortionist rates because they hit the relevant quotas and nobody else does.
    - iguy 7y ago

rednerrus 7y ago

This headline gives me anxiety.

duckqlz 7y ago

TLDR; if your company / city has millions invested in a cyber infrastructure get cyber insurance. :facepalm: Also $76,000 is oddly low... like low enough to be the price of a well established firm to do a pen test on your network...

davinic 7y ago
True, but managers also usually fail to account for the time and cost of remediation after a pentest. Had the city engaged in the pentest, would there have been an appetite to spend 10-20x that amount on remediation?
- NullPrefix 7y ago
  Paying the ransom doesn't cover the need to spend money on fixing the problems. Having a ransom payer mark on the organization raises the security requirement, doesn't it?
magashna 7y ago
For personal ransomware, the requests are average around $250, lower than you'd think.

rsync 7y ago

I guess a failure to do backups of any kind is beyond any solution we could offer, but if one could take even a single step forward beyond nothing, a read-only (immutable) offsite destination is well worth your consideration.[1] Most end users looking for "ransomware protection" probably just want to drag and drop some files, like Dropbox, which is why a simple SFTP (filezilla) solution is nice, but of course you could point any old thing[2] if you were more sophisticated ... [1] https://www.rsync.net/products/ransomware.html https://www.rsync.net/products/ransomware.html [2] borg, restic, rclone, git-annex ... rsync ...

korzun 7y ago
I subscribed to your service a couple of years back, and you were leaking other customers' home directories. The whole thing was just a shared, dedicated server with shell access. Super sketchy. Back then, you also stated that you offer super-secure access and all of that jazz. Funny that you are double downing on all the compliance bullshit and ransomware buzz words now. The compliance statements are pretty questionable and make no sense for the services you are offering. Surprised you don't provide 'military-grade' encryption. Just throw all of that in there keep the thing. Buyers beware. Do your homework.
vultour 7y ago
Surprised someone would use or recommend FileZilla after their numerous adware issues.
- rsync 7y ago
  I'm just thinking of simple SFTP clients ... WinSCP ? psftp.exe ? Whatever works for you.

deleted 7y ago

[deleted]

jrochkind1 7y ago

The techdirt article author suggest they should have just paid the ransom. Even knowing that paying the ransom is no _guarantee_ you actually recover, and that you have to then harden your system anyway after that (which you should have been doing all along)... I tend to agree. I am a Baltimore resident, and I've also had that opinion since the day after the ransomware attack was announced -- just pay the ransom. (In part because from what I experience of Baltimore City government, i was pretty sure the backups and recovery/continuity plans were going to be basically nonexistent). But on previous HN threads, it was a very unpopular opinion, in the articles right after the attacks were announced, very few in the comments threads seemed to think it was acceptable or a good idea to pay the ransom. I'm curious if that remains true?

tempestn 7y ago
Just my personal opinion, but I figure the logical response would be to audit 1) the state of their backups, and 2) the source of the malware. After confirming that all necessary data had indeed been backed up and could be restored, and after blocking the vector used to deliver the malware, they could refuse to pay the ransom and restore from backups. Then they could go about further hardening. If it was discovered that the backups were faulty (as in this case), or if it were not possible to discover how the infection happened prior to the ransom deadline, I would have advised paying the ransom, since 76k is a relatively small amount to the city of Baltimore. (And then of course continuing to investigate and harden to avoid having to do so again.)
nickpeterson 7y ago
People have principles when it isn't their money. The best route would be to pay it, then divert the money and fix the issues. Think of the 76k as cheap penetration testing.
- itake 7y ago
  The problem with this thinking is: 1/ You have to trust the criminals are "good" people and actually took the time to write code to decrypt the data. 2/ Paying the ransom paints you an easy target for the next exploit.

totaldude87 7y ago

>>The person in charge of the city's systems was Frank Johnson, who went on leave (presumably permanently) after a post-attack audit found the IT director hadn't done much IT directing. love this writing man :) :) Dont know why would a city's chief digital officer go backup free?!!! even at worst , backblaze would have helped :|

ping_pong 7y ago

To be fair, the CIO was only on the job for 1.5 years when the ransomware hit. It's not known whether or not he had the budget for reorganizing all of the data, etc. Maybe it was on his list of priorities, but it didn't have the funding, etc, and it was pushed off until the next year. I'm not absolving him of this because backups are a first order object when you're dealing with IT, but it could be part of the reason why it wasn't implemented yet. This is the reality of working in a bureaucracy like city government.

linuxftw 7y ago

Baltimore (like many cities) is run by crooks and morons. They are a shining example of ineptitude and failure in all things.